Scheduling processing method for high-frequency data acquisition of intelligent terminal
Through convolutional neural network and pooling operation algorithm, the problems of low frequency acquisition frequency and unreliable data at intelligent terminals are solved, high-speed and high-quality data acquisition are achieved, and data volume and space occupation are reduced. It is suitable for small samples Machine learning has economical and practical characteristics.
Patent Information
- Application Number
- CN202410877024.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional data processing methods cannot meet the high-frequency data acquisition needs of smart terminals, resulting in the problems of low acquisition frequency and unreliable data.
The convolutional neural network and pooling operation algorithm are used to extract features through the convolutional layer, and the pooling layer carries out data sorting and sampling, combining activation functions and loss functions to optimize the data processing flow to improve data acquisition efficiency.
It realizes high-speed and high-quality data acquisition, reduces data volume and space usage, avoids effective data loss, is suitable for small sample machine learning, avoids local minimum and over-learning and under-learning problems, and is economical and practical and easy to implement. .
Smart Images

Figure CN118965088A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electric power technology, and in particular to a scheduling processing method for high-frequency data collection of intelligent terminals. Background Art
[0002] With the development of the Internet of Things, the amount of data that intelligent data collection terminals need to process is increasing, and traditional data processing methods can no longer meet the demand.
[0003] As a deep learning model, convolutional neural network has achieved remarkable results in the fields of image recognition. Among them, convolutional neural network is mainly composed of convolutional layer, pooling layer and fully connected layer. Among them, the convolutional layer is responsible for extracting the features of the image, the pooling layer is responsible for reducing the dimension of the features, and the fully connected layer is responsible for outputting the final result. Convolutional neural network extracts features by performing convolution operation on the input image. This convolution can be regarded as the processing of the frequency domain information of the image after Fourier transformation. These frequency domain information contains the low-frequency and high-frequency components of the image, and has a good representation ability for the edge, texture and other features of the image. Therefore, we propose a method to apply convolutional neural network to improve the efficiency of high-frequency data collection of intelligent collection terminals. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of low collection frequency and unreliable data caused by large data collection volume in the prior art, and to provide a scheduling and processing method for high-frequency data collection of intelligent terminals. By improving the data processing speed, high-speed and high-quality application data is guaranteed, and a large amount of sensor data is synchronized at the same time, thereby driving the improvement of data collection speed.
[0005] To achieve the above object, the present invention provides a scheduling processing method for high-frequency data collection of intelligent terminals, comprising the following steps:
[0006] S1, obtaining the collected electricity consumption data, processing it and outputting it;
[0007] S2, transmits and outputs the data processed and input by S1, and transmits it to S3;
[0008] S3, using a convolutional neural network algorithm to identify and judge the data output by S2;
[0009] S4: Determine whether the data quality meets the requirements. If it does, go to S5; if it does not, go back to S2.
[0010] S5. Output the data to be processed as other input side.
[0011] Preferably, the electricity consumption data in S1 includes current, voltage and power.
[0012] Preferably, the specific judgment method of performing data recognition and judgment on the data output by S2 in S3 by using a convolutional neural network algorithm includes:
[0013] S301, extracting features from the original data through a convolutional layer of a convolutional neural network;
[0014] S302, extracting important features by performing pooling operation through the pooling layer of the convolutional neural network.
[0015] Preferably, extracting features from raw data in 301 specifically refers to performing a convolution operation on the sensor data, wherein the convolution operation performs spatial convolution and temporal convolution on the sensor data.
[0016] Preferably, the sensor data is subjected to a convolution operation, and extracting important features specifically refers to inputting the sensor data into a convolutional neural network, performing convolutional neural network calculations, and extracting feature information by sliding the convolution kernel on the input data and calculating the weighted sum.
[0017] Preferably, the pooling operation performed by the pooling layer of the convolutional neural network in S302 specifically refers to comprehensively sorting and sampling the previously extracted features to extract important features.
[0018] Preferably, the method further includes S303, discarding part of the data and retaining valuable data.
[0019] Preferably, discarding part of the data in S303 and retaining the valuable data specifically refers to substituting the feature extraction in S302 into the activation function for activation, using the activation function to introduce nonlinear factors, and judging whether to retain or discard the data through the output value.
[0020] Preferably, the nonlinear factor is
[0021] f(x)=max(0,x)
[0022] Among them, x is the input data of the sensor and f(x) is the output value.
[0023] When the sensor input data x<0, the output value f(x)=0, that is, when the input data is empty, the calculation result of the returned data is empty.
[0024] Preferably, the method further includes S304, measuring the difference between the model prediction result and the actual result by using a loss function.
[0025] Beneficial effects of the present invention: The present invention provides a scheduling and processing method for high-frequency data collection of intelligent terminals, which uses a convolution cloud neural network to optimize and solve the shortcomings of low-frequency collection of most sensor data. And supplemented by a pooling operation algorithm, the problem of large data volume is overcome. The present invention uses a convolution cloud neural network algorithm, which has many advantages that traditional methods do not have: such as better solving the problem of large data volume and large space occupation, and will not cause the loss of valid data; it can handle machine learning of small samples, and will not fall into local minima and over-learning and under-learning problems. In addition, this method takes the software algorithm as the starting point, and is economical, practical and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A flowchart of a scheduling processing method for high-frequency data collection of intelligent terminals provided by the present invention;
[0027] Figure 2 A curve showing the relationship between training loss and verification accuracy of a scheduling processing method for high-frequency data collection of intelligent terminals provided by the present invention; DETAILED DESCRIPTION
[0028] The specific implementation of the present invention is described below with reference to the accompanying drawings.
[0029] A scheduling processing method for high-frequency data collection of intelligent terminals comprises the following steps:
[0030] S1, obtaining the collected electricity consumption data, processing it and outputting it;
[0031] S2, transmits and outputs the data processed and input by S1, and transmits it to S3;
[0032] S3, using a convolutional neural network algorithm to identify and judge the data output by S2;
[0033] S4: Determine whether the data quality meets the requirements. If it does, go to S5; if it does not, go back to S2.
[0034] S5. Output the data to be processed as other input side.
[0035] The power consumption data in S1 includes current, voltage and power.
[0036] In S3, the data output by S2 is identified and judged by the convolutional neural network algorithm. The specific judgment method includes:
[0037] S301, extracting features from the original data through a convolutional layer of a convolutional neural network;
[0038] S302, extracting important features by performing pooling operation through the pooling layer of the convolutional neural network.
[0039] Extracting features from raw data in 301 specifically refers to performing a convolution operation on the sensor data, where the convolution operation performs spatial convolution and temporal convolution on the sensor data.
[0040] The sensor data is convolved to extract important features. Specifically, the sensor data is input into the convolutional neural network, and the convolutional neural network is calculated. The feature information is extracted by sliding the convolution kernel on the input data and calculating the weighted sum.
[0041] The pooling operation performed by the pooling layer of the convolutional neural network in S302 specifically refers to comprehensively sorting and sampling the previously extracted features to extract important features.
[0042] The process also includes S303, discarding some data and retaining valuable data.
[0043] S303 discards part of the data and retains the valuable data, specifically refers to substituting the feature extraction in S302 into the activation function for activation, using the activation function to introduce nonlinear factors, and judging whether to retain or discard the data through the output value.
[0044] Preferably, the nonlinear factor is
[0045] f(x)=max(0,x)
[0046] Among them, x is the input data of the sensor and f(x) is the output value.
[0047] When the sensor input data x<0, the output value f(x)=0, that is, when the input data is empty, the calculation result of the returned data is empty.
[0048] The process also includes S304, measuring the difference between the model prediction result and the actual result by using a loss function.
[0049] Example 1
[0050] The present invention provides a scheduling processing method for high-frequency data collection of intelligent terminals, comprising the following steps:
[0051] S1. Rapidly process the collected electricity consumption data;
[0052] The electricity consumption data includes: current, voltage and power.
[0053] S2, transmits and outputs the data processed and input by S1, and transmits it to S3;
[0054] Specifically, S1 performs basic data analysis, filters out insensitive data in the configuration project table, or processes it according to the customer's algorithm; and filters the data, according to the configuration project table, filters out data with too high a frequency of data acquisition, fills in data with insufficient frequency, and transmits and outputs the input data, and then passes it to S3
[0055] In S2, the data collected by S1 are simply processed to remove the data with large errors and keep the data close to the true average, thereby improving the accuracy of the data and the calculation efficiency.
[0056] S3, identify and judge the input data of S2, and analyze the results according to the authenticity and importance of the data. If the data meets the requirements, enter S4, otherwise return to S2;
[0057] The identification and judgment of the input data of S2 specifically refers to the identification and judgment of the output data of S2 by S3 through the convolutional neural network algorithm. The specific judgment method includes:
[0058] S301, extracting features from the original data through a convolutional layer of a convolutional neural network;
[0059] S302, extracting important features by performing pooling operation through the pooling layer of the convolutional neural network.
[0060] Features are extracted from the raw data, and the sensor data is convolved, which performs spatial and temporal convolutions on it.
[0061] S4. The data to be processed is output as other input side. This is the end.
[0062] Example 2
[0063] A scheduling processing method for high-frequency data collection of intelligent terminals comprises the following steps:
[0064] The sensor data is input to the convolutional neural network for convolutional neural network calculation. The feature information is extracted by sliding the convolution kernel on the input data and calculating the weighted sum. Here, it is assumed that the input data a is a two-dimensional array of size M×N, and the convolution kernel w is also a two-dimensional array of size F×F. The size of the output feature map b is (M-F+2P)×(N-F+2P), where P is the padding size.
[0065] First, slide the convolution kernel w on the input data a, each time sliding a step size S. For each position, calculate the sum of the products of the corresponding positions, that is:
[0066]
[0067] b[i,j]=u,v∑a[i+u,j+v]·w[u,v]
[0068] By transforming it, we can get
[0069] a[i,j]=u,v∑b[i+u,j+v]·w[u,v]
[0070] Among them, i, j are the position coordinates on the output feature map b, m, n are the sliding positions on the convolution kernel w, and a[i+mj+n] is
[0071] , the value of the input data a at position (i+m,j+n), w[m,n] is the value of the convolution kernel w at position (m,n).
[0072] This formula is the basic principle of the convolution operation, and the weighted sum of the sliding window of the convolution kernel on the input data is used to extract the corresponding features. Furthermore, the input data of the sensor can be used for feature extraction.
[0073] The next step is to perform pooling, which is to comprehensively organize and sample the previous features, thereby reducing the amount of calculation and extracting more abstract features. That is:
[0074] b[i,j]=maxm,na[i+m,j+n]
[0075] The output value obtained has been secondary optimized for the sensor data. Through such an operation, important features can be extracted from the data obtained by the sensor and compressed into a smaller spatial size. This pooling operation can reduce the number of parameters and the amount of calculation, thereby improving the efficiency and performance of the model.
[0076] At the same time, after the above feature extraction is completed,
[0077] b[i,j]=maxm,na[i+m,j+n]
[0078] Substitute into the activation function to activate the new product, and use the activation function to introduce nonlinear factors, that is,
[0079] f(x)=max(0,x)
[0080] Here, the input data of the sensor is x, and f(x) represents the output value. At this time, when x<0, f(x)=0, that is, when the input data is empty, the calculation result of the returned data is also empty. After the calculation here, part of the data is discarded, and only the most valuable data is retained, which greatly reduces the redundancy and time of the calculation.
[0081] Furthermore, a loss function is defined to measure the difference between the model prediction results and the actual results. The mean square error is used for calculation and analysis, that is:
[0082]
[0083] Here, N represents the number of samples, and yi is the label value at this time. The output result is given to the L value through calculation. At this time, L is the loss value, which calculates the average of the sum of the squared differences between the true label value and the predicted result.
[0084] By comprehensively analyzing and processing large amounts of data, a large amount of data can be filtered and efficiently calculated, which can reduce the actual computing process consumption. Therefore, by connecting to multiple terminal sensors and processing data concurrently, the data can be comprehensively and efficiently analyzed, thereby improving the terminal collection efficiency.
[0085] Compared with the prior art, it has the following beneficial effects:
[0086] (1) The traditional method consumes too many resources and cannot meet the needs of data collection and transmission at the second level or even the millisecond level to meet the 300ms data scheduling requirements in the next stage. After processing, this method can meet the needs of data collection and transmission at the millisecond level to meet the 300ms data scheduling requirements in the next stage.
[0087] (2) Traditional data collection requires more than 5K / second of data per packet, and the resident memory occupies up to 4-5 times the memory, reaching 21K. This method can reduce the data volume to 165 bytes or even less, and the resident memory does not exceed 2 times the data volume.
[0088] In summary, the present invention provides a scheduling and processing method for high-frequency data collection of intelligent terminals, which utilizes the convolution cloud neural network to optimize and solve the shortcomings of low-frequency collection of most sensor data. And supplemented by the pooling operation algorithm, the problem of large data volume is overcome. The present invention utilizes the convolution cloud neural network algorithm, which has many advantages that traditional methods do not have: such as better solving the problem of large data volume and large space occupation, and will not cause the loss of valid data; it can handle machine learning of small samples, and will not fall into local minima and over-learning and under-learning problems. In addition, this method takes the software algorithm as the starting point, and is economical, practical and easy to implement.
[0089] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.
Claims
1. A scheduling processing method for high-frequency data collection of intelligent terminals, characterized in that: The following steps are involved: S1. Obtain the collected electricity consumption data, process it and output it; S2, transmits and outputs the data processed and input by S1, and transmits it to S3; S3, using a convolutional neural network algorithm to identify and judge the data output by S2; S4: Determine whether the data quality meets the requirements. If it does, go to S5; if it does not, go back to S2. S5. Output the data to be processed as other input side.
2. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 1 is characterized in that: The power consumption data in S1 includes current, voltage and power.
3. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 1 is characterized in that: In S3, the data output by S2 is identified and judged by a convolutional neural network algorithm. The specific judgment method includes: S301, extracting features from the original data through a convolutional layer of a convolutional neural network; S302, extracting important features by performing pooling operation through the pooling layer of the convolutional neural network.
4. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 3 is characterized in that: Extracting features from the raw data in 301 specifically refers to performing a convolution operation on the sensor data, where the convolution operation performs spatial convolution and temporal convolution on the sensor data.
5. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 4 is characterized in that: The sensor data is subjected to a convolution operation, and extracting important features specifically refers to inputting the sensor data into a convolutional neural network, performing convolutional neural network calculations, and extracting feature information by sliding the convolution kernel on the input data and calculating a weighted sum.
6. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 3 is characterized in that: The pooling operation performed by the pooling layer of the convolutional neural network in S302 specifically refers to comprehensively sorting and sampling the previously extracted features to extract important features.
7. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 3 is characterized in that: The process also includes S303, discarding some data and retaining valuable data.
8. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 7 is characterized in that: The discarding of part of the data and retaining of the valuable data in S303 specifically refers to substituting the feature extraction in S302 into the activation function for activation, using the activation function to introduce nonlinear factors, and judging whether to retain or discard the data by the output value.
9. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 7 is characterized in that: The nonlinear factor is f(x)=max(0,x) Among them, x is the input data of the sensor, and f(x) is the output value; When the sensor input data x<0, the output value f(x)=0, that is, when the input data is empty, the calculation result of the returned data is empty.
10. The scheduling algorithm for high-frequency data collection of intelligent terminals according to claim 7, characterized in that: The process also includes S304, measuring the difference between the model prediction result and the actual result by using a loss function.
