Device data collection method based on edge computing

Abstract

The present invention discloses a device data collection method based on edge computing, including obtaining working data information of a data collection device; initializing and classifying the data collection device; clustering the data collection device; determining the data items that need to be compressed for each cluster; compressing the data and then uploading it; updating the clustering results; and repeating the above steps to complete the device data collection based on edge computing. The present invention also discloses a system for implementing the device data collection method based on edge computing. The present invention calculates compressible data items without manually specifying clustering and compression items; at the same time, the present invention can effectively utilize the computing power of the data collection device, reduce the amount of uploaded data, and improve the reliability of data collection by reducing the amount of uploaded data; therefore, the present invention has high reliability, good accuracy and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things, and specifically relates to a device data collection method and system based on edge computing. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electricity has become an indispensable secondary energy source in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the stable and reliable supply of electricity has become one of the most important tasks of the power system.

[0003] At present, with the development and improvement of the power Internet, more and more data acquisition devices with computing, storage and communication capabilities are connected to the power system. The data related to the operation of the power grid collected and transmitted by these devices are extremely important to the power system. Therefore, it is particularly important for the power system to collect data from such devices stably and reliably.

[0004] At present, in the actual power system, the management of data acquisition equipment and the collection and processing of data are mainly realized through the Internet of Things management platform. The data acquisition equipment is generally wirelessly connected to the Internet of Things management platform through the APN network and exchanges data with the platform. However, in the current data collection scheme, these data collection devices are isolated and lack unified connection; although at the individual device level, the redundancy of the data uploaded by the device is small, but on the entire platform side, the data uploaded by the massive data collection devices will be extremely redundant. The transmission of these redundant data requires a large amount of additional bandwidth and communication resources, which greatly increases the burden on the Internet of Things management platform; at the same time, the large amount of redundant data transmission also greatly reduces the efficiency of the data collection process. Summary of the invention

[0005] One of the purposes of the present invention is to provide a device data acquisition method based on edge computing with high reliability, good accuracy and high efficiency.

[0006] A second objective of the present invention is to provide a system for implementing the device data collection method based on edge computing.

[0007] The device data collection method based on edge computing provided by the present invention comprises the following steps:

[0008] S1. Obtain working data information of data acquisition equipment;

[0009] S2. Initialize and classify the data acquisition device according to the working data information obtained in step S1;

[0010] S3. Clustering the data acquisition devices based on the classification algorithm and the classification results of step S2;

[0011] S4. Determine the data items that need to be compressed for each cluster based on the currently obtained clustering results;

[0012] S5. Each data collection device compresses the collected data and then uploads it according to the data items confirmed in step S4;

[0013] S6. Update the clustering result obtained in step S3;

[0014] S7. Repeat steps S4 to S6 to complete device data collection based on edge computing.

[0015] The step S2 of initializing and classifying the data acquisition device according to the working data information obtained in step S1 specifically includes the following steps:

[0016] Data acquisition devices that use the same data items are grouped together, and the format of data uploaded by this type of data acquisition devices is set to be the same.

[0017] The step S3 described in which the data acquisition devices are clustered based on the classification algorithm and the classification result of step S2 specifically includes the following steps:

[0018] A. Obtain historical data from all N data collection devices. Each data collection device obtains Z data. The data node set is represented as X = {x 1 ,x 2 ,...,x N}, where x n Indicates the nth data acquisition device D n The characteristic matrix of n The dimension is M×Z, where M is the nth data acquisition device D n The number of data items collected; variables Represents the feature matrix x n The value of the element with coordinates (m,z);

[0019] B. Initialize the cluster head set:

[0020] Randomly select a node from the set X as the initial cluster head C 1 ; Each node calculates the shortest distance d between itself and the current cluster head nk ;

[0021] Calculate the probability p of each node being selected as the next cluster head node n for d n is the distance between the nth node and the existing cluster head;

[0022] According to the calculated probability p of each node n , randomly select a node as the next cluster head node;

[0023] Repeat the above steps until a set number K of cluster head nodes are obtained, so that the cluster head node combination C is C = {C 1 ,C 2 ,...,C K};

[0024] C. Clustering each node: Calculate the distance between each node and all the currently obtained cluster head nodes; each node adds itself to the cluster where the cluster head closest to itself is located, thus completing the clustering process of each node;

[0025] D. Update the cluster head:

[0026] For each cluster, the reference cluster head of the current cluster is calculated using the following formula:

[0027]

[0028] Where |G k | indicates the number of nodes in the current cluster;

[0029] Select the node closest to the reference cluster head from the current cluster as the new cluster head of the current cluster, expressed as in Represents x n and The distance between

[0030] After completing this step for each cluster, the new cluster head set C' = {C 1 ',C' 2 ,...,C' K} and clustering results;

[0031] E. Repeat steps C to D until the cluster head set is no longer updated or the preset maximum number of iterations is reached to obtain the final clustering result and cluster head set.

[0032] The distances described in step B and step C are Hamming distances. In specific implementation, the Hamming distance d is calculated using the following formula: nk :

[0033]

[0034] In the formula is an XOR operation; is the cluster head C' k The element value corresponding to the mid-coordinate (m,z);

[0035] The step S4 of determining the data items to be compressed in each cluster according to the currently obtained clustering result specifically includes the following steps:

[0036] For each cluster G k , get the latest data of each node, the composition dimension is |G k |×M data Y k ,|G k | is the number of nodes in the current cluster;

[0037] For data Y k Perform row-wise difference operation to obtain in The data in the first row of is calculated using the following formula:

[0038]

[0039] in For data Y k The l+1th row of data in the , and the dimension is 1×M; the value of l is 1,2,...,|G k |-1; f(x,y) is a processing function used to normalize the difference results, and the calculation formula of f(x,y) is is the set decimal;

[0040] Based on data Determine the data items to be compressed:

[0041] If the data If all the columns in the table are 0, the data items corresponding to the column are added to the compressed list L k At the same time, add the data items corresponding to the columns that are not all 0 to the normal list

[0042] The compressed list L k The values ​​of the data items are combined and the corresponding hash value h is calculated. k .

[0043] Each data collection device in step S5 compresses the collected data according to the data items confirmed in step S4 and then uploads it, which specifically includes the following steps:

[0044] Each data acquisition device obtains the compressed list L obtained in step S4 k , Normal list hash value h k and the corresponding parameters;

[0045] After each data collection device completes data collection, it selectsk Combine the data items in and calculate the corresponding hash value H k , and make a judgment:

[0046] If H k =h k , the data collection device only uploads the normal list The data of the corresponding data item in ;

[0047] If H k ≠h k , then the data collection device uploads the data of all data items;

[0048] After receiving the data, determine the length of the received data:

[0049] If the received data length is the same as the normal list If the length of the corresponding data in is consistent, the received data is compared with the compression list L k After the corresponding data are aggregated, complete data is obtained;

[0050] If the received data length is the same as the normal list If the corresponding data lengths are inconsistent, the received data will be regarded as complete data.

[0051] Step S6 of updating the clustering result obtained in step S3 specifically includes the following steps:

[0052] The compression ratio p of each cluster is calculated using the following formula: Where B is the amount of compressed data, and A is the total amount of data uploaded by the current cluster;

[0053] Determine the compression ratio:

[0054] If p 2 <p≤p 1 , then the corresponding cluster is processed in step S4 to re-determine the data items that need to be compressed in the corresponding cluster; 1 is the first threshold value, p 2 is the set second threshold;

[0055] If p≤p 2 , then step S3 is used to re-cluster the clusters, and then step S4 is used to determine the data items that need to be compressed for the re-divided clusters.

[0056] Step S6 of updating the clustering result obtained in step S3 specifically includes the following steps:

[0057] Set clustering time T;

[0058] At every set clustering time T, all data acquisition devices are re-divided into clusters in step S3, and then data items to be compressed are determined in the re-divided clusters in step S4.

[0059] The present invention also discloses a system for realizing the device data acquisition method based on edge computing, including a data acquisition module, an initial classification module, a clustering module, a compression module, an upload module, an update module and a circulation module; the data acquisition module, the initial classification module, the clustering module, the compression module, the upload module and the update module are connected in series in sequence; the input end of the circulation module is connected to the output end of the update module, and the output end of the circulation module is connected to the compression module; the data acquisition module is used to acquire the working data information of the data acquisition device and upload the data to the initial classification module; the initial classification module is used to initialize and classify the data acquisition device according to the received data, and upload the data to the clustering module; the clustering module The module is used to cluster the data acquisition devices according to the received data, and upload the data to the compression module; the compression module is used to determine the data items that need to be compressed for each cluster according to the received data and the currently obtained clustering results, and upload the data to the upload module; the upload module is used to compress the collected data according to the confirmed data items of each data acquisition device according to the received data, and then upload the data to the update module; the update module is used to update the clustering results according to the received data, and upload the data to the loop module; the loop module is used to repeat the operation of the compression module, the upload module and the update module to complete the device data acquisition based on edge computing.

[0060] The device data collection method and system based on edge computing provided by the present invention calculates compressible data items without the need for manual specification of clustering and compression items; at the same time, the present invention can effectively utilize the computing power of the data collection device, reduce the amount of uploaded data, and improve the reliability of data collection by reducing the amount of uploaded data; therefore, the present invention has high reliability, good accuracy and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0062] Figure 2 Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0063] like Figure 1 The method flow chart of the method of the present invention is shown as follows: The device data collection method based on edge computing provided by the present invention comprises the following steps:

[0064] S1. Obtain working data information of data acquisition equipment;

[0065] S2. Initialize and classify the data acquisition device according to the working data information obtained in step S1; specifically, the steps include:

[0066] The data collection devices that use the same data items are grouped into one category, and the format of data uploaded by the data collection devices in this category is set to be the same;

[0067] S3. Clustering the data acquisition devices based on the classification algorithm and the classification result of step S2; specifically comprising the following steps:

[0068] A. Obtain historical data from all N data collection devices. Each data collection device obtains Z data. The data node set is represented as X = {x 1 ,x 2 ,...,x N}, where x n Indicates the nth data acquisition device D n The characteristic matrix of n The dimension is M×Z, where M is the nth data acquisition device D n The number of data items collected; variables Represents the feature matrix x n The value of the element with coordinates (m,z);

[0069] B. Initialize the cluster head set:

[0070] Randomly select a node from the set X as the initial cluster head C 1 ; Each node calculates the shortest distance d between itself and the current cluster head nk ; In specific implementation, if the distance selected is the Hamming distance, the following formula is used to calculate the Hamming distance d nk :

[0071]

[0072] In the formula is an XOR operation; is the cluster head C' k The element value corresponding to the mid-coordinate (m,z);

[0073] Calculate the probability p of each node being selected as the next cluster head node n for d n is the distance between the nth node and the existing cluster head;

[0074] According to the calculated probability p of each node n , randomly select a node as the next cluster head node;

[0075] Repeat the above steps until a set number K of cluster head nodes are obtained, so that the cluster head node combination C is C = {C 1 ,C 2 ,...,C K};

[0076] C. Clustering each node: Calculate the distance between each node and all the currently obtained cluster head nodes; each node adds itself to the cluster where the cluster head with the closest distance (preferably Hamming distance) to itself is located, thereby completing the clustering process of each node;

[0077] D. Update the cluster head:

[0078] For each cluster, the reference cluster head of the current cluster is calculated using the following formula:

[0079]

[0080] Where |G k | indicates the number of nodes in the current cluster;

[0081] Select the node closest to the reference cluster head from the current cluster as the new cluster head of the current cluster, expressed as in Represents x n and The distance between

[0082] After completing this step for each cluster, the new cluster head set C' = {C 1 ',C' 2 ,...,C' K} and clustering results;

[0083] E. Repeat steps C to D until the cluster head set is no longer updated or the preset maximum number of iterations is reached, and the final clustering result and cluster head set are obtained;

[0084] The reason for clustering is to group devices with similar data collection into one cluster based on historical data, thereby making compression more likely;

[0085] S4. Determine the data items that need to be compressed in each cluster according to the currently obtained clustering result; specifically, the steps include:

[0086] For each cluster G k , get the latest data of each node, the composition dimension is |G k |×M data Y k ,|G k | is the number of nodes in the current cluster;

[0087] For data Yk Perform row-wise difference operation to obtain in The data in the first row of is calculated using the following formula:

[0088]

[0089] in For data Y k The l+1th row of data in the , and the dimension is 1×M; the value of l is 1,2,...,|G k |-1; f(x,y) is a processing function used to normalize the difference results, and the calculation formula of f(x,y) is is the set decimal;

[0090] Based on data Determine the data items to be compressed:

[0091] If the data If all the columns in the table are 0, the data items corresponding to the column are added to the compressed list L k At the same time, add the data items corresponding to the columns that are not all 0 to the normal list

[0092] The compressed list L k The values ​​of the data items are combined and the corresponding hash value h is calculated. k ;

[0093] S5. Each data collection device compresses the collected data and then uploads it according to the data items confirmed in step S4; specifically, the steps include:

[0094] Each data acquisition device obtains the compressed list L obtained in step S4 k , Normal list hash value h k and the corresponding parameters;

[0095] After each data collection device completes data collection, it selects k Combine the data items in and calculate the corresponding hash value H k , and make a judgment:

[0096] If H k =h k , the data collection device only uploads the normal list The data of the corresponding data item in ;

[0097] If H k ≠h k , then the data collection device uploads the data of all data items;

[0098] After receiving the data, determine the length of the received data:

[0099] If the received data length is the same as the normal list If the length of the corresponding data in is consistent, the received data is compared with the compression list L k After the corresponding data are aggregated, complete data is obtained;

[0100] If the received data length is the same as the normal list If the corresponding data lengths are inconsistent, the received data will be regarded as complete data;

[0101] S6. Update the clustering result obtained in step S3;

[0102] When updating, you can use active update or passive update.

[0103] If active update is adopted, the specific steps include the following:

[0104] The compression ratio p of each cluster is calculated using the following formula: Where B is the amount of compressed data, and A is the total amount of data uploaded by the current cluster;

[0105] Determine the compression ratio:

[0106] If p 2 <p≤p 1 , then the corresponding cluster is processed in step S4 to re-determine the data items that need to be compressed in the corresponding cluster; 1 is the first threshold value, p 2 is the set second threshold;

[0107] If p≤p 2 , then re-clustering is performed in step S3, and then step S4 is performed to determine the data items that need to be compressed for the re-divided clusters;

[0108] If passive update is adopted, the specific steps include:

[0109] Set clustering time T;

[0110] Every set clustering time T, all data acquisition devices are re-divided into clusters in step S3, and then data items that need to be compressed are determined in the re-divided clusters in step S4;

[0111] S7. Repeat steps S4 to S6 to complete device data collection based on edge computing.

[0112] like Figure 2The functional module diagram of the system of the present invention is shown as follows: the system disclosed in the present invention for implementing the device data acquisition method based on edge computing includes a data acquisition module, an initial classification module, a clustering module, a compression module, an upload module, an update module and a circulation module; the data acquisition module, the initial classification module, the clustering module, the compression module, the upload module and the update module are connected in series in sequence; the input end of the circulation module is connected to the output end of the update module, and the output end of the circulation module is connected to the compression module; the data acquisition module is used to acquire the working data information of the data acquisition device and upload the data to the initial classification module; the initial classification module is used to initialize and classify the data acquisition device according to the received data, and upload the data to the initial classification module. The clustering module is used to cluster the data acquisition devices according to the received data, and upload the data to the compression module; the compression module is used to determine the data items that need to be compressed for each cluster according to the received data and the currently obtained clustering results, and upload the data to the upload module; the upload module is used to compress the collected data according to the confirmed data items of each data acquisition device according to the received data, and then upload the data to the update module; the update module is used to update the clustering results according to the received data, and upload the data to the loop module; the loop module is used to repeat the operation of the compression module, the upload module and the update module to complete the device data acquisition based on edge computing.

Claims

1. A device data collection method based on edge computing, comprising the following steps: S1. Obtain working data information of data acquisition equipment; S2. Initialize and classify the data acquisition device according to the working data information obtained in step S1; specifically, the steps include: The data collection devices that use the same data items are grouped into one category, and the format of data uploaded by the data collection devices in this category is set to be the same; S3. Clustering the data acquisition devices based on the classification algorithm and the classification result of step S2; specifically comprising the following steps: A. Obtain the historical data of all N data acquisition devices. Each data acquisition device obtains Z data. The data node set is represented as X = {x1, x2, ..., x N }, where x n Indicates the nth data acquisition device D n The characteristic matrix of n The dimension is M×Z, where M is the nth data acquisition device D n The number of data items collected; B. Initialize the cluster head set: A node is randomly selected from the set X as the initial cluster head C1; each node calculates the shortest distance d between itself and the current cluster head nk ; Calculate the probability p of each node being selected as the next cluster head node n for d n is the distance between the nth node and the existing cluster head; According to the calculated probability p of each node n , randomly select a node as the next cluster head node; Repeat the above steps until a set number K of cluster head nodes are obtained, so that the cluster head node set C is C = {C1, C2, ..., C K }; C. Clustering each node: Calculate the distance between each node and all the currently obtained cluster head nodes; each node adds itself to the cluster where the cluster head closest to itself is located, thus completing the clustering process of each node; D. Update the cluster head: For each cluster, the reference cluster head of the current cluster is calculated using the following formula: Where |G k | indicates the number of nodes in the current cluster; variable Represents the feature matrix x n The value of the element with coordinates (m,z); Select the node closest to the reference cluster head from the current cluster as the new cluster head of the current cluster, expressed as in Represents x n and The distance between After completing this step for each cluster, a new cluster head set C' = {C1', C'2, ..., C' K } and clustering results; E. Repeat steps C to D until the cluster head set is no longer updated or the preset maximum number of iterations is reached, and the final clustering result and cluster head set are obtained; S4. Determine the data items that need to be compressed in each cluster according to the currently obtained clustering result; specifically, the steps include: For each cluster G k , get the latest data of each node, the composition dimension is G k |×M data Y k ,|G k | is the number of nodes in the current cluster; For data Y k Perform row-wise difference operation to obtain in The data in the first row of is calculated using the following formula: in For data Y k The l+1th row of data in the , and the dimension is 1×M; the value of l is 1,2,...,|G k |-1; f(x,y) is a processing function used to normalize the difference results, and the calculation formula of f(x,y) is is the set decimal; Based on data Determine the data items to be compressed: If the data If all the columns in the table are 0, the data items corresponding to the column are added to the compressed list L k At the same time, add the data items corresponding to the columns that are not all 0 to the normal list The compressed list L k The values ​​of the data items are combined and the corresponding hash value h is calculated. k ; S5. Each data collection device compresses the collected data and then uploads it according to the data items confirmed in step S4; S6. Update the clustering result obtained in step S3; S7. Repeat steps S4 to S6 to complete device data collection based on edge computing.

2. The device data collection method based on edge computing according to claim 1 is characterized in that The distances described in step B and step C are Hamming distances. In specific implementation, the Hamming distance d is calculated using the following formula: nk : In the formula is an XOR operation; is the cluster head C' k The element value corresponding to the mid-coordinate (m,z).

3. The device data collection method based on edge computing according to claim 1 is characterized in that Each data collection device in step S5 compresses the collected data according to the data items confirmed in step S4 and then uploads it, which specifically includes the following steps: Each data acquisition device obtains the compressed list L obtained in step S4 k , Normal list hash value h k and the corresponding parameters; After each data collection device completes data collection, it selects k Combine the data items in and calculate the corresponding hash value H k , and make a judgment: If H k =h k , the data collection device only uploads the normal list The data of the corresponding data item in ; If H k ≠h k , then the data collection device uploads the data of all data items; After receiving the data, determine the length of the received data: If the received data length is the same as the normal list If the length of the corresponding data in is consistent, the received data is compared with the compression list L k After the corresponding data are aggregated, complete data is obtained; If the received data length is the same as the normal list If the corresponding data lengths are inconsistent, the received data will be regarded as complete data.

4. The device data collection method based on edge computing according to claim 3 is characterized in that Step S6 of updating the clustering result obtained in step S3 specifically includes the following steps: The compression ratio p of each cluster is calculated using the following formula: Where B is the amount of compressed data, and A is the total amount of data uploaded by the current cluster; Determine the compression ratio: If p2<p≤p1, then the corresponding cluster is processed in step S4 to re-determine the data items that need to be compressed in the corresponding cluster; p1 is the set first threshold, and p2 is the set second threshold; If p≤p2, step S3 is used to re-cluster the clusters, and then step S4 is used to determine the data items that need to be compressed for the re-divided clusters.

5. The device data collection method based on edge computing according to claim 4 is characterized in that Step S6 of updating the clustering result obtained in step S3 specifically includes the following steps: Set clustering time T; At every set clustering time T, all data acquisition devices are re-divided into clusters in step S3, and then data items to be compressed are determined in the re-divided clusters in step S4.

Images