Stream data collection system and method based on edge node community network

Abstract

A stream data collection system based on an edge node community network, by setting an IoT device access module and a stream data collection and processing module in the edge node, for the collected multi-source heterogeneous stream data, the missing data is inferred and supplemented through the community information between the data sources and the temporal relationship of the stream data itself, the data time and space are aligned on demand through the spatiotemporal model, the data is fused, divided, merged, packaged and / or distributed through the data fusion rules, and a variety of data docking methods are provided for subsequent computing nodes, so that data collection is spontaneously performed on the edge node side without the need for unified data collection and calculation by the central node. The present invention can realize the processing and collection of stream data in edge computing scenarios, and can be docked with subsequent computing nodes in a variety of ways to meet the different data request requirements of different production scenarios.

Description

Technical Field

[0001] The present invention relates to a technology in the field of information processing, and specifically to a stream data collection system and method based on an edge node community network. Background Art

[0002] As an important technology in the application and development of the Internet of Things, edge computing can effectively disperse computing from a single-node cloud to edge nodes, thereby supporting the real-time computing needs of a large number of computing nodes in the Internet of Things. With the deepening of Industry 4.0, the Internet of Things and edge computing will play a greater role. Through the interconnection of industrial equipment, data, and control systems, the automation of production processes and the intelligence of industrial management can be achieved. Summary of the invention

[0003] In view of the fact that the data collection technology under the existing industrial big data generally rarely takes into account the community relationship between the upstream and downstream computing nodes of the edge nodes in the actual production process, and the lack of intelligent data collection, the present invention proposes a stream data collection system based on the edge node community network, which can realize the processing and collection of stream data in edge computing scenarios, and can be connected with subsequent computing nodes in various ways to meet the different data request requirements of different production scenarios.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a stream data collection system based on an edge node community network, comprising: a data exchange module, a data collection module constituting a data receiver, a data matching module, a data similarity calculation module constituting a data processor, a data completion module, a spatiotemporal data alignment module, a data packet encapsulation module constituting a data distributor, and a data packet distribution module, wherein: the data collection module is connected to the data source of heterogeneous stream data, and caches and uniformly manages the input stream data through a data pipeline; the data matching module generates metadata describing the data from the data collection module according to its source as a tag for matching the stream data and its data source calculation node, and outputs the labeled data to the data similarity calculation module; the data similarity calculation module calculates the data similarity within a time window for the stream data from the data matching module, and performs community connection enhancement processing; the data completion module calculates the community connection of the data source node from the data exchange module and the data source node from the data similarity calculation module according to the community connection of the data source node from the data exchange module and the data source node from the data similarity calculation module. The computing module uses the timing information of the data itself to infer and complete the missing data in the streaming data; the spatiotemporal data alignment module analyzes the spatiotemporal model, and aligns the completed data from the data completion module in terms of time dimension and space dimension according to the analysis results; the data packet encapsulation module reads and analyzes the fusion rules from the data exchange module and performs corresponding fusion operations on the completed and aligned streaming data from the data matching module, and blocks, merges and packages the fused data and refers to the data interface mapping table to pass the data packets formed by each type of streaming data to the corresponding output data pipeline; the data packet distribution module receives the data packets encapsulated by the data packet encapsulation module through the data pipeline, and sends the data packets to the subsequent computing nodes on demand through the computing node mapping table; the data exchange module completes the unified encapsulation of external information resources, and provides a unified calling interface and interface access security mechanism to support the unified access and update of external information resources within the system.

[0006] The data pipeline is implemented by means of message middleware technology.

[0007] The unified management refers to: adopting a one-to-many design mode in which multiple data sources share one data pipeline, and using a data pipeline mapping table to record the mapping relationship between the data source and the data pipeline.

[0008] The pipeline mapping table described above caches the external input stream data into the data pipeline corresponding to the data source according to the data source information, monitors the operation status of the existing data pipeline, monitors the current resource occupancy of each data pipeline, the amount of garbage inside the pipeline, and regularly performs garbage collection on pipelines with a large amount of garbage.

[0009] Preferably, for data with unclear sources, the data matching module calculates the semantic similarity between each of its fields and the fields of various types of data that have been marked, and classifies it to fuzzily infer its data source.

[0010] The degree of data similarity is calculated in the following manner:

[0011] ① For streaming data related to time information, according to the change trend of the streaming data, i.e., accelerating decline, decline, decelerating decline, basically unchanged, decelerating increase, increase, accelerating increase, it is represented by the pattern set M = {-3, -2, -1, 0, 1, 2, 3} respectively.

[0012] ② Divide the two streaming data to be calculated, S1 = {(v 11 , t 11 ), (v 12 , t 12 ), …, (v 1n , t 1n )} and S2 = {(v 21 , t 21 ), (v 22 , t 22 ), …, (v 2m , t 2m )} into k segments and independently calculate which one of the pattern set M each segment belongs to: For each divided segment, calculate its curve slope k i , and judge that when k i < -th, the pattern m i of this segment ∈ {-3, -2, -1}, where: the threshold of the slope th > 0; then calculate the slope k i+1 of the next segment, and calculate the change in slope Δk i = k i+1 - k i , when Δk i < 0, the pattern of the i-th segment is calibrated as -3; when Δk i = 0, then the pattern of the i-th segment is calibrated as -2; when Δk i > 0, then the pattern of the i-th segment is calibrated as -1; when -th < k i < th, calibrate the pattern of the i-th segment as 0; when k i > th, the calibration judgment is the same as the case of less than.

[0013] ③ After calibration, the marked data S1 = {(m 11 , t 11 ), (m 12 , t 12 ), …, (m 1k , t 1k )}, S2 = {(m21 , t 21 ), (m 22 , t 22 ),…,(m 2k , t 2k )}, and then calculate the forward distance between S1 and S2 in: t i is the time span of the ith mode, t N is the total time length.

[0014] ④ Since there may be a high possibility of reverse correlation, that is, S1 and S2 are negatively correlated, the negative distance between S1 and S2 is further calculated. This is equivalent to taking the opposite number of each mode of S2 to calculate the forward distance and finally obtain the data similarity.

[0015] The community connection enhancement processing refers to: using the similarity of the latest received stream data, the community connection between the previous nodes, that is, the connection and weight value of the two computing nodes in the computing node topology map is updated, specifically: according to the data source recorded in the previously marked data metadata, the community information of this computing node is read, and according to the previously calculated data similarity between the two, the community connection between the data source nodes of similar data is enhanced to make them closer in the community relationship; or the community connection between the data source nodes of dissimilar data is weakened to make them looser in the community relationship, so as to support subsequent community-based missing data reasoning work.

[0016] The enhancement or weakening mentioned above refers to: increasing / decreasing the absolute value of its weight value, but maintaining the positive or negative nature of its original weight.

[0017] The inference and completion described herein include: data time series correlation discovery and completion, community correlation discovery and completion, and missing data reasoning.

[0018] The data time series correlation discovery and completion mentioned above refers to: using the time series characteristics of the stream data itself, and using the generative adversarial network (GAN) to infer the missing data at certain moments. The specific algorithm is: first generate a random time series data z with the same time distribution as the input data, the value of the time series data satisfies the Gaussian distribution, and the average value is zero; then pass the random time series data z into the generative network in the trained generative adversarial network, and use the generative network to convert this random time series data into time series data G(z) with a similar time series relationship with the original data; then use the data in G(z) to complete the original data.

[0019] The generative adversarial network is preferably trained in the following manner: the collected data x0 with missing information is passed into the generative adversarial network (GAN), and the generator and discriminator in the generative adversarial network are trained with this data, wherein: the training goal of the generator network is to obtain a mapping relationship G, which maps the randomly distributed time series data z to the generated data x with similar time series information to the input data x0, that is, G(z)=z→x; the training goal of the discriminator network is to obtain a mapping relationship D, which maps the time series data x to a variable sim, the value of which represents the degree of similarity between x and x0 in terms of the time series relationship, that is, D(x)=x→sim.

[0020] The training further includes three loss functions, among which: reconstruction loss L r The distance between G(z) generated by the generative network and the original data x0; difference loss L d The temporal correlation variable generated by the discriminative network, namely L d =D(G(z)); comprehensive loss L a By L r and L d The generated adversarial network uses the comprehensive loss L during training. a And back propagation method, the generation network and the discrimination network are trained to obtain mappings G and D.

[0021] The community association discovery and completion mentioned above refers to: using the community information of the data source node marked in the metadata and the data generated at each moment by the nodes with similar community relationships to infer the missing data at certain moments, specifically: reading the current data source node, adjacent nodes and edge weights w representing the closeness of community relationships from the data exchange module ei ; Then divide the time interval from t0 to t1 with missing data into k segments, and divide the data generated by the neighboring nodes of the computing node with missing data from t0 to t1 into k segments. For each segment of data, calculate its corresponding data change pattern m i , and use |w ei | is used as the weight, and the weighted average is calculated to finally obtain the data change pattern m of the missing data in the i-th segment, that is, Where: n represents the number of adjacent computing nodes. Afterwards, using this variation pattern, the slope k=m*th of the missing data segment is obtained, and the specific missing data of the i-th segment is calculated using this slope.

[0022] The edge weight w ei The absolute value of is used to represent the similarity between nodes, and its positive or negative nature determines whether the data of the two nodes are positively correlated or negatively correlated.

[0023] The missing data inference refers to: taking a weighted average of the inference results obtained by time series correlation and the inference results obtained by community correlation, inferring the final value of the missing data, and filling it into the original data stream as part of the data stream.

[0024] The weighted average, the weight of the timing correlation dynamically adjusted during operation is w seq , the weight of community association is w edge The data completion module periodically selects the data that is not missing as the target data, calculates the optimal weight under this target data, and adjusts the weight w used in the actual reasoning process according to the calculation result. seq and w edge .

[0025] The space-time model refers to: a series of time series as alignment standards in the time dimension; a series of spatial position points as alignment standards in the space dimension.

[0026] The alignment of the time dimension refers to: using the derivative dynamic time warping algorithm (DDTW) proposed by Eamonn J. Keogh et al. in the article Derivative Dynamic Time Warping in the time dimension to align the stream data with its standard time series in the spatiotemporal model.

[0027] The alignment of the spatial dimension refers to: in terms of the spatial dimension, taking the spatial position point of the alignment standard of the model as a reference, gathering the nodes around the spatial position point and performing spatial position alignment according to the accuracy requirement.

[0028] The fusion operation refers to: simple operations on multiple data fields and reintegration, specifically: performing simple operations on multiple different data fields of multiple stream data, such as addition, subtraction, multiplication, division, power and other mathematical operations and / or using multiple different data fields of multiple stream data as data input, calculating a result according to a custom formula, and then integrating the obtained calculation result into one of the fields of the output stream data.

[0029] The said block division means: dividing the merged stream data into a number of sub-segments of data according to a specified time interval in the time dimension.

[0030] The merging refers to simply piecing together and integrating multiple stream data that are aligned in the time dimension after fusion.

[0031] The packaging refers to: completing a unified packaging operation on the data after the block and merge are completed according to the segmentation boundaries obtained by the block and merge calculations, and outputting them in the form of data packets with a unified data format.

[0032] Preferably, each type of stream data corresponds to the length of the time interval in one-to-one correspondence.

[0033] The on-demand sending means that: the data packet distribution module maintains the mapping relationship between the subsequent computing nodes connected to the system and the data packet access requirements through the computing node mapping table, wherein: for the computing node that obtains data in the "push data" mode, that is, the system actively sends data to the subsequent computing node, the required data packet is pushed to the receiving interface of the designated computing node; for the computing node that obtains data in the "pull data" mode, that is, the subsequent computing node actively requests data from the system, the required data packet is cached in the data pipeline bound to it, waiting for it to actively consume the data.

[0034] The present invention relates to a stream data collection method based on the above system. By setting an Internet of Things device access module and a stream data collection and processing module in the edge node, the missing data of the collected multi-source heterogeneous stream data is inferred and supplemented through the community information between the data sources and the temporal relationship of the stream data itself, the time and space alignment of the data is performed on demand through the spatiotemporal model, the data is fused, blocked, merged, packaged and / or distributed through the data fusion rules, and a variety of data docking methods are provided for subsequent computing nodes, so that data collection is spontaneously performed on the edge node side without the need for unified data collection and calculation by the central node.

[0035] Technical Effects

[0036] The present invention comprehensively solves the problems that the data received by existing computing nodes is too disorganized and seriously missing; the problem that the level of intelligence in the collection of streaming data in the industrial production edge computing environment is low; and the problem that the missing data reasoning ability and data completion effect in edge computing are poor.

[0037] Compared with the prior art, the present invention can package and distribute streaming data in a unified data packet form according to the needs of computing nodes, providing strong data support for subsequent computing nodes in the computing topology. At the same time, the accuracy of missing data reasoning is enhanced by using the community information of computing nodes in edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the system of the present invention;

[0039] Figure 2 Schematic diagram of the embodiment process. DETAILED DESCRIPTION

[0040] like Figure 1As shown, the application scenario of this embodiment includes: a data acquisition system management cloud platform, a data acquisition layer and an infrastructure layer, wherein: the data acquisition system management cloud platform includes a business information configuration module and a real-time data display module, the data acquisition layer includes an Internet of Things device, the data acquisition system of the present invention and an edge computing node, and the infrastructure layer stores the external data that the data acquisition layer depends on and configurable business scenario-related data.

[0041] The business information configuration module calls the RESTful interface through the service to configure the business and computing topology of the data acquisition system; the real-time data display module obtains the data information and system operation status collected in real time by the data acquisition layer through the real-time information push Websocket interface, and performs visual operations on it and displays it to the system administrator.

[0042] The IoT device is the producer of streaming data, the edge computing node is the consumer of streaming data, and the data acquisition system is the middleware that communicates the two. In terms of IoT devices, it connects to the system through the interface provided by the data acquisition module in the data acquisition system, and transmits the data it produces to the data acquisition system. At the same time, as a consumer of streaming data, the edge computing node interacts with the data packet distribution module in the data acquisition system according to its own data consumption needs, and pulls data from it or has the data acquisition system push data to it.

[0043] like Figure 1 As shown, the data acquisition system comprises:

[0044] a. The data receiver used for data collection and preprocessing of IoT devices uses the data pipeline to receive the streaming data generated by IoT devices and caches it in the data pipeline. At the same time, the semantic information of the data field is used to infer, match and annotate the data source of the streaming data with unknown sources. The data receiver uses the Websocket interface to push real-time data collection and data pipeline status to the cloud platform, and uses the garbage collection mechanism to recycle redundant and invalid data to ensure the stable and smooth operation of the data pipeline.

[0045] b. A data processor for analyzing and processing the collected stream data. It calculates the similarity between the stream data and uses the similarity to update the community information between the computing nodes. After that, the community relationship of the computing nodes and the time series information of the time series data itself are used to complete the missing data during the transmission or collection process. At the same time, the externally defined data node spatial model is obtained, and the spatial and temporal dimensions of the completed data are aligned.

[0046] c. A data distributor for packaging and distributing processed stream data. It uniformly packages and encapsulates the processed stream data according to the data requirements of the subsequent computing node configuration. At the same time, for the different data acquisition methods of the subsequent computing nodes, two computing node docking methods, push data and pull data, are provided. In this way, data encapsulation, packaging and distribution operations are realized.

[0047] d. The data exchange module used to connect external information resources with this system. It completes the unified encapsulation of external information resources and provides a unified calling interface to provide a basis for the system to access external information resources. At the same time, it responds to the cloud platform's configuration request for external information resources and configures business-related information. In terms of security, a token-based authentication mechanism is used to verify the identity and permission information of the interface caller, thereby intercepting illegal or unauthorized data requests.

[0048] The infrastructure layer uses BPMN or BPEL files to store business process models, uses CSV files to store topological relationships of computing nodes, and uses non-relational databases to store custom data fusion rules and computing node space models.

[0049] like Figure 2 As shown, the data acquisition module includes: a stream data receiving and acquiring unit, a data pipeline status monitoring unit, a buffer garbage collection unit, and a data pipeline mapping table, wherein: the stream data receiving and acquiring unit performs simple acquisition processing based on the data from the IoT device and outputs the obtained stream data to the data pipeline, the data pipeline status monitoring unit performs resource consumption degree calculation processing based on the data from the data pipeline and outputs the obtained data pipeline operation status to the front-end user interface, the buffer garbage collection unit performs data garbage discovery processing based on the data from the data pipeline and collects and deletes the obtained data garbage, and the data pipeline mapping table records the mapping relationship between the data pipeline and the data source.

[0050] The data matching module includes: a data source dynamic matching unit, a metadata generation and marking unit, and a data information processing unit, wherein: the data source dynamic matching unit performs data source fuzzy matching processing based on data of unknown source from a data pipeline and outputs the obtained data source matching result to the metadata generation and marking unit; the metadata generation and marking unit performs metadata generation and marking processing based on data from the data source dynamic matching unit and the data pipeline and outputs the obtained marked stream data result to the data information processing unit; the data information processing unit performs data similarity calculation, data completion and spatiotemporal data alignment processing based on data from the source data generation and marking unit and outputs the obtained aligned and completed stream data result to the data packet encapsulation module.

[0051] The data packet encapsulation module includes: a fusion rule parsing unit, a data fusion and packaging unit, a data packet sending unit and an output interface mapping table, wherein: the fusion rule parsing unit performs rule parsing processing according to the data fusion rule from the data exchange module and outputs the parsed data fusion rule to the data fusion and packaging unit; the data fusion and packaging unit performs data fusion and packaging processing according to the data from the fusion rule parsing unit and the data matching module and outputs the obtained fused and packaged data packet result to the data packet sending unit; the data packet sending unit performs simple forwarding processing according to the data from the data fusion and packaging unit and outputs the obtained data packet result to the data output interface corresponding to the data in the output interface mapping table; the output interface mapping table records the mapping relationship between the flow data target node and different output interfaces.

[0052] The data packet distribution module includes: a data packet cache unit, a computing node docking unit and a computing node mapping table. Among them: the data packet cache unit performs data cache processing according to the data from the data packet encapsulation module and outputs the obtained data packet results to the data pipeline, the computing node docking unit performs data parsing processing according to the computing node docking method from the computing node mapping table, and according to the parsing result, outputs the data packet in the data pipeline to the designated data recipient, or does not output it and waits for the response recipient to obtain the data by itself, and the computing node mapping table records the mapping relationship between the data recipient and the data pipeline and the data packet acquisition method.

[0053] The data similarity calculation module includes: a community discovery unit and a node community optimization unit. Among them: the community discovery unit performs similarity calculation processing based on the data from the data matching module and outputs the obtained data line similarity calculation result to the node community optimization unit, and the node community optimization unit performs data similarity analysis processing based on the data from the community discovery unit, and combines the node community information data from the data exchange module to calculate the updated node community data, and outputs the obtained updated node community data result to the data exchange module.

[0054] The data completion module includes: a time series correlation discovery unit, a community correlation discovery unit and a missing data reasoning unit. Among them: the time series correlation discovery unit performs stream data time series information discovery processing based on the data from the data similarity calculation module and outputs the obtained stream data time series information result to the missing data reasoning unit, the community correlation discovery unit performs node community information analysis processing based on the data from the data exchange module and outputs the obtained calculation node community correlation result to the missing data reasoning unit, the missing data reasoning unit performs data analysis based on the stream data time series information from the time series correlation discovery unit, and performs data analysis based on the calculation node community correlation information from the community correlation discovery unit, and combines the above two types of data to process and complete the missing stream data from the data similarity calculation module, and outputs the completed stream data result to the spatiotemporal data alignment module.

[0055] The spatiotemporal data alignment module includes: a spatiotemporal model parsing unit, a data time dimension alignment unit and a data spatiotemporal dimension alignment unit. Among them: the spatiotemporal model parsing unit performs spatiotemporal model parsing processing according to the data from the data exchange module and outputs the obtained time dimension alignment mode and space dimension alignment mode results to the data time dimension alignment unit and the data space dimension alignment unit respectively; the data time dimension alignment unit performs time dimension alignment processing according to the time dimension alignment mode data from the spatiotemporal model parsing unit and the stream data from the data completion module and outputs the obtained time dimension aligned stream data results to the data space dimension alignment unit; the data space dimension alignment unit performs space dimension alignment processing according to the space dimension alignment mode data from the spatiotemporal model parsing unit and the stream data from the data time dimension alignment unit and outputs the obtained space dimension aligned stream data results to the data information processing unit of the data matching module.

[0056] This embodiment specifically works in the following manner:

[0057] 1) The management cloud platform uses the service to call the RESTful interface to pass the basic business information such as the BPMN or BPEL file representing the business process model, the CSV file representing the topological relationship of the computing nodes, the data fusion rules, the computing node space information and the data docking method to the data exchange module of the streaming data acquisition system. This module uses the data docking JPA interface to store this information in the infrastructure layer to complete the initial configuration of the operation mode of this system.

[0058] 2) The stream data producer, that is, the IoT device pushes the data to the data collection module in the data collector according to the configured business information. This module caches the collected stream data in the data pipeline, and uses the data matching module to perform data field semantic matching on the stream data with unclear sources, so as to infer the source of the data and mark the source of the stream data. At the same time, the data receiver uses the real-time information push Websocket interface to push the real-time operation status of the current data pipeline to the management cloud platform, and the real-time data display module visualizes the received data.

[0059] 3) The data processor uses the data similarity calculation module to calculate the similarity between the received data and the existing data, thereby increasing or decreasing the positive or negative correlation between the computing nodes. At the same time, the data completion module comprehensively utilizes the time series information of the time series data itself and the community information between the computing nodes to infer and complete the missing data, and uses the predefined time and space model to align the data in a specified format.

[0060] 4) The data distributor uniformly encapsulates and packages the data processed by the data processor according to the predefined data requirements of the computing nodes, and transmits the data packets to the subsequent computing nodes in the form of data packets according to the requirements of the computing nodes in an active or passive manner by the data distribution module, thereby completing the docking and transmission of data.

[0061] The comparison of the technical indicators of the above work with the technical parameters of similar products at home and abroad is shown in Table 1.

[0062] Table 1 Comparison of technical characteristics

[0063]

[0064]

[0065] From the perspective of system functions, this system can collect streaming data with different sources, different data structures, and disordered data time series in the IoT edge computing scenarios, and use the topological relationship of the computing nodes and the similarity between the time series data to automatically discover the community relationship of the computing nodes; at the same time, it further uses the community relationship between the computing nodes and the time series relationship of the data itself to complete the collected incomplete time series data; and then, combined with the computing needs of subsequent nodes, the completed and aligned streaming data is packaged on demand and transmitted to subsequent computing nodes in the form of data packets.

[0066] From the perspective of adaptability, this system allows users to configure the data requirement rules of subsequent nodes, and can perform data fusion and packaging operations on the collected data according to the configured rules, so that subsequent computing nodes can unpack and further calculate the received data. At the same time, this system provides a variety of data docking solutions, and allows different computing nodes to configure access to this system, so that they can be docked with this system according to the computing needs of the computing nodes themselves. Therefore, subsequent computing nodes do not need to develop new docking modules and data analysis modules specifically for the data acquisition system, but only need to care about the processing needs of actual business, and have high adaptability.

[0067] From the perspective of environmental adaptability, this system provides a configurable way to set different computing node topologies. During the process of streaming data collection, this system can use the configured topology to perform automatic community discovery of computing nodes based on streaming data similarity. At the same time, this system can use the similarity information of time series data to further adjust and modify the configured node relationship topology. Therefore, different topology configurations and different data collection results can allow this system to exhibit different system characteristics, so that it can be oriented to different edge computing environments, with high environmental adaptability.

[0068] In terms of temporal relationship adaptability, the system uses generative adversarial networks to automatically learn and generate discriminative networks and generative networks, so that it can adapt to different data temporal characteristics. And it can generate temporal data with specific temporal information based on the adaptation results, that is, the generative network, so as to complete the missing data. Using this method, the system can automatically adapt to different types of data temporal information, and the temporal relationship adaptability is high.

[0069] Compared with the prior art, the adaptability, environmental adaptability and time series relationship adaptability of this system are higher. Since this system allows users to configure the data requirement rules of subsequent nodes, and this system provides a variety of data docking solutions, the adaptability of this system is higher than that of the prior art. Since this system provides a configurable way to set different computing node topologies, and this system can use the similarity information of time series data to further adjust and modify the configured node relationship topology to adapt to different edge computing environments, the environmental adaptability of this system is higher than that of the prior art. Since this system uses a generative adversarial network to learn the time series features of streaming data, the time series relationship adaptability of this system is higher than that of existing related methods.

[0070] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A stream data collection system based on edge node community network, characterized in that: include: A data exchange module, a data acquisition module constituting a data receiver, a data matching module, a data similarity calculation module constituting a data processor, a data completion module, a spatiotemporal data alignment module, a data packet encapsulation module constituting a data distributor, and a data packet distribution module, wherein: the data acquisition module is connected to the data source of the heterogeneous stream data, and caches and uniformly manages the input stream data through a data pipeline; the data matching module generates metadata describing the data from the data acquisition module according to its source as a tag for matching the stream data and its data source computing node, and outputs the labeled data to the data similarity calculation module; the data similarity calculation module calculates the data similarity within the time window for the stream data from the data matching module and performs community connection enhancement processing; the data completion module calculates the community connection of the data source node from the data exchange module and the timing information of the data itself from the data similarity calculation module. Infer and complete the missing data in the stream data; the spatiotemporal data alignment module analyzes the spatiotemporal model, and aligns the time dimension and space dimension of the completed data from the data completion module according to the analysis results; the data packet encapsulation module reads and analyzes the fusion rules from the data exchange module and performs corresponding fusion operations on the completed and aligned stream data from the data matching module, and blocks, merges and packages the fused data and refers to the data interface mapping table to pass the data packets formed by each type of stream data to the corresponding output data pipeline; the data packet distribution module receives the data packets encapsulated by the data packet encapsulation module through the data pipeline, and sends the data packets to the subsequent computing nodes as needed through the computing node mapping table; the data exchange module completes the unified encapsulation of external information resources, and provides a unified calling interface and interface access security mechanism to support the unified access and update of external information resources within the system; The data similarity degree is as follows: dividing the change trend of data into seven different modes, and using the idea of ​​differentiation to divide the continuous stream data into sub-stream data, calculating which of the seven modes the change trend of the sub-stream data belongs to, and replacing this section of data with the change mode; then using the similarity of the change mode to calculate the similarity between the stream data; The inference and completion of missing data in the streaming data refers to: based on the timing characteristics of the streaming data itself and the community information between the computing nodes, for the data missing at a certain moment, the specific data value is inferred from the two perspectives of the timing characteristics of the streaming data and the community information of the computing nodes, and the two results are combined in a weighted average manner to obtain the final inference result.

2. The stream data collection system based on edge node community network according to claim 1 is characterized in that: The data similarity is calculated in the following way: ① For the flow data related to time information, according to the change trend of the flow data, namely, accelerated decline, decline, decelerated decline, basically unchanged, decelerated rise, rise, accelerated rise, the pattern set M = {-3, -2, -1, 0, 1, 2, 3} is used to represent them respectively; ②Divide the two streaming data S1 = {(v 11 , t 11 ), (v 12 , t 12 ), …, (v 1n , t 1n )} and S2 = {(v 21 , t 21 ), (v 22 , t 22 ), …, (v 2m , t 2m )} into k segments respectively and independently calculate which one of the pattern sets M each segment belongs to: For each divided segment, calculate its curve slope k i , and judge that when k i < -th, the pattern m i of this segment ∈ {-3, -2, -1}, where: the threshold th of the slope > 0; then calculate the slope k i+1 of the subsequent segment, and calculate the change Δk i = k i+1 - k i . When Δk i < 0, the pattern of the i-th segment is calibrated as -3; when Δk i = 0, the pattern of the i-th segment is calibrated as -2; when Δk i > 0, the pattern of the i-th segment is calibrated as -1; when -th < k i < th, the pattern of the i-th segment is calibrated as 0; when k i > th, the calibration judgment is the same as the case of less than; ③After the calibration is completed, the marked data S1={(m 11 , t 11 ), (m 12 , t 12 ),…,(m 1k , t 1k )},S2={(m 21 , t 21 ), (m 22 , t 22 ),…,(m 2k , t 2k )}, and then calculate the forward distance between S1 and S2 in: t i is the time span of the ith mode, t N is the total time length; ④ Since there may be a high possibility of reverse correlation, that is, S1 and S2 are negatively correlated, the negative distance between S1 and S2 is further calculated. This is equivalent to taking the opposite number of each mode of S2 to calculate the forward distance and finally obtain the data similarity.

3. The stream data collection system based on edge node community network according to claim 1 is characterized in that: The community connection enhancement processing refers to: using the similarity of the latest received stream data, updating the previous community connection between nodes, that is, the connection and weight value of two computing nodes in the computing node topology map, specifically: reading the community information of this computing node according to the data source recorded in the previously marked data metadata, and enhancing the community connection between the data source nodes of similar data according to the previously calculated data similarity between the two, so that they become closer in the community relationship; Or weaken the community connection between data source nodes of dissimilar data, making them looser in community relations, so as to support subsequent community-based missing data reasoning work.

4. The stream data collection system based on edge node community network according to claim 3 is characterized in that: The enhancement or weakening mentioned above refers to: increasing / decreasing the absolute value of its weight value, but maintaining the positive or negative nature of its original weight; The inference and completion described herein include: data time series correlation discovery and completion, community correlation discovery and completion, and missing data reasoning.

5. The stream data collection system based on edge node community network according to claim 4 is characterized in that: The data time series correlation discovery and completion refers to: utilizing the time series characteristics of the stream data itself and using a generative adversarial network (GAN) to infer missing data at certain moments. The specific algorithm is: first, generate a random time series data z with the same time distribution as the input data, the value of the time series data satisfies the Gaussian distribution, and the average value is zero; then pass the random time series data z into the generative network in the trained generative adversarial network, and use the generative network to convert this random time series data into time series data G(z) with a similar time series relationship to the original data; then use the data in G(z) to complete the original data.

6. The stream data collection system based on edge node community network according to claim 5 is characterized in that: The generative adversarial network is trained in the following manner: the collected data x0 with missing information is passed into the generative adversarial network, and the generative network and the discriminative network in the generative adversarial network are trained with this data, wherein: the training goal of the generative network is to obtain a mapping relationship G, which maps the randomly distributed time series data z to the generated data x with similar time series information to the input data x0, that is, G(z)=z→x; the training goal of the discriminative network is to obtain a mapping relationship D, which maps the time series data x to a variable sim, the value of which indicates the degree of similarity between x and x0 in terms of the time series relationship, that is, D(x)=x→sim; The training further includes three loss functions, among which: reconstruction loss L r The distance between G(z) generated by the generative network and the original data x0; difference loss L d The temporal correlation variable generated by the discriminative network, namely L d =D(G(z)); comprehensive loss L a By L r and L d The generated adversarial network uses the comprehensive loss L during training. a And back propagation method, the generation network and the discrimination network are trained to obtain mappings G and D.

7. The stream data collection system based on edge node community network according to claim 4 is characterized in that: The community association discovery and completion mentioned above refers to: using the community information of the data source node marked in the metadata and the data generated at each moment by the nodes with similar community relationships to infer the missing data at certain moments, specifically: reading the current data source node, adjacent nodes and edge weights w representing the closeness of community relationships from the data exchange module ei ; Then divide the time interval from t0 to t1 with missing data into k segments, and divide the data generated by the neighboring nodes of the computing node with missing data from t0 to t1 into k segments. For each segment of data, calculate its corresponding data change pattern m i , and use |w ei | is used as the weight, and the weighted average is calculated to finally obtain the data change pattern m of the missing data in the i-th segment, that is, Wherein: n represents the number of adjacent computing nodes. Afterwards, using this variation pattern, the slope k=m*th of the missing data in this segment is obtained, and this slope is used to calculate the specific missing data in the i-th segment.

8. The stream data collection system based on edge node community network according to claim 4 is characterized in that: The missing data inference refers to: taking a weighted average of the inference results obtained by time series correlation and the inference results obtained by community correlation, inferring the final value of the missing data, and filling it into the original data stream as part of the data stream.

9. The stream data collection system based on edge node community network according to claim 1 is characterized in that: The space-time model refers to: a series of time series as alignment standards in the time dimension; a series of spatial position points as alignment standards in the space dimension.

10. The stream data collection system based on edge node community network according to claim 1, characterized in that: The fusion operation refers to: simple operations on multiple data fields and re-integration, specifically: simple operations on multiple different data fields of multiple stream data.

11. The stream data collection system based on edge node community network according to claim 1 is characterized in that: The said block division means: dividing the merged stream data into several sub-segments of data according to the specified time interval in the time dimension; The merging means: simply piecing together and integrating multiple stream data aligned in the time dimension after fusion; The packaging refers to: completing a unified packaging operation on the data after the block and merge are completed according to the segmentation boundaries obtained by the block and merge calculations, and outputting them in the form of data packets with a unified data format.

12. The stream data collection system based on edge node community network according to claim 1, characterized in that: The on-demand sending means that: the data packet distribution module maintains the mapping relationship between the subsequent computing nodes connected to the system and the data packet access requirements through the computing node mapping table, wherein: for the computing node that obtains data in the "push data" mode, that is, the system actively sends data to the subsequent computing node, the required data packet is pushed to the receiving interface of the designated computing node; for the computing node that obtains data in the "pull data" mode, that is, the subsequent computing node actively requests data from the system, the required data packet is cached in the data pipeline bound to it, waiting for it to actively consume the data.

13. A method for collecting stream data based on the system according to any one of claims 1 to 12, characterized in that: By setting up an IoT device access module and a stream data acquisition and processing module in the edge node, the missing data of the collected multi-source heterogeneous stream data can be inferred and supplemented through the community information between the data sources and the temporal relationship of the stream data itself, and the time-space alignment of the data can be performed on demand through the spatiotemporal model. The data can be fused, divided, merged, packaged and / or distributed through data fusion rules, and a variety of data docking methods can be provided for subsequent computing nodes, so that data collection can be spontaneously carried out on the edge node side without the need for unified data collection and calculation by the central node.

Images