An IoT Edge Algorithm Based on Multi-Source Heterogeneous Data Aggregation Technology

Abstract

The present invention discloses an Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology, comprising the following steps: S1. Obtain multi-source Internet of Things heterogeneous data and perform standardized aggregation processing; S2. Perform analysis processing to obtain a data set; S3. Perform event element extraction to obtain summary content; S4. Obtain training samples; S5. Calculate the flow session feature vectors and regression model prediction vectors of each training sample after shunting; S6. Obtain a trained target data extraction model; S7. Send it to the cloud database; S8. Send the label field to the cloud database for retrieval, and extract the target data after the retrieval is completed. The present invention optimizes the analysis and processing of a certain type or a certain field of data, enables the joint processing of multi-source service data, can extract data according to the configuration method, greatly reduces the data transmission to the cloud, saves a large amount of equipment and operation and maintenance personnel costs, and effectively improves the retrieval efficiency and accuracy of multi-source heterogeneous data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and specifically to an Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology. Background Art

[0002] The Internet of Things is an information carrier such as the Internet and traditional telecommunication networks, enabling a network where all ordinary objects capable of exercising independent functions can achieve interconnection and interoperability.

[0003] On the Internet of Things, everyone can apply electronic tags to connect real objects to the Internet, and their specific locations can be found on the Internet of Things. Through the Internet of Things, a central computer can be used to centrally manage and control machines, devices, and personnel, and can also remotely control household devices and cars, as well as search for locations and prevent items from being stolen, similar to an automated control system. At the same time, by collecting data on these small things, big data can finally be aggregated, including major social changes such as redesigning roads to reduce car accidents, urban renewal, disaster prediction, and epidemic control, to achieve the connection of things to things.

[0004] When processing data on the Internet of Things, a huge amount of data will be generated. If this data is aggregated together, it will be an astronomical figure. Performing real-time massive data analysis, storage, and extraction in the cloud poses a huge challenge to computing power and network bandwidth. Edge computing is particularly suitable for application scenarios with special service requirements such as low latency, high bandwidth, high reliability, massive connections, heterogeneous aggregation, and local security and privacy protection. Therefore, there is an urgent need for an Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology. Summary of the Invention

[0005] The purpose of the present invention is to provide an Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology. Based on the multi-source heterogeneous data aggregation technology established on the Internet of Things edge algorithm, the analysis and processing of a certain type or field of data are optimized, enabling the joint processing of multi-source service data. Data can be extracted according to the configuration method, greatly reducing data transmission to the cloud, saving a large amount of equipment and operation and maintenance personnel costs, effectively improving the retrieval efficiency and accuracy of multi-source heterogeneous data, and improving the integrity of the extracted data, so as to solve the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] An Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology, comprising the following steps:

[0008] S1. Obtain multi-source Internet of Things heterogeneous data, perform standardized aggregation processing on the multi-source Internet of Things heterogeneous data, and locally store the multi-source Internet of Things heterogeneous data;

[0009] S2. Analyze and process the multi-source IoT heterogeneous data after standardized aggregation to obtain a data set;

[0010] S3. Construct a convolutional neural network to extract event elements from the analyzed and processed data set to obtain summary content;

[0011] S4. Further analyze and process the summary content using a text similarity algorithm and a maximum margin correlation model to obtain multiple data of a certain type or multiple data containing a certain field as training samples;

[0012] S5. Shunt the training samples according to the label type, and calculate the flow session feature vector and the regression model prediction vector of each shunted training sample;

[0013] S6. Input the flow session feature vector, the regression model prediction vector, and the label of each training sample into a data extraction model for training to obtain a trained target data extraction model;

[0014] S7. Send the data and label group to the cloud database according to the data type;

[0015] S8. Receive the input data for retrieving a specific type or data containing a specific field, analyze and process the data for retrieving a specific type or specific field to extract the label field, send the label field to the cloud database for retrieval, and extract the target data after the retrieval is completed.

[0016] Preferably, as an IoT edge algorithm based on a multi-source heterogeneous data aggregation technology of the present invention, the method for analyzing and processing the multi-source IoT heterogeneous data after standardized aggregation in S2 is filtering, merging and deduplicating of the same type.

[0017] Preferably, as an IoT edge algorithm based on a multi-source heterogeneous data aggregation technology of the present invention, the specific steps of S3 are as follows:

[0018] S31. Traverse all samples of the data set, perform single-sentence segmentation and manual annotation on the samples to obtain a model data set D:

[0019]

[0020] where l j is the label of the text single sentence c j after sample segmentation, l j ∈{data of a certain type or data containing a certain field}, j = 1, 2,..., K, and K is the total number of single sentences in the model data set;

[0021] S32. Extract the feature vectors of the text single sentences in the model data set D to obtain a data set feature matrix F;

[0022] S33. Construct a convolutional neural network, denoted as TextCNN, where the network structure of TextCNN is a convolutional layer, a max-pooling layer, two fully connected layers, and a softmax layer;

[0023] S34. Randomly divide the feature F of the above model dataset into a training set, a test set, and a validation set according to the ratio of 4:2:1;

[0024] S35. Use the divided training set and validation set in step S34 to train the convolutional neural network TextCNN obtained in step S33 to obtain a trained network model Model;

[0025] S36. Use the model Model obtained in the above step S35 to extract summaries from the test set in step S34 to obtain a data set that only includes a certain type or contains a certain field, denoted as summary content.

[0026] Preferably, as an Internet of Things edge algorithm based on a multi-source heterogeneous data aggregation technology of the present invention, calculating the flow session feature vector and the regression model prediction vector of each training sample after shunting in S5 includes the following steps:

[0027] S51. Calculate the state transition table column of each training sample after shunting, perform spatial compression on the state transition table column of each training sample, divide the state transition table column into multiple non-overlapping subsets, and perform encoding operations on each subset using different alphabet recodings to obtain the encoding feature information of each subset;

[0028] S52. Merge the similar encoding feature information in the encoding feature information of each subset through the state transition edge corresponding to the label type to obtain the flow session feature vector of each training sample after shunting;

[0029] S53. Perform regression model analysis on the flow session feature vector of each training sample after shunting to obtain the regression model prediction vector of each training sample after shunting.

[0030] Preferably, as an Internet of Things edge algorithm based on a multi-source heterogeneous data aggregation technology of the present invention, merging the similar encoding feature information in the encoding feature information of each subset through the state transition edge corresponding to the label type in S52

[0031] to obtain the flow session feature vector of each training sample after shunting includes the following steps:

[0032] S521. Identify the state transition matrix in the encoding feature information of each subset through the state transition edge corresponding to the label type;

[0033] S522. Obtain the target state transition matrix with the same state transition parameters, and determine the coding feature information corresponding to the target state transition matrix as the similar coding feature information;

[0034] S523. Merge the similar coding feature information in the coding feature information of each subset to obtain the flow session feature vector of each training sample after splitting.

[0035] As an optimization of the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention, the cloud database in S7 also associates the received data with the labels in the label group, and the associated labels form clusters.

[0036] As an optimization of the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention, the cloud database in S7 also extracts classifications from the clusters to form a classification directory and sends the classification directory out.

[0037] As an optimization of the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention, the method of sending the label field to the cloud database for retrieval in S8 is to construct a vector set from the feature vectors extracted from the label field, and the feature vectors include keywords and feature weights.

[0038] As an optimization of the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention, the method of sending the label field to the cloud database for retrieval in S8 further includes:

[0039] Calculate the word frequency of the keyword through a preset formula, and the preset formula is:

[0040]

[0041] where L i is the word frequency, TF is the term frequency, and Ctotal is the total number of words;

[0042] Calculate the updated weight according to the word frequency and the preset part-of-speech weight;

[0043] Adjust the feature weight with the updated weight to obtain the optimized feature vector.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] Based on the multi-source heterogeneous data aggregation technology, the present invention is established on the Internet of Things edge algorithm, optimizes the analysis and processing of certain types or fields of data, enables the joint processing of multi-source service data, can extract data according to the configuration method, greatly reduces the data transmission to the cloud, saves a large amount of equipment and operation and maintenance personnel costs, effectively improves the retrieval efficiency and accuracy of multi-source heterogeneous data, and improves the integrity of the extracted data. Description of the Drawings

[0046] Figure 1 It is a schematic diagram of the Internet of Things edge algorithm process based on the multi-source heterogeneous data aggregation technology of the present invention;

[0047] Figure 2 It is a schematic diagram of the flow session feature vector and the regression model prediction vector of each training sample after the calculation diversion of the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention;

[0048] Figure 3 It is a schematic diagram of the process of merging similar encoded feature information in the encoded feature information of each subset through the state transition edges corresponding to the label types in the Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology of the present invention to obtain the flow session feature vector of each training sample after diversion. Detailed Embodiments

[0049] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0050] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] Embodiment

[0052] Please refer to Figures 1-3 , the present invention provides a technical solution:

[0053] An Internet of Things edge algorithm based on the multi-source heterogeneous data aggregation technology includes the following steps:

[0054] S1. Obtain multi-source Internet of Things heterogeneous data, perform standardized aggregation processing on the multi-source Internet of Things heterogeneous data, and locally store the multi-source Internet of Things heterogeneous data;

[0055] S2. Analyze and process the multi-source Internet of Things heterogeneous data after the standardized aggregation processing to obtain a data set;

[0056] S3. Construct a convolutional neural network to extract event elements from the analyzed and processed data set to obtain summary content;

[0057] S4. Further analyze and process the summary content using a text similarity algorithm and a maximum marginal correlation model to obtain multiple data of a certain category or multiple data containing a certain field as training samples;

[0058] S5. Divide the training samples according to the label type and calculate the flow session feature vector and regression model prediction vector of each training sample after diversion;

[0059] S6. Input the stream session feature vector, regression model prediction vector, and label of each training sample into the data extraction model for training to obtain a trained target data extraction model;

[0060] S7. Sending the data and tag group to the cloud database according to the data type;

[0061] S8. Receive input data for searching a specific type or containing specific fields, analyze and process the data for searching a specific type or containing specific fields, extract label fields, send the label fields to a cloud database for retrieval, and extract the target data after the retrieval is complete.

[0062] As a technical optimization solution of the present invention, the method of analyzing and processing the multi-source IoT heterogeneous data after standardized aggregation processing in S2 is filtering and merging the same type to remove duplicates.

[0063] As a technical optimization solution of the present invention, the S3 specifically includes the following steps:

[0064] S31. Traverse all samples in the dataset, segment and manually label the samples into single sentences, and obtain the model dataset D:

[0065]

[0066] where l j c is the text sentence after sample segmentation j Tags, l j ∈{data of a certain type or containing a certain field}, j = 1, 2, ..., K, K is the total number of single sentences in the model dataset;

[0067] S32. Extract the feature vector of a single sentence in the model dataset D to obtain a dataset feature matrix F;

[0068] S33. Build a convolutional neural network, denoted as TextCNN, where the TextCNN network structure consists of a convolutional layer, a max pooling layer, two fully connected layers, and a softmax layer.

[0069] S34. Randomly divide the model dataset F into a training set, a test set, and a validation set according to a ratio of 4:2:1;

[0070] S35. Use the training set and validation set divided in step S34 to train the convolutional neural network TextCNN obtained in step S33 to obtain a trained network model Model;

[0071] S36. Use the model Model obtained in the above step S35 to extract summaries from the test set in step S34 to obtain a data set that only includes a certain type or contains a certain field, denoted as summary content.

[0072] As a technical optimization solution of the present invention, calculating the flow session feature vector and regression model prediction vector of each training sample after shunting in S5 includes the following steps:

[0073] S51. Calculate the state transition table column of each training sample after shunting, perform spatial compression on the state transition table column of each training sample, divide the state transition table column into multiple non-overlapping subsets, and perform encoding operations on each subset using different alphabet recodings to obtain the encoding feature information of each subset;

[0074] S52. Merge the similar encoding feature information in the encoding feature information of each subset through the state transition edge corresponding to the label type to obtain the flow session feature vector of each training sample after shunting;

[0075] S53. Perform regression model analysis on the flow session feature vector of each training sample after shunting to obtain the regression model prediction vector of each training sample after shunting.

[0076] As a technical optimization solution of the present invention, merging the similar encoding feature information in the encoding feature information of each subset through the state transition edge corresponding to the label type in S52

[0077] to obtain the flow session feature vector of each training sample after shunting includes the following steps:

[0078] S521. Identify the state transition matrix in the encoding feature information of each subset through the state transition edge corresponding to the label type;

[0079] S522. Obtain the target state transition matrix with the same state transition parameters, and determine the encoding feature information corresponding to the target state transition matrix as the similar encoding feature information;

[0080] S523. Merge the similar encoding feature information in the encoding feature information of each subset to obtain the flow session feature vector of each training sample after shunting.

[0081] As a technical optimization solution of the present invention, the cloud database in S7 also associates the received data with the tags in the tag group, and the associated tags form a cluster.

[0082] As a technical optimization solution of the present invention, the cloud database in S7 also extracts classifications from the clusters to form a classification directory and sends the classification directory out.

[0083] As a technical optimization solution of the present invention, the method of sending the tag field to the cloud database for retrieval in S8 is to construct a vector set from the feature vectors extracted from the tag field, and the feature vectors include keywords and feature weights.

[0084] As a technical optimization solution of the present invention, the method of sending the tag field to the cloud database for retrieval in S8 also includes:

[0085] Calculating the word frequency of the keyword through a preset formula, and the preset formula is:

[0086]

[0087] where L i is the word frequency, TF is the term frequency, and Ctotal is the total number of words;

[0088] Calculating and updating the weight according to the word frequency and the preset part-of-speech weight;

[0089] Adjusting the feature weight with the updated weight to obtain an optimized feature vector.

[0090] In summary, the present invention is based on the multi-source heterogeneous data aggregation technology and is built on the Internet of Things edge algorithm, optimizing the analysis and processing of certain types or fields of data, enabling the joint processing of multi-source service data, extracting data according to the configuration method, greatly reducing the data transmission to the cloud, saving a large amount of equipment and operation and maintenance personnel costs, effectively improving the retrieval efficiency and accuracy of multi-source heterogeneous data, and improving the integrity of the extracted data.

[0091] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An Internet of Things edge algorithm based on multi-source heterogeneous data aggregation technology, characterized in that, It includes the following steps: S1. Obtain multi-source Internet of Things heterogeneous data, perform standardized aggregation processing on the multi-source Internet of Things heterogeneous data, and locally store the multi-source Internet of Things heterogeneous data; S2. Analyze and process the multi-source Internet of Things heterogeneous data after standardized aggregation processing to obtain a data set; S3. Construct a convolutional neural network to extract event elements from the analyzed data set to obtain summary content; S4. Further analyze and process the summary content using a text similarity algorithm and a maximum margin correlation model to obtain multiple data of a certain type or multiple data containing a certain field as training samples; S5. Shunt the training samples according to the label type, and calculate the flow session feature vector and regression model prediction vector of each training sample after shunting; S6. Input the flow session feature vector, regression model prediction vector, and label of each training sample into a data extraction model for training to obtain a trained target data extraction model; S7. Send the data and label group to the cloud database according to the data type; S8. Receive the input data for retrieving a specific type or data containing a specific field, analyze and process the data for retrieving a specific type or specific field, extract the label field, send the label field to the cloud database for retrieval, and extract the target data after retrieval; The method for analyzing and processing the multi-source Internet of Things heterogeneous data after standardized aggregation processing in S2 is filtering, merging and deduplicating of the same type; The specific steps of S3 include the following: S31. Traverse all samples in the data set, perform single-sentence segmentation and manual annotation on the samples to obtain a model data set D: where l j is the label of the single sentence c j after sample segmentation, and l j ∈ {a certain type or data containing a certain field}, j = 1, 2,..., K, where K is the total number of single sentences in the model dataset; S32. Extract the feature vectors of the text single sentences in the model data set D to obtain a data set feature matrix F; S33. Construct a convolutional neural network, denoted as TextCNN, where the TextCNN network structure is a convolutional layer, a max-pooling layer, two fully connected layers, and a softmax layer; S34. Randomly divide the above model data set feature F into a training set, a test set, and a validation set according to a ratio of 4:2:1; S35. Use the training set and validation set divided in step S34 to train the convolutional neural network TextCNN obtained in step S33 to obtain a trained network model Model; S36. Use the model Model obtained in the above step S35 to perform summary extraction on the test set in step S34 to obtain a data set that only includes a certain type or data containing a certain field, denoted as summary content; The steps for calculating the flow session feature vector and regression model prediction vector of each training sample after shunting in S5 include the following: S51. Calculate the state transition table column of each training sample after shunting, perform spatial compression on the state transition table column of each training sample, divide the state transition table column into multiple non-overlapping subsets, and perform encoding operations on each subset using different alphabet re-encodings to obtain the encoded feature information of each subset; S52. Merge the similar encoded feature information in the encoded feature information of each subset through the state transition edges corresponding to the label type to obtain the flow session feature vector of each training sample after shunting; S53. Perform regression model analysis on the flow session feature vectors of each training sample after shunting to obtain the regression model prediction vectors of each training sample after shunting; in S52, the similar encoded feature information in the encoded feature information of each subset is merged through the state transition edges corresponding to the label types, and obtaining the flow session feature vectors of each training sample after shunting includes the following steps: S521. Identify the state transition matrix in the encoded feature information of each subset through the state transition edges corresponding to the label types; S522. Obtain the target state transition matrix with the same state transition parameters, and determine the encoded feature information corresponding to the target state transition matrix as the similar encoded feature information; S523. Merge the similar encoded feature information in the encoded feature information of each subset to obtain the flow session feature vectors of each training sample after shunting; the cloud database in S7 also associates the received data with the labels in the label group, and the associated labels form clusters; the cloud database in S7 also extracts classifications from the clusters to form a classification directory and sends the classification directory; the method of sending the label field to the cloud database for retrieval in S8 is to construct a vector set from the feature vectors extracted from the label field, and the feature vectors include keywords and feature weights; the method of sending the label field to the cloud database for retrieval in S8 also includes: Calculating the word frequency of the keyword through a preset formula, and the preset formula is: Calculating the updated weight according to the word frequency and the preset part-of-speech weight; Among them, L i is the word frequency, TF is the term frequency, and Ctotal is the total number of words; Adjusting the feature weight with the updated weight to obtain the optimized feature vector. ​

Images