Sensor data processing method and system based on big data

Abstract

The invention provides a sensor data processing method and system based on big data, and belongs to the technical field of data processing, in the method, abnormal data in an original data set uploaded by a target sensor cluster can be automatically identified and repaired under the condition of determining that the target sensor cluster meets the security condition, and the security of the target sensor cluster is improved. According to the method, a Merkel tree is established, a target data set subjected to data restoration is obtained, the integrity and accuracy of the data are ensured, and meanwhile, newly-added sensor data in each target data set can be stored to a proper target leaf node in the Merkel tree according to the updating moment of the sensor data in each target data set; according to the method, the storage overhead is effectively reduced while orderly storage of the newly-added sensor data is realized, and by performing consensus on the newly-added sensor data in the block chain, illegal control and illegal data tampering of the sensor can be effectively avoided, so that the security of data storage is ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a sensor data processing method and system based on big data. Background Art

[0002] In the field of modern industrial manufacturing, the demand for intelligent production management is increasing day by day, and production management is inseparable from the effective monitoring of industrial big data of each production node in the production system through various sensors. Sensors are detection devices that can sense the measured information and transform the sensed information into electrical signals or other required forms of information output according to certain rules to meet the requirements of information transmission, processing, storage, display, recording and control.

[0003] At present, sensors installed at production sites usually directly transmit the sensed data to the background system, which analyzes and operates the sensor data. However, due to interference from the on-site environment and the increase in sensor usage time, the sensor test accuracy may decrease, making it difficult to effectively identify abnormal data. At the same time, when the types and number of sensors are large, a large amount of sensor data also has greater security risks such as data leakage and data tampering. Summary of the invention

[0004] The present application provides a sensor data processing method and system based on big data. The present application adopts the following technical solutions:

[0005] In a first aspect, a sensor data processing method based on big data is provided, the method comprising:

[0006] Based on the risk characteristic data of the target sensor cluster, determining whether the target sensor cluster meets a preset safety condition;

[0007] In the case where it is determined that the target sensor cluster satisfies the safety condition, obtaining an original data set of each node sensor based on sensor data collected by multiple node sensors in the target sensor cluster within a preset period and acquired in real time;

[0008] For any of the original data sets, perform anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set; perform data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtain a target data set based on the repaired data set and the normal data set;

[0009] For any of the target data sets, determining an update time of the sensor data based on time characteristic data of the sensor data in the target data set;

[0010] Determine, based on the update time of the sensor data in each target data set, a target leaf node of each target data set in a pre-constructed Merkle tree;

[0011] When new sensor data appears in any target data set, a hash value corresponding to the new sensor data is written into a target leaf node corresponding to the target data set to update a target block containing the Merkle tree;

[0012] The updated target block is broadcasted in the blockchain to achieve consensus on the newly added sensor data.

[0013] In an embodiment of the present application, the step of determining whether the target sensor cluster meets a preset safety condition based on the risk characteristic data of the target sensor cluster includes:

[0014] Acquire device risk characteristic data and communication risk characteristic data of multiple node sensors in the target sensor cluster;

[0015] Determining a comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data;

[0016] When the comprehensive risk value is less than the risk threshold, it is determined that the target sensor cluster meets the safety condition.

[0017] In an embodiment of the present application, the step of determining the comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data includes:

[0018] Determining a first risk value based on the equipment risk characteristic data and the first risk weight;

[0019] Determining a second risk value based on the communication risk characteristic data and the second risk weight;

[0020] The comprehensive risk value is determined based on the first risk value and the second risk value.

[0021] In one embodiment of the present application, the step of performing anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set includes:

[0022] Inputting the sensor data in the original data set into a plurality of basic models respectively to obtain the abnormality determination results outputted by the plurality of basic models respectively;

[0023] Obtaining weighted meta-features based on the abnormality determination results output by each of the multiple basic models and the weight parameters corresponding to each of the basic models;

[0024] The weighted meta-features are output to a meta-model to obtain a normal data set and an abnormal data set output by the meta-model.

[0025] In one embodiment of the present application, the plurality of said basic models include statistical models and deep learning models;

[0026] The step of inputting the sensor data in the original data set into a plurality of basic models respectively to obtain the abnormality determination results outputted by each of the plurality of basic models comprises:

[0027] The sensor data in the multiple original data sets are respectively input into the statistical model and the deep learning model to obtain a first abnormality determination result output by the statistical model and a second abnormality determination result output by the deep learning model.

[0028] In one embodiment of the present application, the first abnormality determination result is used to characterize the abnormality score of the sensor data, and the second abnormality determination result is a binary classification result;

[0029] The step of obtaining weighted meta-features based on the abnormality determination results outputted by each of the multiple basic models and the weight parameters corresponding to each of the basic models comprises:

[0030] Performing standardization processing on the first abnormality determination result to obtain a standardized determination result;

[0031] Obtaining a first meta-feature based on the standardized determination result and a first weight parameter corresponding to the statistical model;

[0032] Obtaining a second meta-feature based on the second abnormality determination result and a second weight parameter corresponding to the deep learning model;

[0033] The first meta-feature and the second meta-feature are combined to obtain the weighted meta-feature.

[0034] In one embodiment of the present application, the time characteristic data includes the historical update time and the historical update frequency of the sensor data;

[0035] The step of determining the update time of the sensor data based on the time characteristic data of the sensor data in the target data set comprises:

[0036] The historical update time and the historical update frequency are input into a prediction model, and the update time of the sensor data is output.

[0037] In one embodiment of the present application, based on the update time of the sensor data in each target data set, the step of determining the target leaf node of each target data set in the pre-constructed Merkle tree includes:

[0038] Determine two target data sets whose update time difference is less than a time threshold as associated data sets, and determine data sets other than the associated data sets among the plurality of target data sets as independent data sets;

[0039] For any associated data set, two leaf nodes under the same subtree in the Merkle tree are respectively determined as two target leaf nodes of the associated data set;

[0040] For any independent data set, a free leaf node is randomly selected from the free leaf nodes in the Merkle tree to be determined as the target leaf node of the independent data set.

[0041] In one embodiment of the present application, when new sensor data appears in any target data set, the step of writing a hash value corresponding to the new sensor data into a target leaf node corresponding to the target data set to update a target block containing the Merkle tree includes:

[0042] In the case that there is newly added sensor data in both target data sets in any of the associated data sets, respectively generating first hash values ​​and second hash values ​​corresponding to the newly added sensor data in the two target data sets;

[0043] The first hash value and the second hash value are respectively written into two target leaf nodes corresponding to the associated data set to update the target block containing the Merkle tree.

[0044] In a second aspect, based on the same inventive concept, a sensor data processing system based on big data is provided, the system comprising:

[0045] A condition determination module, used to determine whether the target sensor cluster meets a preset safety condition based on the risk characteristic data of the target sensor cluster;

[0046] A data acquisition module, configured to obtain an original data set of each node sensor based on sensor data collected by multiple node sensors in the target sensor cluster within a preset period acquired in real time when it is determined that the target sensor cluster meets the safety condition;

[0047] A data repair module, for performing anomaly detection on sensor data in any of the original data sets to obtain a normal data set and an abnormal data set; performing data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtaining a target data set based on the repaired data set and the normal data set;

[0048] A time determination module, for determining, for any of the target data sets, an update time of the sensor data based on time characteristic data of the sensor data in the target data set;

[0049] A node determination module is used to determine the target leaf node of each target data set in the pre-constructed Merkle tree based on the update time of the sensor data in each target data set; wherein,

[0050] A block update module, configured to, when new sensor data appears in any target data set, write a hash value corresponding to the new sensor data into a target leaf node corresponding to the target data set, so as to update a target block containing the Merkle tree;

[0051] The data consensus module is used to broadcast the updated target block in the blockchain to achieve consensus on the newly added sensor data.

[0052] In an embodiment of the present application, the condition determination module includes:

[0053] A risk characteristic data acquisition submodule is used to acquire device risk characteristic data and communication risk characteristic data of multiple node sensors in a target sensor cluster;

[0054] A comprehensive risk value determination submodule, configured to determine a comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data;

[0055] The safety condition determination submodule is used to determine that the target sensor cluster meets the safety condition when the comprehensive risk value is less than a risk threshold.

[0056] In one embodiment of the present application, the comprehensive risk value determination submodule includes:

[0057] A first risk value determining unit, configured to determine a first risk value based on the equipment risk feature data and a first risk weight;

[0058] A second risk value unit, configured to determine a second risk value based on the communication risk feature data and a second risk weight;

[0059] A comprehensive risk value determining unit is used to determine the comprehensive risk value based on the first risk value and the second risk value.

[0060] In one embodiment of the present application, the data repair module includes:

[0061] A determination result acquisition submodule, used to input the sensor data in the original data set into multiple basic models respectively, and obtain the abnormal determination results output by each of the multiple basic models;

[0062] A weighted meta-feature determination submodule, used to obtain a weighted meta-feature based on the abnormality determination results output by each of the multiple basic models and the weight parameters corresponding to each of the basic models;

[0063] The data set acquisition submodule is used to output the weighted meta-features to the meta-model to obtain a normal data set and an abnormal data set output by the meta-model.

[0064] In one embodiment of the present application, the plurality of said basic models include statistical models and deep learning models;

[0065] The determination result acquisition submodule is specifically used to input the sensor data in the multiple original data sets into the statistical model and the deep learning model respectively, to obtain the first abnormality determination result output by the statistical model and the second abnormality determination result output by the deep learning model.

[0066] In one embodiment of the present application, the first abnormality determination result is used to characterize the abnormality score of the sensor data, and the second abnormality determination result is a binary classification result;

[0067] The weighted meta-feature determination submodule includes:

[0068] A standardization unit, used for performing standardization processing on the first abnormality determination result to obtain a standardized determination result;

[0069] A first meta-feature determining unit, configured to obtain a first meta-feature based on the standardized determination result and a first weight parameter corresponding to the statistical model;

[0070] A second meta-feature determining unit, configured to obtain a second meta-feature based on the second abnormality determination result and a second weight parameter corresponding to the deep learning model;

[0071] The weighted meta-feature unit is used to perform feature combination on the first meta-feature and the second meta-feature to obtain the weighted meta-feature.

[0072] In one embodiment of the present application, the time characteristic data includes the historical update time and the historical update frequency of the sensor data; the time determination module includes:

[0073] The prediction submodule is used to input the historical update time and the historical update frequency into the prediction model, and output the update time of the sensor data.

[0074] In an embodiment of the present application, the node determination module includes:

[0075] A data set division submodule, used to determine two target data sets whose update time difference is less than a time threshold as associated data sets, and determine data sets other than the associated data sets among the multiple target data sets as independent data sets;

[0076] A first node determination submodule is used to determine, for any associated data set, two leaf nodes under the same subtree in the Merkle tree as two target leaf nodes of the associated data set respectively;

[0077] The second node determination submodule is used to randomly select an empty leaf node from the empty leaf nodes in the Merkle tree for any independent data set and determine it as a target leaf node of the independent data set.

[0078] In one embodiment of the present application, the block update module includes:

[0079] A hash value generation submodule, for respectively generating first hash values ​​and second hash values ​​corresponding to the newly added sensor data in the two target data sets when there is newly added sensor data in the two target data sets in any of the associated data sets;

[0080] The block update submodule is used to write the first hash value and the second hash value into two target leaf nodes corresponding to the associated data set respectively, so as to update the target block containing the Merkle tree.

[0081] In summary, the above-mentioned sensor data processing method and system based on big data have the following technical effects:

[0082] The embodiment of the present application first determines whether the target sensor cluster meets the preset safety conditions based on the risk characteristic data of the target sensor cluster. When it is determined that the target sensor cluster meets the safety conditions, the original data set of each node sensor can be obtained based on the sensor data collected by multiple node sensors in the target sensor cluster within a preset period in real time; then, by performing anomaly detection on the sensor data in the original data set, the sensor data in the abnormal data set can be repaired to obtain a target data set with more complete data, and the update time of the sensor data is determined based on the time characteristic data of the sensor data in each target data set, and then the target leaf node of each target data set in the pre-constructed Merkle tree is determined based on the update time of the sensor data in each target data set; finally, when new sensor data appears in any target data set, the hash value corresponding to the new sensor data is written into the target leaf node corresponding to the target data set to update the target block containing the Merkle tree, and the updated target block is broadcast in the blockchain to achieve consensus on the new sensor data. The embodiments of the present application can not only realize automatic identification and repair of abnormal data to ensure the integrity and accuracy of the data, but also store the newly added sensor data to the appropriate target leaf node in the Merkle tree according to the update time of the sensor data, thereby effectively reducing the storage overhead while realizing the orderly storage of the newly added sensor data, and by reaching a consensus on the newly added sensor data in the blockchain, it can effectively prevent the sensor from being illegally controlled and the data from being illegally tampered with, thereby ensuring the security of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic diagram of the steps of a sensor data processing method based on big data provided in an embodiment of the present application;

[0084] Figure 2 A schematic diagram of the functional modules of a sensor data processing system based on big data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0086] Reference Figure 1 , an embodiment of the present application provides a sensor data processing method based on big data, the method may include the following steps:

[0087] S101: Determine whether the target sensor cluster meets a preset safety condition based on risk feature data of the target sensor cluster.

[0088] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, or an electronic device with the above functions. This implementation will be described using a cloud server as an example.

[0089] In this embodiment, the cloud server is communicatively connected with the target sensor cluster. After completing the authentication on the cloud server, the target sensor cluster can transmit the sensor data to the cloud server.

[0090] In this embodiment, to ensure the security and validity of sensor data, the cloud server will first determine whether the target sensor cluster meets the preset security conditions based on the risk characteristic data of the target sensor cluster before acquiring the sensor data. The security conditions are used to indicate whether the target sensor cluster can transmit sensor data to the cloud server.

[0091] In a specific implementation, S101 may specifically include the following sub-steps:

[0092] S101 - 1 : Acquire device risk characteristic data and communication risk characteristic data of multiple node sensors in a target sensor cluster.

[0093] In this embodiment, considering that the risks of sensors usually exist in the device risks existing in the device itself and the communication risks existing in the network transmission, the cloud server will collect the device risk characteristic data and communication risk characteristic data of multiple node sensors in the target sensor cluster for comprehensive judgment.

[0094] S101 - 2: Determine a comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data.

[0095] In this embodiment, the device risk feature data may include device hardware risk value, device software risk value, and fault risk value, etc.; the communication risk feature data may include gateway device risk value, network security risk value, and transmission mode risk value, etc.

[0096] It should be noted that the device hardware risk value can be determined based on the hardware attributes of the sensor (such as the device usage time, anti-environmental interference ability, etc.); the device software risk value can be determined based on the software security of the sensor (such as software compatibility, whether there is an encryption mechanism, communication protocol type, etc.); the fault risk value can be determined based on the historical fault data of the sensor (such as fault type, fault frequency, etc.); the gateway device risk value represents the security risk of the gateway device connected to the sensor; the network security risk value represents the security risk of the communication network used by the sensor; the transmission mode risk value represents the security risk of the data transmission method used by the sensor. Among them, the transmission mode includes wireless transmission and wired transmission, and the transmission mode risk value of wired transmission is lower than the transmission mode risk value of wireless transmission.

[0097] In a specific implementation, corresponding risk weights can also be configured for the device risk feature data and the communication risk feature data according to actual needs. A first risk weight can be configured for the device risk feature data, and a second risk weight can be configured for the communication risk feature data, and then a first risk value can be determined based on the device risk feature data and the first risk weight; a second risk value can be determined based on the communication risk feature data and the second risk weight; and a comprehensive risk value can be determined based on the first risk value and the second risk value. The sum of the first risk weight and the second risk weight is 1.

[0098] In this embodiment, the first risk weight and the second risk weight can be adjusted according to the enterprise's security requirements for sensor data. For example, for a communication enterprise, the security requirements for network communication are high, so a higher second risk weight can be set for communication risk feature data; for a manufacturing enterprise with a relatively harsh on-site environment, the security requirements for the equipment itself are high, so a higher first risk weight can be set for the equipment risk feature data.

[0099] S101 - 3 : When the comprehensive risk value is less than the risk threshold, determine that the target sensor cluster meets the safety condition.

[0100] It should be noted that the higher the comprehensive risk value, the higher the overall risk of the sensor.

[0101] In this embodiment, by setting a risk threshold, the sensor data can be transmitted to the cloud server for subsequent data analysis and storage under the premise of ensuring that the sensor risk is low.

[0102] In this embodiment, if the cloud server detects that the comprehensive risk value is greater than or equal to the risk threshold, it can send an alarm prompt to the management terminal of the sensor to prompt the management personnel of the risk items existing in the target sensor cluster.

[0103] S102: When it is determined that the target sensor cluster meets the safety condition, based on sensor data collected by multiple node sensors in the target sensor cluster within a preset period and acquired in real time, an original data set of each node sensor is obtained.

[0104] In this embodiment, after determining that the target sensor cluster meets the security conditions, the cloud server will obtain in real time the sensor data collected by multiple node sensors in the target sensor cluster within a preset period. By storing the sensor data collected within the preset period separately, the original data set of each node sensor can be obtained.

[0105] S103: For any original data set, perform anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set; perform data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtain a target data set based on the repaired data set and the normal data set.

[0106] In this embodiment, considering that there may be abnormal sensor data in the original data set, the cloud server will perform abnormality identification on each original data set to obtain a normal data set and an abnormal data set. The normal data set is used to store the normal sensor data in the original data set, and the abnormal data set is used to store the abnormal sensor data in the original data set.

[0107] In this embodiment, compared with removing abnormal sensor data, by repairing the sensor data in the abnormal data set, the integrity of the sensor data can be ensured, avoiding the loss of key data or incomplete signals that lead to inaccurate subsequent analysis.

[0108] In a specific implementation, the abnormal sensor data in the abnormal data set can be deleted in the original data set to obtain data spaces, and then for any data space, a preset number of normal sensor data can be extracted before and after the data space to obtain extracted data, and the average value of the extracted data can be used as the repair data corresponding to the abnormal sensor data, and then based on the repair data corresponding to all the abnormal sensor data, a repair data set can be obtained; by filling all the repair data into the corresponding data spaces, the target data set can be obtained.

[0109] S104: For any target data set, based on the time characteristic data of the sensor data in the target data set, determine the update time of the sensor data.

[0110] In this embodiment, after obtaining the target data set, the cloud server will obtain the time characteristic data of the sensor data in the target data set, and then predict the update time of the sensor data in the target data set based on the time characteristic data.

[0111] In a specific implementation, the time feature data includes the historical update time and the historical update frequency of the sensor data. The historical update time and the historical update frequency can be input into the prediction model to output the update time of the sensor data.

[0112] In this embodiment, the update regularity of the sensor data can be learned based on the historical update time and historical update frequency of the sensor data. Specifically, for sensor data with a relatively fixed update frequency, such as sensors that need to regularly check the operating status of the device, the time of the next check can be determined based on its corresponding check interval; for sensor data with a relatively frequent update frequency, such as sensors that need to check the operating status of the device in real time, a prediction model can be used to perform statistical analysis based on the sensor data collected within a preset period to determine the historical update time and historical update frequency.

[0113] In the specific implementation, the prediction model can be trained based on the regression algorithm or the Long Short-Term Memory (LSTM) network according to the existing historical sensor data to predict the update time of the sensor data. And as time goes by and the historical sensor data increases, the prediction results of the prediction model will be more accurate.

[0114] S105: Determine a target leaf node of each target data set in a pre-constructed Merkle tree based on the update time of the sensor data in each target data set.

[0115] It should be noted that the Merkle Tree, also known as the Hash Tree, is a binary tree data structure used to effectively verify and manage the integrity of large data sets. The construction of the Merkle Tree is based on the hash function, which generates hash values ​​by dividing the data into blocks of fixed size and applying the hash function to each block. These hash values ​​are then organized into a tree structure, in which the value of each non-leaf node is a combination of the hash values ​​of its child node values. This process is repeated until a root node, also known as the Merkle Root, is generated. The characteristic of the Merkle Tree is that any change to the underlying node will be passed to the parent node all the way to the root of the tree, and it is easy to locate the leaf node that has changed, so it is particularly suitable for fast, effective and secure verification of data existence and integrity. The use of the Merkle Tree increases the difficulty of data tampering and can achieve effective recording of the temporal changes of digital objects. Among them, a non-leaf node can be regarded as a tree node, and the two child nodes under a tree node can be regarded as a leaf node.

[0116] In this embodiment, after determining the update time of the sensor data in each target data set, the cloud server can match a suitable target leaf node for each target data set in the pre-built Merkle tree according to the update time, and the target leaf node is used to store the sensor data in the corresponding target data set. In this way, the storage overhead can be effectively reduced while achieving orderly storage of the newly added sensor data.

[0117] In a specific implementation, S105 may specifically include the following sub-steps:

[0118] S105 - 1 : Determine two target data sets whose update time difference is less than a time threshold as associated data sets, and determine data sets other than the associated data sets among the multiple target data sets as independent data sets.

[0119] In this embodiment, considering that any update of any leaf node will cause the hash value from the leaf node to the root node to be updated, therefore, in order to reduce the update frequency of the Merkle tree and thus reduce storage overhead, multiple target data sets will be paired based on the update time, and the two successfully paired target data sets will be associated to obtain an associated data set; the remaining unpaired data sets are independent data sets.

[0120] It should be noted that an associated dataset refers to two target datasets with the same or close update times; an independent dataset refers to a dataset with relatively independent update times, that is, the difference between its update time and the update time of other datasets is greater than or equal to the time threshold.

[0121] S105-2: For any associated data set, two leaf nodes under the same subtree in the Merkle tree are respectively determined as two target leaf nodes of the associated data set.

[0122] In this embodiment, by corresponding the two target data sets in the associated data set to two leaf nodes under the same subtree respectively, the newly added sensor data in the two target data sets can be synchronously updated at the same time, that is, the storage of the corresponding two newly added sensor data can be completed by updating the Merkle tree once.

[0123] Furthermore, for the associated data sets, the average update time of each associated data set can be determined, and then each associated data set can be sorted according to the average update time to obtain a data set sequence, and the two target leaf nodes corresponding to each associated data set can be determined in turn in the order indicated by the data set sequence to achieve orderly storage of all associated data sets.

[0124] S105-3: For any independent data set, randomly select an empty leaf node from the empty leaf nodes in the Merkle tree to determine it as the target leaf node of the independent data set.

[0125] In this embodiment, since the update time of each independent data set is relatively independent, after matching the corresponding target leaf node for the associated data set, a free leaf node can be randomly selected for each independent data set to determine as the target leaf node corresponding to the independent data set.

[0126] S106: When new sensor data appears in any target data set, a hash value corresponding to the new sensor data is written into a target leaf node corresponding to the target data set to update a target block containing a Merkle tree.

[0127] It should be noted that the target block includes a block header and a block body, wherein the block body includes a Merkle tree, the block header includes the root of the Merkle tree and the hash pointer of the previous block, and multiple target blocks are linked into a blockchain through hash pointers.

[0128] In this embodiment, since the target leaf node is used to store the sensor data of the corresponding target data set, whenever new sensor data appears in the target data set, the hash value corresponding to the new sensor data is written into the target leaf node corresponding to the target data set, and it can be updated upward layer by layer until the root node is updated.

[0129] In a specific implementation, in order to reduce the number of hash calculations for the associated data set and the corresponding storage overhead, S106 may specifically include the following sub-steps:

[0130] S106 - 1 : When there is newly added sensor data in both target data sets in any associated data set, first hash values ​​and second hash values ​​corresponding to the newly added sensor data in the two target data sets are generated respectively.

[0131] In this embodiment, if new sensor data appears in both target data sets in the associated data set at the same time, the first hash value and the second hash value corresponding to the two new sensor data can be generated immediately; if new sensor data appears in only one of the target data sets in the associated data set, its update is delayed until the new sensor data appears in the other target data set in the associated data set, and then the first hash value and the second hash value corresponding to the new sensor data in the two target data sets are generated respectively.

[0132] It should be noted that, since the difference between the update times of the two target data sets in the associated data set is small, the number of hash calculations and the corresponding storage overhead for the associated data set can be effectively reduced while ensuring the timeliness of data update.

[0133] S106-2: Write the first hash value and the second hash value into two target leaf nodes corresponding to the associated data set respectively, so as to update the target block containing the Merkle tree.

[0134] In this embodiment, after the first hash value and the second hash value are respectively written into the two target leaf nodes corresponding to the associated data set, the third hash value of the tree node corresponding to the two target leaf nodes can be calculated based on the first hash value and the second hash value, and then the tree node of the tree node is updated according to the third hash value of the tree node until the update of the root node is completed to obtain the reconstructed Merkle tree. The target block can be updated by writing the root node of the Merkle tree into the block header of the target block.

[0135] It should be noted that the tree node is one of the branches of the Merkle tree, and the two leaf nodes under the same tree node correspond to the same parent node, which means that when new sensor data appears in the two target data sets in the associated data set at the same time, only one path from the two leaf nodes to the tree root will be changed. In this way, it can effectively avoid the target data sets being scattered under different tree nodes and causing changes in multiple paths, thereby effectively reducing the number of hash calculations and the corresponding storage overhead, shortening the reconstruction time of the Merkle tree, and thus improving the timeliness of sensor data.

[0136] S107: Broadcast the updated target block in the blockchain to achieve consensus on the newly added sensor data.

[0137] In this embodiment, the broadcast target block is specifically a broadcast means in blockchain technology, that is, when the target block is broadcast, other blocks in the entire blockchain will record and store the newly added sensor data, and package the recorded newly added sensor data into blocks to achieve consensus on the newly added sensor data.

[0138] The embodiments of the present application can not only realize automatic identification and repair of abnormal data to ensure the integrity and accuracy of the data, but also store the newly added sensor data to the appropriate target leaf node in the Merkle tree according to the update time of the sensor data, thereby effectively reducing the storage overhead while realizing the orderly storage of the newly added sensor data, and by reaching a consensus on the newly added sensor data in the blockchain, it can effectively prevent the sensor from being illegally controlled and the data from being illegally tampered with, thereby ensuring the security of data storage.

[0139] In a feasible implementation, the step of performing anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set in S103 may include the following sub-steps:

[0140] S103 - 1 : Input the sensor data in the original data set into multiple basic models respectively, and obtain the abnormality determination results output by each of the multiple basic models.

[0141] In this embodiment, the sensor data in the multiple original data sets are identified by a preset comprehensive model, and the comprehensive model includes multiple basic models and a meta-model, and the output ends of the multiple basic models are connected to the input end of the meta-model.

[0142] In this embodiment, by performing anomaly detection on the number of sensors in the original data set through multiple basic models, the anomaly judgment results output by each of the multiple basic models can be obtained. By using the meta-model to comprehensively identify the multiple anomaly judgment results, the performance and robustness of abnormal data recognition can be effectively improved.

[0143] In a specific implementation, multiple basic models include a statistical model and a deep learning model, and the sensor data in multiple original data sets can be input into the statistical model and the deep learning model respectively to obtain a first abnormality judgment result output by the statistical model and a second abnormality judgment result output by the deep learning model.

[0144] In this embodiment, a plurality of basic models with different algorithms or parameter settings are usually selected, so as to give full play to the advantages of different models and obtain abnormality determination results in different dimensions.

[0145] S103-2: Obtain weighted meta-features based on the abnormality determination results output by each of the multiple basic models and the weight parameters corresponding to each basic model.

[0146] In this embodiment, corresponding weight parameters may also be set for each basic model. For example, in the model training stage and the verification stage, if the detection effect of the deep learning model is better than that of the statistical model, a higher weight parameter may be configured for the deep learning model.

[0147] In a specific implementation, the statistical model can analyze the statistical properties of each feature of the sensor data, such as mean, variance, maximum value, minimum value, etc., and can identify abnormal data points that are significantly deviated from normal data by setting a threshold or using an outlier detection algorithm (such as Z-Score, box plot, etc.). Among them, the first abnormality determination result can be determined based on the degree of deviation between the abnormal data point and the set threshold, that is, the first abnormality determination result is used to characterize the abnormal score of the sensor data.

[0148] In a specific implementation, the deep learning model can be trained using supervised learning algorithms (such as support vector machines, random forests, etc.) or unsupervised learning algorithms (such as clustering, anomaly detection algorithms, etc.). During the training process, normal data is used as training samples so that the model can learn the pattern of normal data, and then the trained deep learning model is used to classify or detect anomalies in the sensor data. Among them, the second anomaly determination result output by the deep learning model is a binary classification result, that is, the sensor data is abnormal data or normal data.

[0149] In a specific implementation, when the first abnormality determination result is used to characterize the abnormality score of the sensor data, and the second abnormality determination result is a binary classification result; S103-2 may specifically include the following sub-steps:

[0150] S103-2-1: Standardize the first abnormality determination result to obtain a standardized determination result.

[0151] In this embodiment, considering that if the meta-model is constructed using a neural network model, the requirements for input data are usually high, and the meta-model has a poor recognition effect on abnormal scores, the first abnormality judgment result will be subjected to normalization and other standardized processing to obtain a standardized judgment result that is easy for the meta-model to recognize.

[0152] S103-2-2: Obtain a first meta-feature based on the standardized determination result and a first weight parameter corresponding to the statistical model.

[0153] In this embodiment, the first meta-feature can be obtained by combining the standardized determination result and the first weight parameter corresponding to the statistical model. The first meta-feature can be represented as a two-dimensional array including the standardized determination result and the first weight parameter.

[0154] S103-2-3: Obtain a second meta-feature based on the second abnormality determination result and the second weight parameter corresponding to the deep learning model.

[0155] It should be noted that, since the second abnormality determination result is a binary classification result, there is no need to process the second abnormality determination result.

[0156] In this embodiment, by combining the second abnormality determination result and the second weight parameter corresponding to the deep learning model, a second meta-feature can be obtained. The second meta-feature can be represented as a two-dimensional array including the second abnormality determination result and the second weight parameter.

[0157] S103-2-4: Combining the first meta-feature and the second meta-feature to obtain a weighted meta-feature.

[0158] In this embodiment, by combining the first meta-feature and the second meta-feature, such as concatenating or multiplying, a weighted meta-feature with richer features can be obtained.

[0159] S103-3: Output the weighted meta-features to the meta-model to obtain a normal data set and an abnormal data set output by the meta-model.

[0160] In this embodiment, the meta-model can obtain more accurate abnormal data identification results by performing feature recognition on weighted meta-features, thereby accurately dividing the original data set into a normal data set and an abnormal data set.

[0161] In the specific implementation, the following methods can be used to train a comprehensive model containing multiple basic models and meta-models:

[0162] Step 1: Data preparation: Divide the sample dataset into training set and test set. For the training set, it can be further divided into multiple subsets.

[0163] Step 2: Base model training: Use each subset to train multiple different base models.

[0164] Step 3: Basic model prediction: For the samples in the test set, use each basic model to make predictions and obtain the anomaly probability or anomaly score of each model.

[0165] Step 4: Construct new features: The prediction results of each basic model are weighted as meta-features to obtain weighted meta-features.

[0166] Step 5: Meta-model training: Use the training set with weighted meta-features to train a meta-model. The meta-model can be any classifier suitable for abnormal data identification, such as logistic regression, support vector machine (SVM), etc.

[0167] Step 6: Metamodel prediction: For the samples in the test set, use the trained metamodel to predict. The prediction results can be used as the final abnormal data identification results.

[0168] In the second aspect, based on the same inventive concept, refer to Figure 2 , shows a sensor data processing system 200 based on big data provided by an embodiment of the present application, the system comprising:

[0169] The condition determination module 201 is used to determine whether the target sensor cluster meets the preset safety condition based on the risk characteristic data of the target sensor cluster;

[0170] The data acquisition module 202 is used to obtain the original data set of each node sensor based on the sensor data collected by multiple node sensors in the target sensor cluster within a preset period acquired in real time when it is determined that the target sensor cluster meets the safety condition;

[0171] The data repair module 203 is used to perform anomaly detection on the sensor data in the original data set for any original data set to obtain a normal data set and an abnormal data set; perform data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtain a target data set based on the repaired data set and the normal data set;

[0172] A time determination module 204, for determining, for any target data set, an update time of the sensor data based on the time characteristic data of the sensor data in the target data set;

[0173] The node determination module 205 is used to determine the target leaf node of each target data set in the pre-constructed Merkle tree based on the update time of the sensor data in each target data set; wherein,

[0174] The block update module 206 is used to write the hash value corresponding to the newly added sensor data into the target leaf node corresponding to the target data set to update the target block containing the Merkle tree when the newly added sensor data appears in any target data set;

[0175] The data consensus module 207 is used to broadcast the updated target block in the blockchain to achieve consensus on the newly added sensor data.

[0176] In an embodiment of the present application, the condition determination module 201 includes:

[0177] A risk characteristic data acquisition submodule is used to acquire device risk characteristic data and communication risk characteristic data of multiple node sensors in a target sensor cluster;

[0178] A comprehensive risk value determination submodule, used to determine the comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data;

[0179] The safety condition determination submodule is used to determine whether the target sensor cluster meets the safety condition when the comprehensive risk value is less than the risk threshold.

[0180] In one embodiment of the present application, the comprehensive risk value determination submodule includes:

[0181] A first risk value determining unit, configured to determine a first risk value based on the equipment risk feature data and a first risk weight;

[0182] A second risk value unit, configured to determine a second risk value based on the communication risk feature data and the second risk weight;

[0183] The comprehensive risk value determining unit is used to determine the comprehensive risk value based on the first risk value and the second risk value.

[0184] In one embodiment of the present application, the data repair module 203 includes:

[0185] The determination result acquisition submodule is used to input the sensor data in the original data set into multiple basic models respectively to obtain the abnormal determination results output by each of the multiple basic models;

[0186] A weighted meta-feature determination submodule, used to obtain weighted meta-features based on the abnormality determination results output by each of the multiple basic models and the weight parameters corresponding to each basic model;

[0187] The data set acquisition submodule is used to output the weighted meta-features to the meta-model to obtain the normal data set and the abnormal data set output by the meta-model.

[0188] In one embodiment of the present application, the multiple basic models include a statistical model and a deep learning model;

[0189] The determination result acquisition submodule is specifically used to input the sensor data in multiple original data sets into the statistical model and the deep learning model respectively, to obtain the first abnormality determination result output by the statistical model and the second abnormality determination result output by the deep learning model.

[0190] In one embodiment of the present application, the first abnormality determination result is used to characterize the abnormality score of the sensor data, and the second abnormality determination result is a binary classification result;

[0191] The weighted meta-feature determination submodule includes:

[0192] A standardization unit, used for performing standardization processing on the first abnormality determination result to obtain a standardized determination result;

[0193] A first meta-feature determination unit, configured to obtain a first meta-feature based on the standardized determination result and a first weight parameter corresponding to the statistical model;

[0194] A second meta-feature determination unit, configured to obtain a second meta-feature based on the second abnormality determination result and a second weight parameter corresponding to the deep learning model;

[0195] The weighted meta-feature unit is used to perform feature combination on the first meta-feature and the second meta-feature to obtain a weighted meta-feature.

[0196] In one embodiment of the present application, the time characteristic data includes the historical update time and the historical update frequency of the sensor data; the time determination module includes:

[0197] The prediction submodule is used to input the historical update time and the historical update frequency into the prediction model, and output the update time of the sensor data.

[0198] In an embodiment of the present application, the node determination module 205 includes:

[0199] A data set division submodule, used to determine two target data sets whose update time difference is less than a time threshold as associated data sets, and determine data sets other than the associated data sets among the multiple target data sets as independent data sets;

[0200] A first node determination submodule is used to determine, for any associated data set, two leaf nodes under the same subtree in the Merkle tree as two target leaf nodes of the associated data set respectively;

[0201] The second node determination submodule is used to randomly select an empty leaf node from the empty leaf nodes in the Merkle tree for any independent data set and determine it as a target leaf node of the independent data set.

[0202] In one embodiment of the present application, the block update module 206 includes:

[0203] A hash value generation submodule, for respectively generating first hash values ​​and second hash values ​​corresponding to the newly added sensor data in the two target data sets when there is newly added sensor data in the two target data sets in any associated data set;

[0204] The block update submodule is used to write the first hash value and the second hash value into two target leaf nodes corresponding to the associated data set respectively, so as to update the target block containing the Merkle tree.

[0205] It should be noted that the specific implementation of the sensor data processing system 200 based on big data in the embodiment of the present application refers to the specific implementation of the sensor data processing method based on big data proposed in the first aspect of the embodiment of the present application, which will not be repeated here.

[0206] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0207] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0208] In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0209] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0210] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0212] In the several embodiments provided in the present application, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0215] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A sensor data processing method based on big data, characterized in that: The method comprises: Based on the risk characteristic data of the target sensor cluster, determining whether the target sensor cluster meets a preset safety condition; In the case where it is determined that the target sensor cluster satisfies the safety condition, obtaining an original data set of each node sensor based on sensor data collected by multiple node sensors in the target sensor cluster within a preset period and acquired in real time; For any of the original data sets, perform anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set; perform data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtain a target data set based on the repaired data set and the normal data set; For any of the target data sets, determining an update time of the sensor data based on time characteristic data of the sensor data in the target data set; Determine, based on the update time of the sensor data in each target data set, a target leaf node of each target data set in a pre-constructed Merkle tree; When new sensor data appears in any target data set, a hash value corresponding to the new sensor data is written into a target leaf node corresponding to the target data set to update a target block containing the Merkle tree; The updated target block is broadcasted in the blockchain to achieve consensus on the newly added sensor data.

2. The sensor data processing method based on big data according to claim 1, characterized in that: The step of determining whether the target sensor cluster meets a preset safety condition based on the risk characteristic data of the target sensor cluster includes: Acquire device risk characteristic data and communication risk characteristic data of multiple node sensors in the target sensor cluster; Determining a comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data; When the comprehensive risk value is less than the risk threshold, it is determined that the target sensor cluster meets the safety condition.

3. The sensor data processing method based on big data according to claim 2 is characterized in that: The step of determining the comprehensive risk value of the target sensor cluster based on the device risk feature data and the communication risk feature data comprises: Determining a first risk value based on the equipment risk characteristic data and the first risk weight; Determining a second risk value based on the communication risk characteristic data and the second risk weight; The comprehensive risk value is determined based on the first risk value and the second risk value.

4. The sensor data processing method based on big data according to claim 1, characterized in that: The step of performing anomaly detection on the sensor data in the original data set to obtain a normal data set and an abnormal data set includes: Inputting the sensor data in the original data set into a plurality of basic models respectively to obtain the abnormality determination results outputted by the plurality of basic models respectively; Obtaining weighted meta-features based on the abnormality determination results output by each of the multiple basic models and the weight parameters corresponding to each of the basic models; The weighted meta-features are output to a meta-model to obtain a normal data set and an abnormal data set output by the meta-model.

5. The sensor data processing method based on big data according to claim 4 is characterized in that: The multiple basic models include statistical models and deep learning models; The step of inputting the sensor data in the original data set into a plurality of basic models respectively to obtain the abnormality determination results outputted by each of the plurality of basic models comprises: The sensor data in the multiple original data sets are respectively input into the statistical model and the deep learning model to obtain a first abnormality determination result output by the statistical model and a second abnormality determination result output by the deep learning model.

6. The sensor data processing method based on big data according to claim 5, characterized in that: The first abnormality determination result is used to characterize the abnormality score of the sensor data, and the second abnormality determination result is a binary classification result; The step of obtaining weighted meta-features based on the abnormality determination results outputted by each of the multiple basic models and the weight parameters corresponding to each of the basic models comprises: Performing standardization processing on the first abnormality determination result to obtain a standardized determination result; Obtaining a first meta-feature based on the standardized determination result and a first weight parameter corresponding to the statistical model; Obtaining a second meta-feature based on the second abnormality determination result and a second weight parameter corresponding to the deep learning model; The first meta-feature and the second meta-feature are combined to obtain the weighted meta-feature.

7. The sensor data processing method based on big data according to claim 1, characterized in that: The time characteristic data includes the historical update time and historical update frequency of the sensor data; The step of determining the update time of the sensor data based on the time characteristic data of the sensor data in the target data set comprises: The historical update time and the historical update frequency are input into a prediction model, and the update time of the sensor data is output.

8. The sensor data processing method based on big data according to claim 1, characterized in that: The step of determining a target leaf node of each target data set in a pre-constructed Merkle tree based on an update time of sensor data in each target data set comprises: Determine two target data sets whose update time difference is less than a time threshold as associated data sets, and determine data sets other than the associated data sets among the plurality of target data sets as independent data sets; For any associated data set, two leaf nodes under the same subtree in the Merkle tree are respectively determined as two target leaf nodes of the associated data set; For any independent data set, a free leaf node is randomly selected from the free leaf nodes in the Merkle tree to be determined as the target leaf node of the independent data set.

9. The sensor data processing method based on big data according to claim 8, characterized in that: In the case where new sensor data appears in any target data set, the step of writing a hash value corresponding to the new sensor data into a target leaf node corresponding to the target data set to update a target block containing the Merkle tree includes: In the case that there is newly added sensor data in both target data sets in any of the associated data sets, respectively generating first hash values ​​and second hash values ​​corresponding to the newly added sensor data in the two target data sets; The first hash value and the second hash value are respectively written into two target leaf nodes corresponding to the associated data set to update the target block containing the Merkle tree.

10. A sensor data processing system based on big data, characterized in that: The system comprises: A condition determination module, used to determine whether the target sensor cluster meets a preset safety condition based on the risk characteristic data of the target sensor cluster; A data acquisition module, configured to obtain an original data set of each node sensor based on sensor data collected by multiple node sensors in the target sensor cluster within a preset period acquired in real time when it is determined that the target sensor cluster meets the safety condition; Anomaly detection to obtain a normal data set and an abnormal data set; performing data repair on the sensor data in the abnormal data set to obtain a repaired data set, and obtaining a target data set based on the repaired data set and the normal data set; A time determination module, for determining, for any of the target data sets, an update time of the sensor data based on time characteristic data of the sensor data in the target data set; A node determination module is used to determine the target leaf node of each target data set in the pre-constructed Merkle tree based on the update time of the sensor data in each target data set; wherein, A block update module, configured to, when new sensor data appears in any target data set, write a hash value corresponding to the new sensor data into a target leaf node corresponding to the target data set, so as to update a target block containing the Merkle tree; The data consensus module is used to broadcast the updated target block in the blockchain to achieve consensus on the newly added sensor data.

Images