Data reduction system and method based on dynamic correlation analysis of internet of things data

Abstract

This invention provides a data reduction system and method based on dynamic correlation analysis of IoT data. The system includes an environmental change detection module, a correlation set construction module, a substitution relationship generation module, a sensor substitution model solving module, and a substitution relationship dynamic adjustment module. The method dynamically groups multiple sensors to obtain a correlation set, and generates a sensor substitution model based on the correlation set for data reduction and reconstruction. Simultaneously, it dynamically corrects the correlations between sensor data according to real-time changes in data correlation, and can dynamically update the substitution relationships and substitution models, thus ensuring the adaptive reduction of the predicted sensor transmission rate through inter-sensor substitution relationships.

Description

Technical Field

[0001] This invention belongs to the field of Internet of Things (IoT) technology, specifically relating to a data reduction system and method based on dynamic correlation analysis of IoT data. Background Technology

[0002] In the Internet of Things (IoT) environment, multi-source heterogeneous sensors are typically used collaboratively for applications such as environmental monitoring, event monitoring, and smart healthcare. The multi-source time-series data collected by these sensors often exhibit correlations, which can be used for data completion and service fault tolerance. Two sensor devices placed in the same location may carry highly correlated information. For example, in a smart desk lamp application, infrared and acoustic sensors may generate highly correlated information. To some extent, the readings of the acoustic sensor can be predicted by the infrared sensor, thereby reducing the sampling required by the acoustic sensor.

[0003] Existing research largely addresses the issues of missing, complete, and reduced sensor data based on the spatiotemporal correlation of sensors. For example, spatial correlation is used to model and predict data from environmental sensor networks, and data from multiple neighboring nodes is combined to estimate missing data at each node. Alternatively, temporal correlation of sensors is used for data reduction, and a predictive model is built using kernel linear regression. In communication between sensor nodes and receivers, only regression coefficients are transmitted, not the actual measured values.

[0004] Currently, data reduction methods for correlation analysis of multiple sensor data focus more on the spatiotemporal correlation between sensor data, lacking adaptability to the dynamic changes in the correlation patterns between sensor data. In certain IoT environmental monitoring scenarios, data collected by one sensor may show the same or opposite direction of change as other data, and the changes in the data may exhibit certain periodic characteristics. Ignoring the periodic and dynamic changes in correlation will make data reduction algorithms unable to adapt to the dynamic IoT environment, leading to a decrease in reduction rate and accuracy. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a data reduction system and method based on dynamic correlation analysis of IoT data. This invention performs dynamic correlation analysis on multi-dimensional data values ​​and dynamically constructs the correlation relationships between multiple sensor data based on the real-time changes in data correlation. This allows for the prediction of the values ​​of certain sensors based on the relationships between multiple correlated sensors, thereby achieving the reduction objective of reducing the transmission rate of the predicted sensors.

[0006] This invention provides a data reduction system based on dynamic correlation analysis of Internet of Things (IoT) data, comprising an environmental change detection module, a correlation set construction module, a substitution relationship generation module, a sensor substitution model solving module, and a substitution relationship dynamic adjustment module.

[0007] The environmental change detection module is used to receive sensor data sensed by the Internet of Things and perform concept drift detection;

[0008] The correlation set construction module is used to dynamically construct a multi-sensor correlation set subSi based on changes in the correlation of sensor data to maximize the data reduction rate.

[0009] The substitution relationship generation module is used to determine the substitution relationship between sensors based on the correlation between a single sensor and the other sensors and the remaining energy, and to construct a sensor substitution evaluation function to determine the optimal substitution sensor.

[0010] The sensor replacement model solving module is used to determine whether to construct a reconstruction function for sensor data based on the number of sensors in the set subSi, which is used for the optimal sensor replacement model.

[0011] The substitution relationship dynamic adjustment module is used to adjust the substitution relationship between multiple sensors according to changes in data correlation, thereby adapting to fluctuations in sensor relationships.

[0012] This invention provides a data reduction method based on dynamic correlation analysis of IoT data, implemented using a data reduction system based on dynamic correlation analysis of IoT data. The method includes:

[0013] Step 1: Load the IoT sensing data into the environmental change detection module;

[0014] Step 2: Construct a correlation set based on the received IoT sensing data, group the sensors, and form a multi-sensor correlation set subSi;

[0015] Step 3: Based on the multi-sensor correlation set subSi, construct a sensor substitution evaluation function oriented towards energy constraints to select the optimal alternative sensor;

[0016] Step 4: Solve the sensor substitution model for the data in the multi-sensor correlation set subSi to obtain the data reconstruction function of the optimal substitution sensor;

[0017] Step 5: When sensor attribute k in a certain correlation set subSi is no longer correlated with other attributes in subSi, update the correlation and regenerate the substitution relationship between multiple sensors, return to step 1 and perform concept drift detection; if multiple attributes in the correlation set subSi have undergone concept drift, the edge needs to detect whether the error of the current multiple linear regression prediction model exceeds the predefined error. If it exceeds, return to step 4 to update the data reconstruction function; otherwise, run with the current data reconstruction function.

[0018] Step 2 includes:

[0019] Step 2.1: Construct a correlation matrix A based on the Pearson correlation coefficient table among multiple sensors;

[0020]

[0021] In the formula, a ij Let represent the Pearson correlation coefficient between the i-th sensor and the j-th sensor, where i = 1, 2, ..., n, j = 1, 2, ..., n;

[0022] Step 2.2: Set the lower triangular elements of matrix A to 0, and set the Pearson correlation values ​​that are less than the set threshold to 0;

[0023]

[0024] Step 2.3: Form a multi-sensor correlation set subSi based on the correlation between sensors; specifically:

[0025] Step 2.3.1: Repeatedly select the maximum value max(a) from matrix A. ij ), and determine whether to max(a) ij The corresponding row i and column j are added to the preparatory set D; specifically:

[0026] ① If neither i nor j is in the preliminary set D, then the pair will be...<i,j> Add to the preparatory set D;

[0027] ②If only one of i and j is in the preliminary set D, then first determine the binary pair formed by sensor i or j and other sensors x.<j,x> or<i,x> Then the triplet<i,j,x> Add it to the relevant subset V, and set the i-th row and j-th column of matrix A to 0;

[0028] ③ If i and j are both in the preliminary set D, then a in matrix A ij Set to 0;

[0029] Step 2.3.2: Traverse all elements in matrix A until all elements in matrix A are 0;

[0030] Step 2.3.3: Add all tuples in the preparatory set D to the multi-sensor correlation set subSi.

[0031] Step 3 includes:

[0032] Step 3.1: Normalize the matrix A of Pearson correlation coefficients between multiple sensors in the correlation set subSi using formula (3) and the matrix E composed of the remaining power of the sensors using formula (4);

[0033]

[0034]

[0035] In the formula, e i This indicates the remaining battery power of sensor i;

[0036] Step 3.3 Construct the alternative evaluation function WAAw for sensor i:

[0037]

[0038] Among them, Data i =(a i1 a i2 , ..., a ij ..., a in e i Let w be a set of IoT sensing data about sensor i, where w = (w1, w2, ... w2). n ,w n+1 ) is a data set (a i1 a i2 , ..., a in e i The weight vector;

[0039] Step 3.4: Based on the sensor substitution evaluation function, select the sensor with the maximum function value as the optimal substitution sensor.

[0040] Step 4 includes:

[0041] Step 4.1: If the number of sensors in the correlation set subSi is 1, then the sensor substitution model cannot be solved, and step 4 is exited.

[0042] Step 4.2: If the number of sensors in the correlation set SubSi is 2, take out another sensor x in subSi, obtain the data of sensor x in the current window, solve the sensor substitution model by linear regression, and exit step 4;

[0043] Step 4.3: If the number of sensors in the correlation set SubSi is greater than 2, extract the remaining sensors x1, x2, ... from subSi, obtain the data of the remaining sensors in the current window, and solve the sensor substitution model through multiple linear regression.

[0044] Step 5 includes:

[0045] Step 5.1: If no concept drift is detected in the current sensor data from the sensor end to the edge end, continue with data reduction without adjusting the substitution relationship, and exit Step 5;

[0046] Step 5.2: If concept drift has occurred in s sensor data, find the relevant relation set subSi to which each of the s sensors belongs;

[0047] Step 5.3: Detect whether the correlation in the correlation set subSi in Step 5.2 has changed using the correlation solution method in Step 2. If it has changed, return to Step 2; if it has not changed, check whether the difference between the current data reconstruction value and the actual sensor data value is greater than the threshold error. If it is greater than the threshold error, return to Step 3.

[0048] The beneficial effects of this invention are:

[0049] This invention proposes a data reduction system and method based on dynamic correlation analysis of IoT data. The reduction rate is dynamically adjusted according to the data change pattern. In IoT environments where there are substitution relationships between sensors, the data reduction rate can be improved while ensuring the accuracy of data reconstruction, thereby reducing bandwidth and energy consumption. For dynamic IoT environments, the system can adapt to the semantic changes of sensor substitution relationships by dynamically adjusting substitution relationships and sensor substitution models, thereby maintaining high data reconstruction accuracy and maximizing the data reduction rate. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the data reduction system based on dynamic correlation analysis of IoT data in this invention.

[0051] Figure 2 This is a flowchart of the data reduction method based on dynamic correlation analysis of IoT data in this invention.

[0052] Figure 3 This is a schematic diagram of the correlation analysis process between sensors in this invention.

[0053] Figure 4 This is a schematic diagram of the construction of the preparatory set D in this invention; wherein, (a) is a schematic diagram in which both row i and column j are added to the preparatory set D; and (b) is a schematic diagram in which either row i or column j is added to the preparatory set D. Detailed Implementation

[0054] The invention will be further explained below with reference to the accompanying drawings and specific implementation examples.

[0055] A data reduction system based on dynamic correlation analysis of Internet of Things (IoT) data includes an environmental monitoring module, a correlation set construction module, a substitution relation generation module, a sensor substitution model solving module, and a substitution relation dynamic adjustment module.

[0056] The environmental change detection module is used to receive sensor data sensed by the Internet of Things and perform concept drift detection;

[0057] The correlation set construction module is used to dynamically construct the optimal sensor correlation set subSi based on changes in sensor data correlation to maximize data reduction. Due to the unpredictability of the environment, the correlation between multiple sensor data is highly susceptible to change. When fewer sensors are selected as a group, the correlation is relatively stable, making it easier to update the substitution relationship between sensor data and the sensor grouping based on sensor correlation. Therefore, sensors with high correlation are added to the same correlation set subSi.

[0058] The substitution relationship generation module is used to determine the substitution relationship between sensors based on the correlation between a single sensor and the other sensors and the remaining energy, and to construct a sensor substitution evaluation function to determine the optimal substitute sensor. This module comprehensively considers the correlation between a single sensor and the other sensors and the remaining energy as factors in determining the substitution relationship between sensors. Considering that the correlation between each sensor and other sensors is different in the correlation relationship set subSi, and that each sensor consumes different amounts of energy, when the sensor energy weight is determined, the optimal substitute sensor is determined through the sensor substitution evaluation function.

[0059] The sensor substitution model solving module is used to determine whether to construct a sensor data reconstruction function based on the number of sensors in the set subSi, for the optimal sensor substitution model. For each correlation set subSi, the optimal substitution sensor y in that correlation set subSi is first calculated, and data about y is obtained. Then, the number of elements (i.e., sensors) in subSi is determined. If there is only one element in subSi, a data reconstruction function cannot be constructed; if there are two elements in subSi, a simple linear regression data reconstruction function is constructed; if there are more than two elements in subSi, a multiple linear regression data reconstruction function is constructed.

[0060] The dynamic adjustment module for substitution relationships is used to adjust the substitution relationships between multiple sensors based on changes in data correlation, thereby adapting to fluctuations in sensor relationships. Considering the unpredictability of the IoT environment, it is necessary to promptly determine whether there is correlation between multiple sensor data in the current state, and how the strength of this correlation changes. If the correlation between multiple sensor data changes, it will trigger an update of the sensor substitution relationships and a regeneration of the data reconstruction function between the multiple sensors. If only the strength of the correlation between multiple sensors changes, only the regression prediction model will be updated.

[0061] like Figure 1As shown, during the reduction process, multiple sensors with different attributes are divided into multiple correlation sets subSi (where i = 1, 2, ..., n). When a sensor attribute k in a correlation set subSi no longer has a correlation with other attributes in subSi, the correlation needs to be updated and the substitution relationships between multiple sensors need to be regenerated. If multiple attributes in the correlation set subSi experience conceptual drift at this time, the edge needs to detect whether the error of the current multiple linear regression prediction model is large. If it exceeds the predefined error, the multiple linear regression data reconstruction function needs to be regenerated.

[0062] like Figure 2 As shown, a data reduction method based on dynamic correlation analysis of IoT data is implemented based on the aforementioned data reduction system based on dynamic correlation analysis of IoT data. The method includes:

[0063] Step 1: Load the IoT sensing data into the environmental change detection module;

[0064] Step 2: Construct a correlation set based on the received IoT sensing data, group the sensors, and form a multi-sensor correlation set subSi; for example... Figure 3 As shown, the specific description is as follows:

[0065] Step 2.1: Construct a correlation matrix A based on the Pearson correlation coefficient table among multiple sensors;

[0066]

[0067] In the formula, a ij This represents the Pearson correlation coefficient between the i-th sensor and the j-th sensor, where i = 1, 2, ..., n.

[0068] j = 1, 2, ..., n;

[0069] Step 2.2: Set the lower triangular elements of matrix A to 0, and set the Pearson correlation values ​​that are less than the set threshold to 0;

[0070]

[0071] Step 2.3: Form a multi-sensor correlation set subSi based on the correlation between sensors; specifically:

[0072] Step 2.3.1: Repeatedly select the maximum value max(a) from matrix A. ij ), and determine whether to max(a) ij The corresponding row i and column j are added to the preparatory set D; specifically:

[0073] ① If neither i nor j is in the preliminary set D, then the pair will be...<i,j> Add to the preparatory set D; Figure 4 (a) A diagram showing that both row i and column j are added to the preparatory set D; as shown Figure 4 As shown in (a), if neither sensor J nor I is in the preparatory set D, then sensor J and I are added to the preparatory set D as a pair.

[0074] ②If only one of i and j is in the preliminary set D, then first determine the binary pair formed by sensor i or j and other sensors x.<j,x> or<i,x> Then the triplet<i,j,x> Add it to the relevant subset V, and set the i-th row and j-th column of matrix A to 0; Figure 4 (b) A diagram showing the addition of row i or column j to the preparatory set D; such as Figure 4 As shown in (b), if sensor J is not in the preliminary set D, and if there is a binary pair consisting of sensor I and H in the preliminary set D, then sensor I, J and H are added to the preliminary set D as a triple.

[0075] ③ If i and j are both in the preliminary set D, then a in matrix A ij Set to 0;

[0076] Step 2.3.2: Traverse all elements in matrix A until all elements in matrix A are 0;

[0077] Step 2.3.3: Add all tuples in the preparatory set D to the multi-sensor correlation set subSi.

[0078] Step 3: Based on the multi-sensor correlation set subSi, construct a sensor substitution evaluation function oriented towards energy constraints to select the optimal alternative sensor; specifically, as follows:

[0079] Step 3.1: Normalize the matrix A of Pearson correlation coefficients between multiple sensors in the correlation set subSi using formula (3) and the matrix E composed of the remaining power of the sensors using formula (4);

[0080]

[0081]

[0082] In the formula, e i This indicates the remaining battery power of sensor i;

[0083] Step 3.3 Construct the alternative evaluation function WAAw for sensor i:

[0084]

[0085] Among them, Data i =(a i1 a i2 , ..., a ij ..., a in e i Let w be a set of IoT sensing data about sensor i, where w = (w1, w2, ... w2). n ,w n+1 ) is a data set (a i1 a i2 , ..., a in e i The weight vector;

[0086] Step 3.4: Based on the sensor substitution evaluation function, select the sensor with the maximum function value as the optimal substitution sensor.

[0087] Step 4: Solve the sensor substitution model for the data in the multi-sensor correlation set subSi to obtain the data reconstruction function of the optimal substitution sensor; specifically, it is described as follows:

[0088] Step 4.1: If the number of sensors in the correlation set subSi is 1, then the sensor substitution model cannot be solved, and step 4 is exited.

[0089] Step 4.2: If the number of sensors in the correlation set SubSi is 2, take out another sensor x in subSi, obtain the data of sensor x in the current window, solve the sensor substitution model by linear regression, and exit step 4;

[0090] Step 4.3: If the number of sensors in the correlation set SubSi is greater than 2, extract the remaining sensors x1, x2, ... from subSi, obtain the data of the remaining sensors in the current window, and solve the sensor substitution model through multiple linear regression.

[0091] Step 5: When sensor attribute k in a correlation set subSi is no longer correlated with other attributes in subSi, update the correlation and regenerate the substitution relationship between multiple sensors, return to step 1 and perform concept drift detection; if multiple attributes in the correlation set subSi have experienced concept drift, the edge needs to detect whether the error of the current multiple linear regression prediction model exceeds the predefined error. If it does, return to step 4 to update the data reconstruction function; otherwise, run with the current data reconstruction function; specifically:

[0092] Step 5.1: If no concept drift is detected in the current sensor data from the sensor end to the edge end, continue with data reduction without adjusting the substitution relationship, and exit Step 5;

[0093] Step 5.2: If concept drift has occurred in s sensor data, find the relevant relation set subSi to which each of the s sensors belongs;

[0094] Step 5.3: Detect whether the correlation in the correlation set subSi in Step 5.2 has changed using the correlation solution method in Step 2. If it has changed, return to Step 2; if it has not changed, check whether the difference between the current data reconstruction value and the actual sensor data value is greater than the threshold error. If it is greater than the threshold error, return to Step 3.

Claims

1. A data reduction system based on dynamic correlation analysis of Internet of Things data, characterized in that, The method comprises: The environmental change detection module is used for receiving Internet of Things sensing sensor data and performing concept drift detection; The correlation set construction module is used for dynamically constructing a multi-sensor correlation set subSi according to changes in sensor data correlation to maximize data reduction rate; The replacement relationship generation module is used for determining a replacement relationship between sensors according to the correlation degree of a single sensor with the remaining sensors and the remaining energy, and constructing a sensor replacement evaluation function to determine an optimal replacement sensor; The sensor replacement model solving module is used for determining whether to construct a sensor data reconstruction function according to the number of sensors in the set subSi, for an optimal sensor replacement model; The replacement relationship dynamic adjustment module is used for adjusting the replacement relationship between multiple sensors according to changes in data correlation to adapt to fluctuations in sensor relationships. The method comprises:

2. A data reduction method based on dynamic correlation analysis of Internet of Things data, implemented based on the data reduction system based on dynamic correlation analysis of Internet of Things data according to claim 1, characterized in that, Step 1: uniformly loading Internet of Things sensing data into the environmental change detection module; Step 2: constructing a correlation set according to the received Internet of Things sensing data, grouping sensors, and forming a multi-sensor correlation set subSi; Step 3: constructing a sensor replacement evaluation function for energy constraints according to the multi-sensor correlation set subSi to select an optimal replacement sensor; Step 4: solving a sensor replacement model for data in the multi-sensor correlation set subSi to obtain a data reconstruction function of the optimal replacement sensor, and solving the sensor replacement model through multiple linear regression; Step 5: when a sensor attribute k in a certain correlation set subSi no longer has a correlation with other attributes in subSi, updating the correlation and regenerating the replacement relationship between multiple sensors, returning to step 1 and performing concept drift detection; if multiple attributes in the correlation set subSi have concept drift, the edge needs to detect whether the prediction model error of the current multiple linear regression exceeds the predefined error error, and if it exceeds, return to step 4 to update the data reconstruction function; otherwise, run with the current data reconstruction function. The step 2 comprises: 3.The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 2, characterized in that, Step 2.3: forming a multi-sensor correlation set subSi according to the correlation between sensors. Step 2.1: Building a correlation matrix from the Pearson correlation coefficient table among multiple sensors ; (1) wherein denotes the Pearson correlation coefficient between the ith sensor and the jth sensor, , ; Step 2.2: Set the lower triangle elements to 0 and set Pearson correlation values less than a set threshold to 0. Step 2.2: Set the lower triangle elements to 0 and set Pearson correlation values less than a set threshold to 0. (2) The step 2.3 comprises:

4. The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 3, characterized in that, Step 2.3.3: adding all tuples in the preliminary set D to the multi-sensor correlation set subSi. Step 2.3.1: Repeat from matrix Select the maximum value from the list. ), and determine whether to max( The corresponding row i and column j are added to the preparatory set D; Step 2.3.2: Traverse all elements in the matrix until all elements in the matrix are 0; The step 2.3.1 is specifically expressed as:

5. The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 4, characterized in that, The step 3 comprises: If neither i nor j is in the preliminary set D, then the pair <i,j> is added to the preliminary set D. If only one of i and j is in the preliminary set D, then the pair <j, x> or <i, x> is determined, where x is the sensor not in the pair, and the triplet <i, j, x> is added to the correlation subset V, and the i-th row and j-th column of the matrix are set to zero. If both i and j are already in the preliminary set D, set the element in the matrix to 0.

6. The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 2, characterized in that, Step 3.4: according to the sensor replacement evaluation function, taking the sensor corresponding to the maximum function value as the optimal replacement sensor. Step 3.1: a matrix of Pearson correlation coefficients between multiple sensors in the correlation set subSi The matrix E of the remaining power of the sensors is normalized using formula (4) respectively; （3） （4） In the formula, represents the remaining power of sensor i; Step 3.3 Constructing an alternative evaluation function for sensor i : （5） wherein Data i = (a i1 , a i2 ,..., ..., , ) is a set of Internet of Things perception data about sensor i, is a weight vector for the set of data (a i1 , a i2 ,..., , ). The step 4 comprises:

7. The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 2, characterized in that, Step 4.1: if the number of sensors in the correlation set subSi is 1, the sensor replacement model cannot be solved, and step 4 is exited; Step 4.2: if the number of sensors in the correlation set subSi is 2, take out another sensor x in subSi, obtain the data of sensor x in the current window, and solve the sensor replacement model through linear regression, and exit step 4; ​ Step 4.3: If the number of sensors in the correlation set SubSi is greater than 2, take out the remaining sensors x1, x2,.., in subSi, obtain the data of the remaining sensors in the current window, and solve the sensor substitution model by multiple linear regression.

8. The data reduction method based on dynamic correlation analysis of Internet of Things data according to claim 2, characterized in that, The step 5 comprises: Step 5.1: If no concept drift is detected in the current sensor data from the sensor end to the edge end, continue with the data reduction, no substitution relationship adjustment is needed, and exit step 5; Step 5.2: If concept drift occurs in s sensor data, find out the correlation set subSi to which each of the s sensors belongs; Step 5.3: Detect whether the correlation in the correlation set subSi in step 5.2 has changed by the correlation solving method in step 2, if it has changed, return to step 2; if it has not changed, detect whether the difference between the current data reconstruction value and the actual sensor data value is greater than the threshold error, if it is greater than the threshold error, return to step 4.

Images