Sensor data acquisition and processing method and system based on Internet of Things
By dividing the degree of dispersion and correlation of sensor nodes into low density or high density, the problem of inaccurate node density caused by inaccurate sensor data coverage area is solved, and the denoising effect and filtering accuracy are improved.
Patent Information
- Application Number
- CN202411020300.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The coverage area of sensor data is affected by a variety of factors, resulting in inaccurate division of node density and affecting the denoising effect.
According to the degree of dispersion and correlation of sensor nodes, different denoising strategies are adopted to avoid errors relying on coverage area measurement and improve filtering effect.
More precisely identify the concentration and distribution of node data, accurately define high-density and low-density nodes, improve the denoising effect, and save computing resources.
Smart Images

Figure CN118568583B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a sensor data collection and processing method and system based on the Internet of Things. Background Art
[0002] With the rapid development of Internet of Things technology, various sensor devices are widely used in various fields, such as industry, agriculture, medical treatment, transportation, etc. These sensor devices can collect diverse information data such as environment and equipment status in real time, providing a rich data source for Internet of Things applications. However, sensor data often has problems such as noise, outliers and missing values, and appropriate methods need to be used to process them to improve data quality.
[0003] In "Computer Engineering and Applications", a sensor network perception data adaptive denoising method, referred to as the DHA algorithm, is disclosed. This method can perform adaptive denoising according to the density of nodes in the sensor network. In areas with high node density, a more complex denoising algorithm is used to improve data accuracy; in areas with low node density, a simpler denoising algorithm is used to save energy.
[0004] In the above-mentioned disclosed adaptive denoising method for sensor network perception data, each sensor is a node, and the ratio of the number of data collected by the sensor in a section to the coverage area of the sensor is taken as the node density. However, the coverage area of the sensor may be a vague concept, and its actual size may be affected by many factors, such as environmental obstacles, signal attenuation, etc. Therefore, there is an error when measuring whether a node is a low-density node or a high-density node based on the coverage area of the sensor, resulting in a poor denoising effect. Summary of the invention
[0005] In order to solve the problem that the coverage area of the sensor may be affected by various factors and become inaccurate, and therefore there will be errors when measuring whether the node is a low-density node or a high-density node based on the coverage area of the sensor, thereby reducing the denoising effect, the present invention provides a sensor data collection and processing method and system based on the Internet of Things.
[0006] In the first aspect, the present invention provides a sensor data collection and processing method based on the Internet of Things, which adopts the following technical solutions:
[0007] The sensor data collection and processing method based on the Internet of Things includes the following steps:
[0008] Collect data from each sensor;
[0009] Taking each sensor as each node, obtaining the possibility that each data of each node is the node center; obtaining the node center of each node based on the possibility; obtaining the degree of dispersion of each node, wherein the degree of dispersion is the average value of the distance between each data in each node and the node center of each node;
[0010] Get the degree of association between each node and all other nodes except this node:
[0011] , where Represents the degree of association between the ith node and the cth node except the ith node; Represents the similarity between the i-th node and the c-th node except the i-th node; represents the Euclidean distance between the node center of the i-th node and the node center of the c-th node excluding the i-th node; Represents the variance of all data in the i-th node; Represents the variance of all data in the cth node;
[0012] Based on the degree of association and the discreteness of each node, the possibility of each node being divided into a low-density node is obtained; based on the possibility of each node being divided into a low-density node, low-density nodes and high-density nodes are obtained; different strategies are used to denoise the high-density nodes and the low-density nodes.
[0013] The innovation of the present invention lies in obtaining the possibility of dividing each node into a low-density node according to the discreteness of each node and the importance of each node, and dividing each node into a low-density node or a high-density node for denoising, thereby avoiding the phenomenon of inaccurate node density due to inaccurate sensor coverage area, and being able to more finely identify the concentration and distribution of data of different nodes, thereby more accurately defining high-density and low-density nodes and improving the filtering effect. Furthermore, the discreteness of each data of each node is obtained based on the mean of the distance between each data of each node and the node center of each node, which can reflect the data distribution in each node. The larger the value, the more discrete the data distribution is, and the more likely the node is a low-density node. Furthermore, the importance of each node is obtained by the degree of association between each node and other nodes. The greater the degree of association, the more important the node is and the more it should be a high-density node.
[0014] Preferably, the obtaining of each data of each node as a possibility of the node center comprises the steps of:
[0015] For any node, obtain the stability characteristics of each data in the node; take the absolute value of the difference between the j-th data in the node and the mean of all the data in the node as the deviation characteristic of the j-th data in the node; take the stability characteristic and the deviation characteristic of the j-th data in the node as a vector, recorded as the central characteristic vector of the j-th data in the node, and take the modulus of the central characteristic vector of the j-th data in the node as the possibility that the j-th data in the node is the node center.
[0016] The node center of each node is obtained, so that the discrete degree of each node can be obtained according to the distance between each data of each node and the node center of each node.
[0017] Preferably, the step of obtaining the stability characteristics of each data in the node comprises the steps of:
[0018] For any node, obtain the absolute value of the difference between the previous data and the j-th data at the collection time of the j-th data in the node, and record it as the first absolute value; obtain the absolute value of the difference between the next data and the j-th data at the collection time of the j-th data in the node, and record it as the second absolute value; and take the sum of the first absolute value and the second absolute value as the stability feature of the j-th data in the node.
[0019] Preferably, the node center of each node is obtained by calculating the formula:
[0020] According to the possibility of each data in any node being the node center, the data corresponding to the maximum value of the possibility is taken as the node center of the node.
[0021] Preferably, the similarity is obtained by calculating the formula:
[0022] The cosine similarity of the central feature vector corresponding to the node center of the i-th node and each node except the i-th node is obtained as the similarity between the i-th node and each node except the i-th node.
[0023] This makes it easier to obtain the degree of association between nodes based on the similarity between nodes, and then obtain the importance of the nodes.
[0024] Preferably, the obtaining of the possibility of dividing each node into low-density nodes includes:
[0025] Get the mean of the association degree between the ith node and each node except the ith node as the importance of the ith node;
[0026] Get the possibility of dividing each node into low-density nodes according to the importance of each node:
[0027] ;
[0028] In the formula, Represents the possibility of the i-th node being divided into a low-density node; Represents the discrete degree of the i-th node; Represents the importance of the i-th node; norm() represents the normalization function.
[0029] The greater the dispersion, the more likely nodes with lower importance are to be low-density nodes.
[0030] Preferably, the calculation formula for obtaining low-density nodes and high-density nodes is:
[0031] A possibility threshold T1 is preset. If the possibility of any node being classified as a low-density node is greater than or equal to the possibility threshold T1, the node is a low-density node, otherwise it is a high-density node.
[0032] Based on the possibility threshold, each node is divided into low-density nodes and high-density nodes, and different strategies are used for denoising, which improves the filtering effect.
[0033] In a second aspect, the present invention provides a sensor data acquisition and processing system based on the Internet of Things, which adopts the following technical solutions:
[0034] The sensor data collection and processing system based on the Internet of Things includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the sensor data collection and processing method based on the Internet of Things is implemented.
[0035] By adopting the above technical solution, the above-mentioned sensor data collection and processing method based on the Internet of Things is generated into a computer program and stored in a memory to be loaded and executed by a processor, so that a terminal device is made according to the memory and the processor for easy use.
[0036] The present invention has the following technical effects: the present invention obtains the possibility of dividing each node into a low-density node according to the discreteness of each node and the importance of each node, and divides each node into a low-density node or a high-density node for denoising, thereby avoiding the phenomenon of inaccurate node density caused by inaccurate sensor coverage area, and can more finely identify the concentration and distribution of data of different nodes, so as to more accurately define high-density and low-density nodes and improve the filtering effect. Furthermore, the discreteness of each data of each node is obtained according to the mean of the distance between each data of each node and the node center of each node, which can reflect the data distribution in each node. The larger the value, the more discrete the data distribution is, and the more likely the node is a low-density node. Furthermore, the importance of each node is obtained by the degree of association between each node and other nodes. The greater the degree of association, the more important the node is, and the more it should be a high-density node. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts.
[0038] Figure 1 It is a method flow chart of a sensor data collection and processing method based on the Internet of Things in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0040] It should be understood that when the terms "first", "second", etc. are used in the claims, descriptions, and drawings of the present invention, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.
[0041] The embodiment of the present invention discloses a sensor data collection and processing method based on the Internet of Things, referring to Figure 1 , comprising steps S1 to S3:
[0042] S1: Collect data from each sensor.
[0043] Step S1 includes step S10, which is specifically as follows:
[0044] S10: Collect data from each sensor.
[0045] In the embodiment of the present invention, multiple sensors are installed at different locations in the same area, and sampling is performed every 1 minute for 1 hour. Each data of each sensor is collected each time to obtain the data of each sensor.
[0046] S2: Take each sensor as each node, obtain the possibility that each data in each node is the node center, and based on the possibility, obtain the node center of each node; based on the distance between the data in the node and the node center, obtain the discrete degree of each node; obtain the degree of association between each node and each node except each node, and based on the degree of association, obtain the importance of each node.
[0047] It should be noted that in the process of data collection by sensors, due to electromagnetic interference in environmental factors, noisy data will inevitably exist. Therefore, denoising the sensor data is a common means of operation. The DHA algorithm is a denoising algorithm developed for sensor data. The core of the algorithm is to use different denoising algorithms for high-density nodes and low-density nodes to reasonably allocate computing resources to improve denoising efficiency. Among them, the division of high-density nodes and low-density nodes depends on the node density threshold. In the traditional DHA algorithm, each sensor is regarded as a node, and the ratio of the number of data collected by the sensor in a section to the coverage area of the sensor is used as the node density. In actual applications, obstacles in the environment will affect the coverage area of the sensor, and then affect the node density. At this time, the high and low division of node density based on a fixed threshold may result in misjudgment, resulting in unreasonable allocation of computing resources and affecting the final denoising effect.
[0048] Step S2 includes steps S20-S21, which are specifically as follows:
[0049] S20: Taking each sensor as each node, obtaining the possibility that each data in each node is the node center, and based on the possibility, obtaining the node center of each node; based on the distance between the data in the node and the node center, obtaining the discrete degree of each node.
[0050] It should be noted that, taking each sensor as each node, it is known that the data distribution characteristic is a characteristic quantity that changes with time and environment. According to the distribution characteristics of the data corresponding to each node, each node is divided into a high-density node or a low-density node. This not only avoids the uncertainty of measuring the coverage area of the sensor, but also can more finely identify the data concentration and distribution of different nodes, thereby more accurately defining high-density and low-density nodes. Therefore, the present invention proposes a method for denoising the sensor data of the Internet of Things using the DHA algorithm, in which the node density is divided into high and low according to the data distribution characteristics of the node, so as to adapt to denoising in different scenarios.
[0051] It should be further explained that the discreteness of known data is an important dimension to describe data distribution. It reflects the degree of dispersion or aggregation between data. Therefore, for the data of any sensor, if the discreteness of the data of the sensor is high, it means that the data of the sensor is relatively sparse. At this time, it will be divided into low-density nodes and denoised by a simple denoising algorithm, which can save computing resources. Otherwise, it will be divided into high-density nodes.
[0052] In an embodiment of the present invention, each sensor is taken as each node. For any node, the absolute value of the difference between the previous data of the collection time of the j-th data in the node and the j-th data is obtained, which is recorded as the first absolute value; the absolute value of the difference between the next data of the collection time of the j-th data in the node and the j-th data is taken, which is recorded as the second absolute value, and the sum of the first absolute value and the second absolute value is taken as the stability feature of the j-th data in the node, and the absolute value of the difference between the j-th data in the node and the mean of all the data in the node is taken as the deviation feature of the j-th data in the node; the stability feature and the deviation feature of the j-th data in the node are taken as a vector, which is recorded as the central feature vector of the j-th data in the node, and the modulus of the central feature vector of the j-th data in the node is taken as the possibility that the j-th data in the node is the node center.
[0053] The possibility that each data in the node is the node center is obtained, and the data corresponding to the maximum value of the possibility is used as the node center of the node.
[0054] It should be noted that the discrete degree of each node is obtained:
[0055] ;
[0056] In the formula, Represents the discrete degree of the i-th node; Represents the number of data of the i-th node; represents the Euclidean distance between the vth data in the ith node and the node center of the ith node; exp() represents an exponential function with a natural constant as the base, The larger the value of , the more discrete the data distribution in the i-th node.
[0057] S21: Obtain the degree of association between each node and each node other than each node, and obtain the importance of each node based on the degree of association.
[0058] It should be noted that it is inaccurate to judge whether a node is a high-density node or a low-density node only based on the discreteness of the data in the node. The importance of the node also needs to be considered. The higher the degree of correlation between any node and other nodes, the more important the node is. It may be a key data source. At this time, it needs to be divided into high-density nodes and use complex denoising algorithms to perform more detailed denoising on it to improve the data quality of important information. Otherwise, it should be divided into low-density nodes and use simple denoising algorithms to denoise it.
[0059] In the embodiment of the present invention, the cosine similarity of the central feature vector corresponding to the node center of the ith node and each node except the ith node is obtained as the similarity between the ith node and each node except the ith node. It should be noted that the larger the value of the cosine similarity, the greater the similarity between the two nodes and the greater the degree of association.
[0060] Get the degree of association between the ith node and the cth node except the ith node:
[0061] ;
[0062] In the formula, Represents the degree of association between the ith node and the cth node except the ith node; Represents the similarity between the i-th node and the c-th node except the i-th node; represents the Euclidean distance between the node center of the i-th node and the node center of the c-th node excluding the i-th node; Represents the variance of all data in the i-th node; Represents the variance of all data in the cth node; the greater the similarity between two nodes, the greater the degree of association, and the smaller the Euclidean distance between the node center of the ith node and the node center of the cth node excluding the ith node, and the smaller the difference in the variance of all data in the ith node and the cth node excluding the ith node, the more similar the distribution of the ith node and the cth node excluding the ith node is, and the more credible the degree of association between the two is.
[0063] The average value of the association degree between the ith node and each node except the ith node is obtained as the importance of the ith node. The larger the value, the more important the ith node is.
[0064] S3: According to the discreteness and importance of each node, the possibility of each node being divided into a low-density node is obtained, and based on the possibility of each node being divided into a low-density node, low-density nodes and high-density nodes are obtained.
[0065] Step S3 includes steps S30 and S31, which are specifically as follows:
[0066] S30: According to the discreteness and importance of each node, the possibility of each node being divided into a low-density node is obtained. Based on the possibility of each node being divided into a low-density node, low-density nodes and high-density nodes are obtained, and different strategies are used to denoise the high-density nodes and the low-density nodes.
[0067] It should be noted that when the degree of discreteness of any node is greater, the data corresponding to the node is sparser, and the possibility of classifying it as a low-density node is greater. At the same time, when the importance of the node is lower, it is classified as a low-density node, and the possibility of denoising using a simple denoising algorithm is greater. Therefore, according to the degree of discreteness and importance of the node, the possibility of classifying the node as a low-density node is obtained. The greater the degree of discreteness of the node and the higher the importance of the node, the greater the possibility of classifying it as a low-density node.
[0068] In an embodiment of the present invention, the possibility of the i-th node being divided into a low-density node is obtained:
[0069] ;
[0070] In the formula, Represents the possibility of the i-th node being divided into a low-density node; Represents the discrete degree of the i-th node; Represents the importance of the i-th node; norm() represents the normalization function, which adopts the maximum and minimum value normalization method.
[0071] The possibility threshold T1 is preset. If the possibility of the i-th node being divided into a low-density node is greater than or equal to the possibility threshold T1, the i-th node is a low-density node, otherwise it is a high-density node. The division is completed for each node. In the embodiment of the present invention, the possibility threshold T1 is preset to 0.6. In other embodiments, the implementer can preset the value of the possibility threshold T1 according to the specific implementation return.
[0072] S31: Different strategies are used to denoise high-density nodes and low-density nodes.
[0073] In an embodiment of the present invention, the DHA algorithm is used to denoise high-density nodes and low-density nodes using different strategies. For example, for sensors with high node density, since the data collected by the sensor is relatively concentrated and has a high degree of correlation with other sensors, the importance of the data collected by the sensor is higher, and a more complex denoising method needs to be selected to perform deep denoising, such as Kalman filtering or a model-based method. For sensors with low node density, since the data collected by the sensor is relatively sparse and has a low degree of correlation with other sensors, the importance of the data collected by the sensor is lower, and a simpler denoising method can be selected, such as a moving average filter or a median filter. It should be noted that the DHA algorithm is a prior art and will not be described in detail in this solution.
[0074] The present invention obtains the possibility of each node being divided into a low-density node according to the discreteness of each node and the importance of each node, and divides each node into a low-density node or a high-density node for denoising, thereby avoiding the phenomenon of inaccurate node density caused by inaccurate sensor coverage area, and can more finely identify the concentration and distribution of different node data, so as to more accurately define high-density and low-density nodes and improve the filtering effect. Furthermore, the discreteness of each data of each node is obtained according to the mean of the distance between each data of each node and the node center of each node, which can reflect the data distribution in each node. The larger the value, the more discrete the data distribution is, and the more likely the node is a low-density node. Furthermore, the importance of each node is obtained by the degree of association between each node and other nodes. The greater the degree of association, the more important the node is, and the more it should be a high-density node.
[0075] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, and their configuration and functions are known in the art, so they will not be described in detail here.
[0076] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory, a dynamic random access memory, a static random access memory, an enhanced dynamic random access memory, a high bandwidth memory, a hybrid storage cube, etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device.
[0077] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.
[0078] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A sensor data collection and processing method based on the Internet of Things, characterized in that: Includes steps: Collect data from each sensor; Taking each sensor as each node, obtaining each data of each node as the possibility of node center; Based on the possibility, the node center of each node is obtained, including: according to the possibility that each data in any node is the node center, the data corresponding to the maximum value of the possibility is used as the node center of the node; the discrete degree of each node is obtained, and the discrete degree is the average value of the distance between each data in each node and the node center of each node; Get the degree of association between each node and all other nodes except this node: , where Represents the degree of association between the ith node and the cth node except the ith node; Represents the similarity between the i-th node and the c-th node except the i-th node; represents the Euclidean distance between the node center of the ith node and the node center of the cth node excluding the ith node; Represents the variance of all data in the i-th node; Represents the variance of all data in the cth node; Based on the degree of association and the discreteness of each node, the possibility of each node being divided into a low-density node is obtained; based on the possibility of each node being divided into a low-density node, low-density nodes and high-density nodes are obtained; different strategies are used to denoise the high-density nodes and the low-density nodes.
2. The sensor data collection and processing method based on the Internet of Things according to claim 1 is characterized in that: The method of obtaining the possibility of each data of each node being a node center comprises the steps of: For any node, obtain the stability characteristics of each data in the node; take the absolute value of the difference between the j-th data in the node and the mean of all the data in the node as the deviation characteristic of the j-th data in the node; take the stability characteristic and the deviation characteristic of the j-th data in the node as a vector, recorded as the central characteristic vector of the j-th data in the node, and take the modulus of the central characteristic vector of the j-th data in the node as the possibility that the j-th data in the node is the node center.
3. The sensor data collection and processing method based on the Internet of Things according to claim 2 is characterized in that: The step of obtaining the stability characteristics of each data in the node comprises the following steps: For any node, obtain the absolute value of the difference between the previous data and the j-th data at the collection time of the j-th data in the node, and record it as the first absolute value; obtain the absolute value of the difference between the next data and the j-th data at the collection time of the j-th data in the node, and record it as the second absolute value; and take the sum of the first absolute value and the second absolute value as the stability feature of the j-th data in the node.
4. The sensor data collection and processing method based on the Internet of Things according to claim 1 is characterized in that: The similarity is obtained by calculating the formula: The cosine similarity of the central feature vector corresponding to the node center of the i-th node and each node except the i-th node is obtained as the similarity between the i-th node and each node except the i-th node.
5. The sensor data collection and processing method based on the Internet of Things according to claim 1 is characterized in that: The obtaining of the possibility of dividing each node into a low-density node includes: Get the mean of the association degree between the ith node and each node except the ith node as the importance of the ith node; Get the possibility of dividing each node into low-density nodes according to the importance of each node: ; In the formula, Represents the possibility of the i-th node being divided into a low-density node; Represents the discrete degree of the i-th node; Represents the importance of the i-th node; norm() represents the normalization function.
6. The sensor data collection and processing method based on the Internet of Things according to claim 1 is characterized in that: The calculation formula for obtaining low-density nodes and high-density nodes is: A possibility threshold T1 is preset. If the possibility of any node being classified as a low-density node is greater than or equal to the possibility threshold T1, the node is a low-density node, otherwise it is a high-density node.
7. The sensor data acquisition and processing system based on the Internet of Things is characterized by: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the sensor data collection and processing method based on the Internet of Things according to any one of claims 1 to 6 is implemented.
