IoT platform data cleaning method and system
Through dynamic segmentation and merging analysis of the working mode of IoT devices, adjusting the number of wavelet decomposition layers according to the noise intensity, solving the problem that the fixed decomposition layer cannot eliminate noise, and improving the effect of data cleaning.
Patent Information
- Application Number
- CN202410992515.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-23
AI Technical Summary
When existing methods only perform wavelet denoising based on fixed decomposition layers, they cannot completely eliminate noise interference in data collected by IoT devices, reducing the effect of data cleaning.
By obtaining the electrical data and temperature data sequences of IoT devices within the preset time period, analyzing the working modes of different time periods in segments, combining time periods of similar modes, dynamically adjusting the number of wavelet decomposition layers according to the noise intensity, and performing wavelet denoising.
It effectively eliminates noise interference, improves the accuracy and effect of data cleaning, and adapts to changes in electrical data in different time periods.
Smart Images

Figure CN119066329A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data cleaning, and in particular to a method and system for cleaning data on an IoT platform. Background Art
[0002] The Internet of Things (IoT) platform is a software system used to manage and connect IoT devices. It allows IoT devices to collect various electrical data and perform data analysis to achieve automated monitoring of the operating status of IoT devices. However, the data collection process is easily affected by other devices and the surrounding electric field, resulting in noise in the collected data, affecting the subsequent monitoring accuracy. Therefore, the collected data needs to be cleaned to improve the data quality.
[0003] In the related technology, wavelet denoising is usually performed on the collected data based on a fixed number of decomposition layers to achieve data cleaning. However, since the electrical data of IoT devices change differently in different time periods and the electrical data in different time periods are subject to different intensities of noise interference, the existing methods cannot completely eliminate the interference of noise when only performing wavelet denoising based on a fixed number of decomposition layers, thereby reducing the effect of data cleaning. Summary of the invention
[0004] In order to solve the technical problem that the existing method cannot completely eliminate the interference of noise and reduces the effect of data cleaning when performing wavelet denoising based only on a fixed number of decomposition layers, the purpose of the present invention is to provide an IoT platform data cleaning method and system, and the technical solutions adopted are as follows:
[0005] The present invention proposes a method for cleaning IoT platform data, the method comprising:
[0006] Acquire an electrical data sequence and a temperature data sequence of the IoT device to be tested within a preset time period, wherein the electrical data sequence includes electrical data at different times, and the temperature data sequence includes temperature data at different times;
[0007] According to the changes of electrical data at different times in the electrical data sequence, the preset time length is segmented to obtain different reference time periods; according to the fluctuations of electrical data at different times in each reference time period, adjacent reference time periods are merged to obtain different working time periods of the IoT device to be tested;
[0008] Taking any working period as the target working period, taking any one of the electrical data sequence and the temperature data sequence as the target sequence, the target sequence includes target data at different moments, and obtaining the local variation characteristic value of the target sequence at each moment in the target working period according to the difference of the target data at each moment in the preset short-time window of the target sequence at each moment in the target working period; obtaining the noise intensity of the electrical data sequence in the target working period according to the difference of the local variation characteristic value between the electrical data sequence and the temperature data sequence at the same moment in the target working period; obtaining the number of decomposition layers of the target working period according to the noise intensity of the target working period;
[0009] Based on the decomposition level number of each working period, wavelet denoising is performed on the electrical data of the electrical data sequence in each working period to obtain a pure electrical data sequence.
[0010] Furthermore, the step of merging adjacent reference time periods according to the fluctuation of electrical data at different times in each reference time period to obtain different working time periods of the IoT device to be tested includes:
[0011] Taking any reference period as the target reference period, obtaining the data fluctuation amount at each moment in the target reference period according to the difference of electrical data between each moment in the target reference period and the next adjacent moment;
[0012] Taking any two adjacent moments in the target reference period as an adjacent moment group, and obtaining a data fluctuation parameter of each adjacent moment group according to a difference in the data fluctuation amounts of the two moments in each adjacent moment group;
[0013] Analyzing the overall level of the data fluctuation parameters of all the adjacent time groups in the target reference period to obtain the data fluctuation degree of the target reference period;
[0014] Taking any two adjacent reference time periods as an adjacent reference time period group, and obtaining the possibility of merging each adjacent reference time period group according to the difference in the data fluctuation degree of the two reference time periods in each adjacent reference time period group and the difference in the electrical data between the two reference time periods;
[0015] Two reference time periods in the group of adjacent reference time periods whose merging possibility is greater than a preset merging threshold are connected end to end, and the interconnected reference time periods are used as a working time period.
[0016] Further, the obtaining of the possibility of merging each adjacent reference time period group includes:
[0017] Obtaining a fluctuation difference parameter for each adjacent reference time period group according to a difference in the data fluctuation degree of two reference time periods in each adjacent reference time period group;
[0018] Performing dynamic time warping processing on the electrical data between two reference time periods in each adjacent reference time period group to obtain the best matching distance for each adjacent reference time period group;
[0019] The fluctuation difference parameter and the best matching distance of each adjacent reference time period group are integrated and negatively correlated normalized to obtain the merging possibility of each adjacent reference time period group.
[0020] Further, the obtaining of the local variation characteristic value of the target sequence at each moment in the target working period includes:
[0021] Take any moment in the target working period as the moment to be tested;
[0022] In a preset short-time window at the time to be tested, analyzing the deviation of the target data of the target sequence at the time to be tested relative to the overall level of the target data at all times in the preset short-time window, and obtaining the data deviation value of the target sequence at the time to be tested;
[0023] According to the difference of the target data between each moment and the next adjacent moment in the preset short-time window, the degree of data change at each moment is obtained;
[0024] Analyzing the overall level of the data change degree of the target sequence at all moments in the preset short-time window of the time to be tested, to obtain the overall data change degree of the preset short-time window;
[0025] Obtaining the data change difference of the target sequence at the time to be tested according to the difference between the data change degree of the target sequence at the time to be tested and the overall data change degree of the preset short-time window;
[0026] The data deviation value and the data change difference of the target sequence at the time to be tested are integrated to obtain the local change characteristic value of the target sequence at the time to be tested in the target working period.
[0027] Further, obtaining the data deviation value of the target sequence at the time to be tested includes:
[0028] Analyze the overall level of target data of the target sequence at all moments in the preset short-time window of the time to be tested to obtain the overall data value of the preset short-time window;
[0029] According to the difference between the overall data value and the target data of the target sequence at the time to be tested, the data deviation value of the target sequence at the time to be tested is obtained.
[0030] Further, obtaining the noise intensity of the electrical data sequence in the target working period includes:
[0031] Obtaining a noise influence coefficient of the electrical data sequence at the time to be measured in the target working period according to a difference in the local change characteristic value between the electrical data sequence and the temperature data sequence at the time to be measured in the target working period;
[0032] The overall level of the noise influence coefficient of the electrical data sequence at all moments in the target working period is analyzed and normalized to obtain the noise intensity of the electrical data sequence in the target working period.
[0033] Further, obtaining the number of decomposition layers of the target working period according to the noise intensity of the target working period includes:
[0034] The product value of the noise intensity of the target working period and the preset hyperparameter is rounded to obtain the number of decomposition layers of the target working period.
[0035] Furthermore, based on the number of decomposition layers in each working period, performing wavelet denoising on the electrical data of the electrical data sequence in each working period to obtain a pure electrical data sequence includes:
[0036] Based on the wavelet threshold denoising algorithm, the electrical data of the electrical data sequence in each working period is decomposed and reconstructed according to the decomposition layer number of each working period to obtain a pure electrical data sequence.
[0037] Furthermore, segmenting the preset duration according to the changes of the electrical data at different moments in the electrical data sequence to obtain different reference time periods includes:
[0038] Based on the STL decomposition algorithm, the electrical data series is decomposed to obtain the seasonal curve corresponding to the electrical data series;
[0039] Obtain extreme value points on the seasonal curve, wherein the extreme value points include maximum value points and minimum value points;
[0040] The time period between the moments corresponding to any two adjacent extreme value points on the seasonal curve is used as a reference period in the preset time length.
[0041] The present invention also proposes an IoT platform data cleaning system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of an IoT platform data cleaning method when executing the computer program.
[0042] The present invention has the following beneficial effects:
[0043] The present invention takes into account that the existing methods cannot completely eliminate the interference of noise when performing wavelet denoising based only on a fixed number of decomposition layers, thereby reducing the effect of data cleaning. Therefore, the present invention first obtains the electrical data sequence and temperature data sequence of the IoT device to be tested within a preset time length. Since the IoT device to be tested is usually able to adaptively adjust the working mode, resulting in large differences in the changes in electrical data in different time periods, the present invention preliminarily segments the preset time length to obtain different reference time periods. Considering that the IoT device to be tested has multiple working modes, and a specific working mode may last for a long time, resulting in multiple adjacent reference time periods belonging to the same working mode. At the same time, considering that the fluctuations of electrical data under the same working mode are relatively similar, the adjacent reference time periods can be further merged to obtain multiple working time periods, each of which represents a time period in which the IoT device to be tested works in a specific working mode. Subsequently, the interference intensity of noise can be analyzed in each working period to avoid the situation where large changes in electrical data are caused by changes in the working mode of the Internet device to be tested, thereby eliminating the influence of the change in the working mode of the Internet device to be tested on the analysis of noise interference intensity. Considering that under ideal conditions without noise interference, the changes in electrical data and temperature data in the target working period have a certain correlation, and noise will break this correlation, the present invention first reflects the local changes of the target data of the target sequence at each moment of the target working period through the acquired local change characteristic values, and then reflects the intensity of noise interference to the electrical data sequence in the target working period through the acquired noise intensity, so as to facilitate the subsequent setting of different decomposition layers for the electrical data of each working period based on the noise intensity, and perform wavelet denoising on the electrical data of each working period, so as to more effectively eliminate the interference of noise and improve the effect of cleaning the electrical data. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flow chart of a method for cleaning IoT platform data provided by an embodiment of the present invention;
[0046] Figure 2 A flow chart of a method for obtaining different working time periods provided by an embodiment of the present invention;
[0047] Figure 3A flow chart of a method for acquiring local variation characteristic values of a target sequence at each moment in a target working period provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the IoT platform data cleaning method and system proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0050] The following is a detailed description of a method and system for cleaning IoT platform data provided by the present invention in conjunction with the accompanying drawings.
[0051] See also Figure 1 , which shows a flow chart of an IoT platform data cleaning method provided by an embodiment of the present invention, the method comprising:
[0052] Step S1: obtaining an electrical data sequence and a temperature data sequence of the IoT device to be tested within a preset time period, wherein the electrical data sequence includes electrical data at different times, and the temperature data sequence includes temperature data at different times.
[0053] The Internet of Things platform is a software system for managing and connecting Internet of Things devices. It allows Internet of Things devices to collect various electrical data, such as current or voltage data, and perform data analysis to achieve automatic monitoring of the operating status of Internet of Things devices. However, during the data collection process, it is easily affected by other devices and the electric field of the surrounding environment, resulting in noise in the collected data, affecting the subsequent monitoring accuracy. Therefore, the collected data needs to be cleaned to improve the data quality. In the related art, wavelet denoising is usually performed on the collected data based on a fixed number of decomposition layers to achieve data cleaning. However, since Internet of Things devices usually have multiple working modes, there are large differences in the changes in electrical data during the continuous working time in different modes, and the electrical data in different time periods are subject to different intensities of noise interference. As a result, when the existing method only performs wavelet denoising on the electrical data based on a fixed number of decomposition layers, it is impossible to completely eliminate the interference of noise, thereby reducing the effect of data cleaning. Therefore, an embodiment of the present invention proposes an IoT platform data cleaning method to solve this problem.
[0054] In reality, there are many devices that can be interconnected through the Internet of Things, especially some household electrical appliances, such as refrigerators or air conditioners. The embodiment of the present invention takes a refrigerator as an example, and takes the refrigerator as the Internet of Things device to be tested in the embodiment of the present invention. First, a current or voltage sensor is used to collect an electrical data sequence of the Internet of Things device to be tested within a preset time length. The time interval for electrical data collection is set to 0.1 seconds. The time interval for data collection can also be set by the implementer according to the specific implementation scenario, which is not limited here. At this time, the electrical data sequence contains electrical data at different times, wherein the electrical data may be data about current or voltage.
[0055] At the same time, in order to facilitate the subsequent analysis of the interference intensity of the noise, the embodiment of the present invention also needs to use a high-sensitivity temperature sensor arranged at the condenser tube inside the IoT device to be tested, i.e., the refrigerator, to collect a temperature data sequence of the IoT device to be tested within a preset time period. At this time, the temperature data sequence includes temperature data at different times, wherein the time interval for collecting the temperature data is the same as the time interval for collecting the electrical data.
[0056] The value range of the preset duration is set to [0.5, 1], in hours. In one embodiment of the present invention, the preset duration is set to 0.5 hours. The specific value of the preset duration can also be set by the implementer according to the specific implementation scenario, which is not limited here.
[0057] Step S2: According to the changes of electrical data at different times in the electrical data sequence, the preset time length is segmented to obtain different reference time periods; according to the fluctuations of electrical data at different times in each reference time period, adjacent reference time periods are merged to obtain different working time periods of the IoT device to be tested.
[0058] Considering that devices interconnected by the Internet of Things usually have basic intelligent functions, for example, for a refrigerator, there may be multiple working modes, and the working mode can be adaptively adjusted during operation to achieve energy saving. That is to say, the electric power of the IoT device to be tested is different in different working modes, resulting in large differences in the changes in electrical data in different working modes. Therefore, the embodiment of the present invention first analyzes the changes in electrical data at different times in the electrical data sequence, and segments the preset time length to obtain different reference time periods, so as to facilitate the subsequent further analysis of the electrical data of each reference time period, thereby obtaining the electrical data under the same working mode, thereby eliminating the influence of the working mode change factor on the noise intensity analysis.
[0059] Preferably, in an embodiment of the present invention, a method for obtaining different reference time periods specifically includes:
[0060] Since the IoT devices to be tested have multiple working modes, the changes in electrical data are similar under the same working mode, and there are differences in the changes in electrical data under different working modes. In addition, the IoT devices to be tested switch between multiple modes during operation, which causes the changes in electrical data to have a certain periodicity. Therefore, the electrical data sequence can be first decomposed based on the STL decomposition algorithm to obtain the seasonal curve corresponding to the electrical data sequence. The seasonal curve can reflect the periodicity of the changes in electrical data. The STL decomposition algorithm is a technical means well known to those skilled in the art and will not be elaborated here.
[0061] Then, based on Newton's method, extreme points on the seasonal curve are obtained, wherein the extreme points include maximum points and minimum points, and the time period between the moments corresponding to any two adjacent extreme points on the seasonal curve is used as a reference period in the preset time length. Using Newton's method to extract extreme points is a technical means well known to those skilled in the art and will not be elaborated here. In other embodiments of the present invention, extreme points can also be extracted based on other methods.
[0062] Since the IoT device to be tested has multiple working modes and a specific working mode may last for a long time, multiple adjacent reference time periods may belong to the same working mode. Considering that the fluctuations of electrical data under the same working mode are relatively similar, the fluctuations of electrical data at different times in each reference time period can be further analyzed, so as to merge adjacent reference time periods to obtain different working time periods of the IoT device to be tested. Each working time period represents the time period when the IoT device to be tested works in a specific working mode. Subsequently, the interference intensity of noise can be analyzed in each working time period to avoid the situation where large changes in electrical data are caused by changes in the working mode of the Internet device to be tested, thereby eliminating the influence of the change in the working mode of the Internet device to be tested on the noise interference intensity analysis, improving the accuracy of the noise interference intensity analysis, and thus improving the effect of electrical data cleaning.
[0063] Preferably, in one embodiment of the present invention, the method for acquiring different working periods of the IoT device to be tested specifically includes:
[0064] See also Figure 2 , which shows a flow chart of a method for obtaining different working time periods provided by an embodiment of the present invention.
[0065] Step S201: taking any reference period as a target reference period, and obtaining the data fluctuation amount at each moment in the target reference period according to the difference in electrical data between each moment in the target reference period and the next adjacent moment.
[0066] From the above analysis, it can be seen that multiple adjacent reference time periods may belong to the same working mode, and the fluctuations of the electrical data of multiple reference time periods belonging to the same working mode are relatively similar. Therefore, we can first analyze any reference time period, take any reference time period as the target reference time period, and obtain the data fluctuation amount at each moment in the target reference time period based on the difference in electrical data between each moment in the target reference time period and the next adjacent moment, so as to provide a data basis for the subsequent calculation and analysis of the data fluctuation of the target reference time period.
[0067] In one embodiment of the present invention, the absolute value of the difference in electrical data between each moment in the target reference period and the next adjacent moment can be used as the data fluctuation amount at each moment in the target reference period to analyze the difference in electrical data between each moment in the target reference period and the next adjacent moment.
[0068] It should be noted that there are boundary problems in the process of calculating data fluctuations. For the last moment in the target reference period, there are no other moments belonging to the target reference period. At this time, it can be divided into two cases. First, if the target reference period is not the last reference period, the absolute value of the difference in electrical data between the last moment in the target reference period and the first moment in the next adjacent reference period can be used as the data fluctuation of the last moment in the target reference period. Second, if the target reference period is the last reference period, there are no other reference periods after the target reference period. At this time, the electrical data of the next moment of the last moment in the target reference period can be considered to be 0, and the electrical data of the last moment in the target reference period can be directly used as the data fluctuation of that moment. The boundary problems in the subsequent calculation process can also be solved by the above method.
[0069] Step S202: Take any two adjacent moments in the target reference period as an adjacent moment group, and obtain the data fluctuation parameter of each adjacent moment group according to the difference in the data fluctuation amount between the two moments in each adjacent moment group; analyze the overall level of the data fluctuation parameters of all adjacent moment groups in the target reference period to obtain the data fluctuation degree of the target reference period.
[0070] After obtaining the data fluctuation amount at each moment in the target reference time period, the greater the difference in the data fluctuation amount between adjacent moments in the target reference time period, the greater the fluctuation degree of the electrical data in the target reference time period. Therefore, any two adjacent moments in the target reference time period can be first taken as adjacent moment groups, and the difference in the data fluctuation amount between the two moments in each adjacent moment group can be analyzed to obtain the data fluctuation parameter of each adjacent moment group. The larger the data fluctuation parameter, the greater the local fluctuation degree of the electrical data between the two moments in the adjacent moment group. Then, the overall level of the data fluctuation parameters of all adjacent moment groups in the target reference time period is analyzed to obtain the data fluctuation degree of the target reference time period. The greater the data fluctuation degree, the greater the fluctuation degree of the electrical data in the target reference time period. Subsequently, based on the data fluctuation degree, the similarity of the fluctuation of the electrical data between the reference time periods can be analyzed to realize the merging of the reference time periods.
[0071] In one embodiment of the present invention, the absolute value of the difference between the data fluctuation amounts of two moments in each adjacent moment group can be used as the data fluctuation parameter of each adjacent moment group to analyze the difference between the data fluctuation amounts of two moments in each adjacent moment group.
[0072] In an embodiment of the present invention, the average value or median of the data fluctuation parameters of all adjacent moment groups in the target reference period can be used as the data fluctuation degree of the target reference period to realize the overall level analysis of the data fluctuation parameters of all adjacent moment groups in the target reference period, which is not limited here.
[0073] The expression of the data fluctuation degree of the target reference period can be specifically, for example, as follows:
[0074]
[0075] Where F represents the data fluctuation degree of the target reference period; w n and w' n They respectively represent the data fluctuation amount of two moments in the nth adjacent moment group of the target reference period; N represents the number of all moments in the target reference period, and N-1 represents the number of all adjacent moment groups in the target reference period.
[0076] The data fluctuation degree of each reference period can be obtained by the same method as above.
[0077] Step S203: taking any two adjacent reference periods as an adjacent reference period group, and obtaining the possibility of merging each adjacent reference period group according to the difference in data fluctuation degree between the two reference periods in each adjacent reference period group and the difference in electrical data between the two reference periods.
[0078] From the above analysis, it can be seen that the fluctuations of the electrical data of multiple reference time periods belonging to the same working mode are relatively similar, and the IoT device to be tested works continuously in a certain fixed mode, that is, multiple adjacent reference data segments may belong to the same working mode. Therefore, any two adjacent reference time periods can be first taken as adjacent reference time period groups, and the difference in the degree of data fluctuation of the two reference time periods in each adjacent reference time period group and the difference in electrical data between the two reference time periods are analyzed to obtain the possibility of merging each adjacent reference time period group. The greater the possibility of merging, the more similar the fluctuations of the electrical data of the two reference time periods in the adjacent reference time period group are, which further indicates that the more likely the two reference time periods in the adjacent reference time period group are to belong to the same working mode, the greater the possibility of merging the two reference time periods.
[0079] Preferably, in one embodiment of the present invention, a method for obtaining the possibility of merging each adjacent reference time period group specifically includes:
[0080] First, the difference in data fluctuation levels between the two reference time periods in adjacent reference time period groups indicates that the less similar the fluctuations in the electrical data of the two reference time periods are, the smaller the possibility that they belong to the same working mode is. Therefore, the fluctuation difference parameter of each adjacent reference time period group can be obtained based on the difference in data fluctuation levels between the two reference time periods in each adjacent reference time period group. The smaller the fluctuation difference parameter, the smaller the difference in electrical data fluctuations between the two reference time periods in the adjacent reference time period group, the greater the possibility that they belong to the same working mode, and the greater the possibility that the two reference time periods are merged, providing a data basis for the subsequent calculation of the possibility of merging adjacent reference time period groups.
[0081] In one embodiment of the present invention, the absolute value of the difference in data fluctuation levels between two reference time periods in each adjacent reference time period group can be used as a fluctuation difference parameter for each adjacent reference time period group to analyze the difference in data fluctuation levels between the two reference time periods in each adjacent reference time period group.
[0082] Then, the electrical data between the two reference time periods in each adjacent reference time period group are dynamically time-warped to obtain the best matching distance for each adjacent reference time period group. The smaller the best matching distance is, the greater the similarity of the electrical data of the two reference time periods in the adjacent reference time period group is, and the greater the possibility of belonging to the same working mode is, the greater the possibility of merging the two reference time periods is, which provides a data basis for the subsequent calculation of the possibility of merging adjacent reference time period groups.
[0083] Among them, dynamic time warping is a technical means well known to technical personnel in this field and will not be elaborated here.
[0084] Finally, the fluctuation difference parameters and the best matching distance of each adjacent reference period group are integrated and normalized with negative correlation to obtain the merging possibility of each adjacent reference period group.
[0085] In the embodiment of the present invention, the fluctuation difference parameter and the best matching distance of each adjacent reference time period group may be multiplied or added to achieve a combination of the two, which is not limited here.
[0086] In an embodiment of the present invention, a negative exponential function with a natural constant e as the base or a function form of 1-norm() can be used to implement normalization processing of negative correlation, which is not limited here, wherein norm() represents a normalization function. In one embodiment of the present invention, the normalization processing can be specifically, for example, maximum and minimum value normalization processing, and the normalization in subsequent steps can adopt maximum and minimum value normalization processing. In other embodiments of the present invention, other normalization methods can be selected according to the specific range of numerical values, which will not be described in detail.
[0087] The expression of the possibility of merging each adjacent reference time period group can be specifically, for example, as follows:
[0088] C i =1-norm(|F i -F' i |×DTW i )
[0089] Among them, C i represents the possibility of merging the i-th adjacent reference time period group; F i and F' i Respectively represent the data fluctuation degree of two reference time periods in the i-th adjacent reference time period group; |F i -F' i | represents the fluctuation difference parameter of the i-th adjacent reference period group; DTW i represents the best matching distance of the i-th adjacent reference time period group; norm() represents the normalization function.
[0090] Step S204: two reference time periods in a group of adjacent reference time periods whose merging possibility is greater than a preset merging threshold are connected end to end, and the connected reference time periods are used as a working time period.
[0091] From the above analysis, it can be seen that the greater the possibility of merging, the greater the similarity of the electrical data of the two reference time periods in the adjacent reference time period group, the greater the possibility of belonging to the same working mode, and the greater the possibility of merging the two. Therefore, the two reference time periods in the adjacent reference time period group whose merging possibility is greater than the preset merging threshold can be connected end to end, and the interconnected reference time periods are regarded as one working period. The preset merging threshold is set to 0.7. The specific value of the preset merging threshold can also be set by the implementer according to the specific implementation scenario, and is not limited here.
[0092] It should be noted that if a reference time period does not meet the merging conditions with the adjacent reference time period, the reference time period will be treated as a working time period alone. For example: assuming that there are four reference time periods A, B, C, and D, the adjacent reference time period groups are AB, BC, and CD respectively. If the two reference time periods A and B in the adjacent reference time period group AB meet the merging conditions, A and B will be connected end to end to form a larger time period. If the two reference time periods B and C in the adjacent reference time period group BC meet the merging conditions, B and C will be connected end to end. If the two reference time periods C and D in the adjacent reference time period group CD do not meet the merging conditions, C and D will not be connected end to end. At this time, A and B are connected end to end, and B and C are connected end to end. The reference time periods A, B, and C are treated as a working time period, but C and D are not connected end to end. In this case, the reference time period D will be treated as a working time period alone.
[0093] At this point, multiple working periods of the IoT device to be tested are obtained. Each working period represents the time period when the IoT device to be tested works in a specific working mode. Subsequently, the interference intensity of the noise can be accurately analyzed in each working period to improve the effect of electrical data cleaning.
[0094] Step S3: Take any working period as the target working period, and take any one of the electrical data sequence and the temperature data sequence as the target sequence, the target sequence contains target data at different times, and obtain the local change characteristic value of the target sequence at each moment in the target working period according to the difference in the target data of the target sequence at each moment in the preset short-time window at each moment in the target working period; obtain the noise intensity of the electrical data sequence in the target working period according to the difference in the local change characteristic values between the electrical data sequence and the temperature data sequence at the same moment in the target working period; obtain the number of decomposition layers of the target working period according to the noise intensity of the target working period.
[0095] The purpose of the embodiment of the present invention is to analyze the intensity of noise interference on electrical data in different time periods, so as to achieve more effective data cleaning. The interference of noise will change the change of electrical data. At the same time, the change of electrical data will also be changed when multiple working modes of the Internet of Things device to be tested are converted to each other. Therefore, in order to eliminate the influence of the factor of multiple working modes of the Internet device to be tested switching to each other on the subsequent noise interference intensity analysis, it is necessary to analyze any working period in the future, and first take any working period as the target working period.
[0096] Considering that under ideal conditions without noise interference, the changes in electrical data and temperature data within the target working period have a certain correlation. For example, when the working current of the refrigerator increases, the circulation rate of the refrigerator's refrigeration pipe increases, thereby increasing the temperature reduction rate, so that the refrigerator can cool down quickly. Since noise only affects the electrical data but not the temperature data, the existence of noise will break the correlation between the changes in electrical data and temperature data within the target working period. Therefore, the embodiment of the present invention first takes any one of the electrical data sequence and the temperature data sequence as the target sequence, and the target sequence contains target data at different times. If the target sequence is an electrical data sequence, the target data in the target sequence is electrical data. If the target sequence is a temperature data sequence, the target data in the target sequence is temperature data, and then The target data of the target sequence at each moment in the preset short-time window in the target working period are analyzed, and the local change characteristic value obtained reflects the local change of the target sequence at each moment in the target working period. Subsequently, based on the local change characteristic value, the difference in data change between the electrical data sequence and the temperature data sequence in the target working period can be further analyzed, so as to accurately calculate the intensity of noise interference to the electrical data in the target working period. The length of the preset short-time window is set to 7, that is, the preset short-time window for each moment in the target working period includes the moment itself and the 6 other moments closest to the moment. At the same time, all moments included in the preset short-time window belong to the target working period. The length of the preset short-time window can also be set by the implementer according to the specific implementation scenario, which is not limited here.
[0097] Preferably, in one embodiment of the present invention, the method for acquiring the local change characteristic value of the target sequence at each moment in the target working period specifically includes:
[0098] See also Figure 3 , which shows a flow chart of a method for obtaining a local change characteristic value of a target sequence at each moment in a target working period provided by an embodiment of the present invention.
[0099] Step S301: Take any moment in the target working period as the moment to be tested, and analyze the deviation of the target data of the target sequence at the moment to be tested relative to the overall level of the target data of all moments in the preset short-time window in the preset short-time window in the moment to be tested, so as to obtain the data deviation value of the target sequence at the moment to be tested.
[0100] First, it is necessary to analyze the deviation of the target data at each moment in the target working period relative to the local target data, take any moment in the target working period as the test moment, and then analyze the deviation of the target data of the target sequence at the test moment relative to the overall level of the target data of all moments in the preset short-time window, and obtain the data deviation value of the target sequence at the test moment. The larger the data deviation value, the more the target data of the target sequence at the test moment deviates from other local target data, which provides a data basis for the subsequent calculation of the local change characteristic value.
[0101] Preferably, in one embodiment of the present invention, the method for obtaining the data deviation value of the target sequence at the time to be tested specifically includes:
[0102] First, the overall level of the target data of the target sequence at all moments in the preset short-time window of the time to be tested is analyzed to obtain the overall data value of the preset short-time window, which reflects the overall level of all target data in the preset short-time window.
[0103] In an embodiment of the present invention, the average value or median of the target data of the target sequence at all moments in the preset short-time window of the time to be tested can be used as the overall data value of the preset short-time window to realize the analysis of the overall level of the target data of the target sequence at all moments in the preset short-time window of the time to be tested, which is not limited here.
[0104] Then, according to the difference between the overall data value and the target data of the target sequence at the time to be tested, the data deviation value of the target sequence at the time to be tested is obtained.
[0105] In one embodiment of the present invention, the absolute value of the difference between the overall data value and the target data of the target sequence at the time to be tested may be used as the data deviation value of the target sequence at the time to be tested to analyze the difference between the two.
[0106] Step S302: Obtain the degree of data change at each moment based on the difference in target data between each moment in a preset short-time window and the next adjacent moment; analyze the overall level of data change of the target sequence at all moments in the preset short-time window at the moment to be tested to obtain the overall data change degree of the preset short-time window; obtain the data change difference of the target sequence at the moment to be tested based on the difference between the data change degree of the target sequence at the moment to be tested and the overall data change degree of the preset short-time window.
[0107] Secondly, it is necessary to analyze the difference in the degree of change of the target data at each moment in the target working period relative to the local target data, and obtain the degree of data change at each moment according to the difference in target data between each moment and the next adjacent moment in the preset short-time window. The greater the data change degree, the greater the change in the target data after each moment transitions to the next moment. Then, the overall level of the data change degree of the target sequence at all moments in the preset short-time window of the test moment is analyzed to obtain the overall data change degree of the preset short-time window, and according to the difference between the data change degree of the target sequence at the test moment and the overall data change degree of the preset short-time window, the data change difference degree of the target sequence at the test moment is obtained. The greater the data change difference degree, the greater the difference in the degree of change of the target data at the test moment compared with the local target data. Subsequently, the data change difference degree and the data deviation value obtained above can be combined to accurately calculate the local change characteristic value of the target sequence at the test moment in the target working period.
[0108] In one embodiment of the present invention, the absolute value of the difference between the target data at each moment in a preset short-time window and the next adjacent moment can be used as the degree of data change at each moment to analyze the difference between the two. It should be noted that there will be boundary problems in the process of calculating the degree of data change, which can be solved using a similar method as mentioned above and will not be elaborated here.
[0109] In an embodiment of the present invention, the average value or median of the data change degree of the target sequence at all moments in the preset short-time window of the time to be tested can be used as the overall data change degree of the preset short-time window, so as to realize the overall level analysis of the data change degree of the target sequence at all moments in the preset short-time window of the time to be tested, which is not limited here.
[0110] In one embodiment of the present invention, the absolute value of the difference between the data change degree of the target sequence at the time to be tested and the overall data change degree of the preset short-time window can be used as the data change difference degree of the target sequence at the time to be tested to analyze the difference between the two.
[0111] Step S303: synthesizing the data deviation value and data change difference of the target sequence at the time to be tested to obtain the local change characteristic value of the target sequence at the time to be tested in the target working period.
[0112] The larger the data deviation value of the target sequence at the time to be tested, the greater the deviation of the target data at the time to be tested relative to the local target data. At the same time, the larger the data change difference of the target sequence at the time to be tested, the greater the difference between the change of the target data at the time to be tested and the change of the local target data. Therefore, the data deviation value and the data change difference of the target sequence at the time to be tested can be integrated, and the local change of the target data at the time to be tested in the target working period can be reflected by the obtained local change characteristic value.
[0113] In the embodiment of the present invention, the product value or sum of the data deviation value and the data change difference of the target sequence at the time to be tested can be used as the local change characteristic value of the target sequence at the time to be tested in the target working period to achieve the integration of the two, which is not limited here.
[0114] The expression of the local variation characteristic value of the target sequence at the time to be measured in the target working period can be specifically, for example, as follows:
[0115]
[0116] Wherein, A represents the local change characteristic value of the target sequence at the time to be tested in the target working period; D represents the target data of the target sequence at the time to be tested; Represents the overall data value of the preset short-time window corresponding to the time to be tested; represents the data deviation value of the target sequence at the time to be tested; E represents the degree of data change of the target sequence at the time to be tested; Indicates the overall data change degree of the preset short-term window corresponding to the time to be tested; Indicates the data change difference of the target sequence at the time to be tested.
[0117] The same method as above can be used to obtain the local change characteristic value of the target sequence at each moment in the target working period. At the same time, since the target sequence is a sequence selected arbitrarily from the electrical data sequence and the temperature data sequence, that is, the above method can be used to analyze the electrical data sequence and the temperature data sequence respectively, so as to obtain the local change characteristic value of the electrical data sequence at each moment in the target working period, and the local change characteristic value of the temperature data sequence at each moment in the target working period.
[0118] From the above analysis, it can be seen that the presence of noise will break the correlation between the changes in electrical data and temperature data during the target working period, resulting in a large difference in the changes in electrical data and temperature data during the target working period. Specifically, the difference in the local change characteristic values between the electrical data sequence and the temperature data sequence at the same time in the target working period is large, and the greater the interference intensity of the noise, the greater the difference will be. Therefore, the difference in the local change characteristic values between the electrical data sequence and the temperature data sequence at the same time in the target working period can be analyzed to obtain the noise intensity of the electrical data sequence in the target working period. The greater the noise intensity, the greater the intensity of noise interference to the electrical data in the target working period, and more effective data cleaning can be performed based on the noise intensity in the future.
[0119] Preferably, in one embodiment of the present invention, the method for acquiring the noise intensity of the electrical data sequence in the target working period specifically includes:
[0120] According to the difference in the local change characteristic values between the electrical data sequence and the temperature data sequence at the time to be measured in the target working period, the noise influence coefficient of the electrical data sequence at the time to be measured in the target working period is obtained. The larger the noise influence coefficient, the greater the degree to which the electrical data at the time to be measured is affected by noise.
[0121] In one embodiment of the present invention, the absolute value of the difference between the local change characteristic values of the electrical data sequence and the temperature data sequence at the time to be measured in the target working period can be used as the noise influence coefficient of the electrical data sequence at the time to be measured in the target working period to analyze the difference between the two.
[0122] Then, the overall level of the noise influence coefficient of the electrical data sequence at all moments in the target working period is analyzed and normalized to obtain the noise intensity of the electrical data sequence in the target working period.
[0123] In an embodiment of the present invention, the overall level of the noise impact coefficient of the electrical data sequence at all moments in the target working period can be analyzed by calculating the average or median of the noise impact coefficient of the electrical data sequence at all moments in the target working period, which is not limited here.
[0124] The expression of the noise intensity of the electrical data sequence in the target working period can be specifically, for example, as follows:
[0125]
[0126] Where U represents the noise intensity of the electrical data sequence during the target working period; A (1,m) A represents the local change characteristic value of the electrical data sequence at the mth moment in the target working period;(2,m) represents the local change characteristic value of the temperature data sequence at the mth moment in the target working period; M represents the number of all moments in the target working period; norm() represents the normalization function.
[0127] Considering that when using the existing wavelet denoising method to clean electrical data, the traditional wavelet denoising method is to decompose the electrical data to separate the noise and reconstruct the noise-free real electrical data to achieve the cleaning of the electrical data. The traditional wavelet denoising method usually directly selects a fixed decomposition layer number to decompose the electrical data, and does not take into account the different intensities of noise interference to the electrical data in different time periods, resulting in the traditional wavelet denoising method being unable to completely eliminate the interference of noise. The greater the intensity of the noise interference to the electrical data, the more noise components exist in the electrical data. In order to more effectively separate the noise and achieve better denoising effect, it is necessary to increase the number of wavelet decomposition layers. Therefore, the number of decomposition layers of the target working period can be obtained according to the noise intensity of the target working period. Subsequently, the electrical data in different working periods can be subjected to wavelet denoising based on the number of decomposition layers, which can more effectively eliminate the noise existing in different working periods and improve the effect of cleaning the electrical data.
[0128] Preferably, in one embodiment of the present invention, the method for obtaining the number of decomposition layers of the target working period specifically includes:
[0129] From the above analysis, it can be seen that the greater the noise intensity of the target working period, the more decomposition layers are required. Therefore, the product value of the noise intensity of the target working period and the preset hyperparameter can be rounded to obtain the number of decomposition layers of the target working period.
[0130] The expression of the decomposition level of the target working period can be specifically, for example, as follows:
[0131]
[0132] Wherein, H represents the number of decomposition layers in the target working period; U represents the noise intensity of the electrical data sequence in the target working period; Represents the rounding up symbol; K represents a preset hyperparameter, and the value range of the preset hyperparameter is [5,15]. In one embodiment of the present invention, K is set to 10. The specific value of K can also be set by the implementer according to the specific implementation scenario, which is not limited here.
[0133] The number of decomposition layers for each working period can be obtained by the same method as above.
[0134] Step S4: Based on the number of decomposition layers of each working period, wavelet denoising is performed on the electrical data of the electrical data sequence in each working period to obtain a pure electrical data sequence.
[0135] After obtaining the number of decomposition layers of each working period, wavelet denoising can be performed on the electrical data of the electrical data sequence in each working period based on the number of decomposition layers of each working period, effectively eliminating the noise in different working periods, thereby obtaining a pure electrical data sequence and improving the effect of electrical data cleaning.
[0136] Preferably, in one embodiment of the present invention, the method for obtaining the number of decomposition layers of the target working period specifically includes:
[0137] Based on the wavelet threshold denoising algorithm, the electrical data of the electrical data sequence in each working period is decomposed and reconstructed according to the number of decomposition layers in each working period to obtain a pure electrical data sequence. The wavelet threshold denoising algorithm is a technical means well known to technicians in this field and will not be elaborated here.
[0138] An embodiment of the present invention provides an IoT platform data cleaning system, which includes a memory, a processor and a computer program, wherein the memory is used to store the corresponding computer program, and the processor is used to run the corresponding computer program. When the computer program runs in the processor, the method described in steps S1 to S4 can be implemented.
[0139] In summary, the embodiment of the present invention first obtains the electrical data sequence and the temperature data sequence of the IoT device to be tested within a preset time period, wherein the electrical data sequence includes electrical data at different times, and the temperature data sequence includes temperature data at different times; according to the changes of the electrical data at different times in the electrical data sequence, the preset time period is segmented to obtain different reference time periods; according to the fluctuations of the electrical data at different times in each reference time period, adjacent reference time periods are merged to obtain different working time periods of the IoT device to be tested; any working time period is taken as the target working time period, and any sequence of the electrical data sequence and the temperature data sequence is taken as the target sequence, and the target The sequence contains target data at different moments. According to the difference of target data at each moment in a preset short-time window of the target sequence at each moment in the target working period, the local change characteristic value of the target sequence at each moment in the target working period is obtained; according to the difference of local change characteristic values between the electrical data sequence and the temperature data sequence at the same moment in the target working period, the noise intensity of the electrical data sequence in the target working period is obtained; according to the noise intensity of the target working period, the number of decomposition layers of the target working period is obtained; based on the number of decomposition layers of each working period, the electrical data of the electrical data sequence in each working period is subjected to wavelet denoising to obtain a pure electrical data sequence.
[0140] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0141] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
Claims
1. A method for cleaning IoT platform data, characterized in that: The method comprises: Acquire an electrical data sequence and a temperature data sequence of the IoT device to be tested within a preset time period, wherein the electrical data sequence includes electrical data at different times, and the temperature data sequence includes temperature data at different times; According to the changes of electrical data at different times in the electrical data sequence, the preset time length is segmented to obtain different reference time periods; according to the fluctuations of electrical data at different times in each reference time period, adjacent reference time periods are merged to obtain different working time periods of the IoT device to be tested; Taking any working period as the target working period, taking any one of the electrical data sequence and the temperature data sequence as the target sequence, the target sequence includes target data at different moments, and obtaining the local variation characteristic value of the target sequence at each moment in the target working period according to the difference of the target data at each moment in the preset short-time window of the target sequence at each moment in the target working period; obtaining the noise intensity of the electrical data sequence in the target working period according to the difference of the local variation characteristic value between the electrical data sequence and the temperature data sequence at the same moment in the target working period; obtaining the number of decomposition layers of the target working period according to the noise intensity of the target working period; Based on the decomposition level number of each working period, wavelet denoising is performed on the electrical data of the electrical data sequence in each working period to obtain a pure electrical data sequence.
2. The IoT platform data cleaning method according to claim 1, characterized in that: The method of merging adjacent reference time periods according to the fluctuation of electrical data at different times in each reference time period to obtain different working time periods of the IoT device to be tested includes: Taking any reference period as the target reference period, obtaining the data fluctuation amount at each moment in the target reference period according to the difference of electrical data between each moment in the target reference period and the next adjacent moment; Taking any two adjacent moments in the target reference period as an adjacent moment group, and obtaining a data fluctuation parameter of each adjacent moment group according to a difference in the data fluctuation amounts of the two moments in each adjacent moment group; Analyzing the overall level of the data fluctuation parameters of all the adjacent time groups in the target reference period to obtain the data fluctuation degree of the target reference period; Taking any two adjacent reference time periods as an adjacent reference time period group, and obtaining the possibility of merging each adjacent reference time period group according to the difference in the data fluctuation degree of the two reference time periods in each adjacent reference time period group and the difference in the electrical data between the two reference time periods; Two reference time periods in the group of adjacent reference time periods whose merging possibility is greater than a preset merging threshold are connected end to end, and the interconnected reference time periods are used as a working time period.
3. The IoT platform data cleaning method according to claim 2, characterized in that: The possibility of obtaining a combination of each adjacent reference time period group includes: Obtaining a fluctuation difference parameter for each adjacent reference time period group according to a difference in the data fluctuation degree of two reference time periods in each adjacent reference time period group; Performing dynamic time warping processing on the electrical data between two reference time periods in each adjacent reference time period group to obtain the best matching distance for each adjacent reference time period group; The fluctuation difference parameter and the best matching distance of each adjacent reference time period group are integrated and negatively correlated normalized to obtain the merging possibility of each adjacent reference time period group.
4. The IoT platform data cleaning method according to claim 1, characterized in that: The step of obtaining the local variation characteristic value of the target sequence at each moment in the target working period includes: Take any moment in the target working period as the moment to be tested; In a preset short-time window at the time to be tested, analyzing the deviation of the target data of the target sequence at the time to be tested relative to the overall level of the target data at all times in the preset short-time window, and obtaining the data deviation value of the target sequence at the time to be tested; According to the difference of the target data between each moment and the next adjacent moment in the preset short-time window, the degree of data change at each moment is obtained; Analyzing the overall level of the data change degree of the target sequence at all moments in the preset short-time window of the time to be tested, to obtain the overall data change degree of the preset short-time window; According to the difference between the data change degree of the target sequence at the time to be tested and the overall data change degree of the preset short-time window, the data change difference degree of the target sequence at the time to be tested is obtained; The data deviation value and the data change difference of the target sequence at the time to be tested are integrated to obtain the local change characteristic value of the target sequence at the time to be tested in the target working period.
5. The IoT platform data cleaning method according to claim 4, characterized in that: The step of obtaining the data deviation value of the target sequence at the time to be tested includes: Analyze the overall level of target data of the target sequence at all moments in the preset short-time window of the time to be tested to obtain the overall data value of the preset short-time window; According to the difference between the overall data value and the target data of the target sequence at the time to be tested, the data deviation value of the target sequence at the time to be tested is obtained.
6. The IoT platform data cleaning method according to claim 4, characterized in that: The step of obtaining the noise intensity of the electrical data sequence during the target working period includes: Obtaining a noise influence coefficient of the electrical data sequence at the time to be measured in the target working period according to a difference in the local change characteristic value between the electrical data sequence and the temperature data sequence at the time to be measured in the target working period; The overall level of the noise influence coefficient of the electrical data sequence at all moments in the target working period is analyzed and normalized to obtain the noise intensity of the electrical data sequence in the target working period.
7. The IoT platform data cleaning method according to claim 1, characterized in that: The step of obtaining the number of decomposition levels of the target working period according to the noise intensity of the target working period comprises: The product value of the noise intensity of the target working period and the preset hyperparameter is rounded to obtain the number of decomposition layers of the target working period.
8. The IoT platform data cleaning method according to claim 1, characterized in that: The step of performing wavelet denoising on the electrical data of the electrical data sequence in each working period based on the number of decomposition layers in each working period to obtain a pure electrical data sequence includes: Based on the wavelet threshold denoising algorithm, the electrical data of the electrical data sequence in each working period is decomposed and reconstructed according to the decomposition layer number of each working period to obtain a pure electrical data sequence.
9. The IoT platform data cleaning method according to claim 1, characterized in that: The step of segmenting the preset duration according to the changes of the electrical data at different moments in the electrical data sequence to obtain different reference time periods includes: Based on the STL decomposition algorithm, the electrical data series is decomposed to obtain the seasonal curve corresponding to the electrical data series; Obtain extreme value points on the seasonal curve, wherein the extreme value points include maximum value points and minimum value points; The time period between the moments corresponding to any two adjacent extreme value points on the seasonal curve is used as a reference period in the preset time length.
10. An IoT platform data cleaning system, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
Citation Information
Patent Citations
Wind power prediction method and device and electronic equipment
CN114298132A
VOCs monitoring and data processing system based on Internet of Things
CN115184440A
X-ray thickness gauge data optimization and correction method
CN117171516A
Intelligent Internet of Things multimedia teaching management method and system
CN118197344A
Method and device for processing operating condition data of battery, and storage medium
WO2024113910A1
Cited By
Mining mine pressure intelligent analyzing and monitoring method based on machine learning
CN119474758A
Intelligent analysis and monitoring method of mine pressure based on machine learning
CN119474758B
Equipment operation state monitoring method and system based on Internet of Things
CN119807576A
