Wind direction sensor abnormal data detection method

Abstract

The application provides a wind direction sensor abnormal data detection method, comprising the following steps: acquiring Gray code data of a wind direction sensor; selecting one bit value, dividing the one bit value into data units with the same data according to time sequence, and counting the number of data in each data unit as a data point, and storing the data point into a to-be-detected data set; then, a non-replacement sampling is performed to generate a sub-sample, a plurality of isolated trees are constructed until all data points are sampled, and the skewness of the sub-sample is calculated; a split point is generated according to the skewness, the data of the sub-sample is recursively divided into two subsets until a stop condition is met; the average path length of each data point in the isolated tree is calculated to determine whether the data point is an abnormal data point; the foregoing steps are repeated until each bit value of the Gray code data is selected, the skewness is used to dynamically adjust the split strategy, the split point is closer to the abnormal data dense area, and the abnormal data elimination speed and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of meteorological monitoring technology, and in particular to a method for detecting abnormal data from a wind direction sensor. Background Technology

[0002] In the field of meteorological monitoring, data quality directly affects the accuracy of weather forecasts and the timeliness of disaster warnings. However, meteorological data is characterized by its multi-source nature, high dimensionality, and strong spatiotemporal correlation. Furthermore, it is susceptible to environmental interference (such as extreme weather and sensor malfunctions), which can easily generate abnormal data that needs to be removed. Summary of the Invention

[0003] The purpose of this invention is to provide a method for detecting abnormal data from wind direction sensors, which can efficiently eliminate abnormal data.

[0004] To address the above problems, this invention provides a method for detecting abnormal data from a wind direction sensor, comprising:

[0005] Step S10000: Obtain Gray code data from the wind direction sensor;

[0006] Step S20000: Select one bit of the Gray code data and divide it into several data units in chronological order, so that the data in each data unit is the same. Then count the number of data in each data unit and use it as a data point. Store the data point in the dataset to be detected.

[0007] Step S30000: Repeatedly sample a first number of data points without replacement from the dataset to be detected to generate subsamples and construct several isolation trees until the number of remaining data points in the dataset to be detected is insufficient to complete two samplings without replacement. Use the remaining data points in the dataset to be detected as subsamples to construct the isolation trees and calculate the skewness of each subsample.

[0008] Step S40000: Generate split points based on the skewness, recursively split the data of the subsample into two subsets, and continue to split the subsets until the stopping condition is met;

[0009] Step S50000: Calculate the average path length of each data point in the isolation tree, and determine whether the data point is an abnormal data point based on the average path length;

[0010] Step S60000: Select another value of the Gray code data, and repeat steps S30000 to S50000 until every value of the Gray code data has been selected.

[0011] Furthermore, the method for detecting abnormal data from the wind direction sensor described above, after step S60000, also includes:

[0012] Step S70000: Mark the Gray code data corresponding to the abnormal data points as abnormal data.

[0013] Furthermore, the formula for calculating the skewness of the sub-sample in step S30000 of the above-mentioned wind direction sensor abnormal data detection method is as follows:

[0014] (1)

[0015] in, The skewness of the subsample The size of the subsample, For the first The data points mentioned above, The mean of the subsample is given.

[0016] Furthermore, the rule for generating the segmentation points in step S40000 of the above-mentioned method for detecting abnormal data from wind direction sensors is as follows:

[0017] (2)

[0018] in, A random number between 0 and 1. The maximum value in the subset of the current isolation tree. It is the minimum value in the subset of the current isolation tree.

[0019] Furthermore, the stopping conditions in step S40000 of the above-mentioned wind direction sensor abnormal data detection method include: the subset has only one data point left, or all the data points have the same value, or the height of the isolation tree exceeds a preset value;

[0020] The preset value for the height of the isolation tree is:

[0021] (3)

[0022] in, The preset value is the height of the isolation tree. The size of the subsample generated by the first sampling without replacement.

[0023] Furthermore, step S50000 in the above-mentioned method for detecting abnormal data from wind direction sensors also includes:

[0024] Step S50100: Calculate the anomaly score for each data point based on the average path length, and determine whether the data point is an anomaly based on the anomaly score.

[0025] Furthermore, the formula for calculating the anomaly score in step S50100 of the above-mentioned wind direction sensor anomaly data detection method is as follows:

[0026] (4)

[0027] in, These are abnormal scores. This represents the average path length. This represents the path length of the data point within the current isolation tree. For the first A harmonic number, , It is a constant, usually .

[0028] By leveraging skewed dynamic adjustment of feature selection and segmentation strategies, feature selection weights are assigned to each feature, thereby selecting more key features during feature selection and allowing segmentation points to be closer to areas with dense outlier data, thus improving the speed and accuracy of outlier removal. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the wind direction sensor.

[0030] Figure 2 This is a schematic diagram of the Gray code disk;

[0031] Figure 3 This is a schematic diagram illustrating the working principle of the Gray code disk;

[0032] Figure 4 This is a flowchart illustrating the steps of an embodiment of the wind direction sensor abnormal data detection method of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0034] refer to Figure 1 The wind direction sensor consists of a wind vane component 1, a housing 2, and a socket 3, wherein the rotating shaft of the wind vane component 1 is connected to a... Figure 2The Gray code disk shown in this embodiment uses a 7-bit Gray code disk. The disk consists of 7 equally divided concentric circles, which are divided into 2, 21, 22, 23, 24, 25, and 26 equal parts from the inside out. Each part is sequentially etched with intervals for light transmission. Figure 3 As shown, seven sets of optocouplers, namely infrared LEDs 5 and photosensitive elements 6, are respectively set on both sides of the code track of the Gray code disk 4. When the wind vane rotates, it drives the coaxial Gray code disk 4 to rotate. The photosensitive elements 6 at different positions of the code track can sense the signal of the infrared LEDs 5 to form a 7-bit Gray code data, which is transmitted to the data acquisition unit through the signal cable. After conversion, the corresponding wind direction angle can be obtained. If the wind direction sensor, signal cable, or data acquisition channel of the data acquisition unit fails, for example, if one or more sets of optocouplers fail, one or more signal cables are broken, or one or more data acquisition channels of the data acquisition unit fail, the signal received by the data acquisition unit will not change when the angle of the wind vane changes. Therefore, the value of the Gray code data will not change for a period of time, resulting in a large number of abnormal data in the measured data. These abnormal data need to be removed in time, and the faults of the optocouplers, signal cables, or data acquisition units need to be eliminated in time.

[0035] refer to Figure 4 The wind direction sensor abnormal data detection method of the embodiment of the present invention includes:

[0036] Step S10000: Obtain Gray code data from the wind direction sensor;

[0037] Step S20000: Select one bit of Gray code data and divide it into several data units according to the time sequence, so that the data in each data unit is the same. Then count the number of data in each data unit and use it as a data point. Store the data points in the dataset to be detected.

[0038] Step S30000: Repeatedly sample the first number of data points without replacement from the dataset to be tested to generate subsamples and construct several isolation trees until the number of remaining data points in the dataset to be tested is insufficient to complete two rounds of sampling without replacement. Use the remaining data points in the dataset to be tested as subsamples to construct isolation trees and calculate the skewness of each subsample.

[0039] Specifically, the formula for calculating the skewness of a subsample is:

[0040] (1)

[0041] in, This represents the skewness of the subsample. For subsample size, For the first Data points, This is the mean of the subsample.

[0042] Step S40000: Generate split points based on skewness, recursively split the data of the subsample into two subsets, and continue to split the subsets until the stopping condition is met;

[0043] The rules for generating split points are as follows:

[0044] (2)

[0045] in, A random number between 0 and 1. The maximum value in the current subset of the isolated tree. It is the minimum value in the current subset of the isolated tree.

[0046] The stopping conditions include: the subset has only one data point left, or all data points have the same value, or the height of the isolation tree exceeds the preset value;

[0047] The default height of the isolation tree is:

[0048] (3)

[0049] in, This is the preset height for the isolation tree. The size of the subsample generated by the first sampling without replacement is the first number of data points.

[0050] Step S50000: Calculate the average path length of each data point in its isolation tree, and determine whether the data point is an abnormal data point based on the average path length;

[0051] Step S50000 also includes:

[0052] Step S50100: Calculate the anomaly score for each data point based on the average path length, and determine whether a data point is an anomaly based on the anomaly score.

[0053] Specifically, the formula for calculating the abnormal score is:

[0054] (4)

[0055] in, These are abnormal scores. This represents the average path length. For data points The path length in the current isolation tree. For the first A harmonic number, , It is a constant, namely the Euler-Macheroni constant, usually .

[0056] Step S60000: Select another value of the Gray code data, and repeat steps S30000 to S50000 until every value of the Gray code data has been selected.

[0057] Step S70000: Mark the Gray code data corresponding to the abnormal data points as abnormal data.

[0058] The principle of the wind direction sensor abnormal data detection method in the illustrated embodiment is explained below. If the wind direction sensor malfunctions, for example, if the optocoupler cannot receive light signals normally, then the corresponding Gray code data will not change for a long time, and this abnormal data needs to be identified.

[0059] The raw data from the wind direction sensor was acquired and converted into Gray code data, as shown in Table 1:

[0060] 1st position 2nd position 3rd position 4th position 5th position 6th position 7th position

[0061]

[0062] Selecting the first digit of the Gray code data yields: 0, 0, 1, 0, 1, 0, 1, 1, 1...1. Then, the data is divided into units: (0, 0), (1), (0), (1), (0), (1, 1, 1)...(1). The number of data points in each unit is calculated, resulting in the dataset to be tested: 2, 1, 1, 1, 3..., 1. Next, subsamples are generated from the dataset to be tested without replacement to construct an isolation tree. According to the default settings of scikit-learn (sklearn, an open-source Python machine learning library), the common subsample size is 256. Normal data points are usually smaller, while abnormal data points are larger. It is evident that abnormal data points are not evenly distributed but exhibit a clear right-skewed distribution. If the standard isolated forest algorithm is used to process abnormal data points, when selecting split points, the split points are randomly selected between the minimum and maximum values, i.e. Because the number of outliers is large (usually above 400, or even thousands), too many split points will fall in low-value areas, increasing path length and delaying the isolation of outlier data points in high-value areas. Therefore, this embodiment uses an Isolation Forest algorithm based on skewness adjustment to dynamically adjust the splitting strategy and generate split points according to the skewed distribution, making it more likely that split points will fall in areas with dense outlier data. This setting allows the Isolation Forest algorithm to be more focused.

[0063] The performance of the standard isolated forest algorithm and the isolated forest algorithm based on skewness adjustment (i.e., the algorithm used in this invention) is shown in Table 1.

[0064] Table 1

[0065]

[0066] As can be seen, the embodiments shown in this invention outperform the standard isolated forest algorithm in terms of accuracy, precision, and the harmonic mean of precision and recall, indicating that it performs better in reducing false alarms and improving overall classification performance. It can effectively identify all abnormal data, thereby determining the faults of sensors, signal cables, or data acquisition devices.

[0067] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for detecting abnormal data from a wind direction sensor, characterized in that, include: Step S10000: Obtain Gray code data from the wind direction sensor; Step S20000: Select one bit of the Gray code data and divide it into several data units in chronological order, so that the data in each data unit is the same. Then count the number of data in each data unit and use it as a data point. Store the data point in the dataset to be detected. Step S30000: Repeatedly sample a first number of data points without replacement from the dataset to be detected to generate subsamples and construct several isolation trees until the number of remaining data points in the dataset to be detected is insufficient to complete two samplings without replacement. Use the remaining data points in the dataset to be detected as subsamples to construct the isolation trees and calculate the skewness of each subsample. Step S40000: Generate split points based on the skewness, recursively split the data of the subsample into two subsets, and continue to split the subsets until the stopping condition is met; Step S50000: Calculate the average path length of each data point in the isolation tree, and determine whether the data point is an abnormal data point based on the average path length; Step S60000: Select another value of the Gray code data, and repeat steps S20000 to S50000 until every value of the Gray code data has been selected; Step S70000: Mark the Gray code data corresponding to the abnormal data points as abnormal data.

2. The method for detecting abnormal data from a wind direction sensor according to claim 1, characterized in that: The formula for calculating the skewness of the subsample in step S30000 is as follows: （1） in, The skewness of the subsample The size of the subsample, For the first The data points mentioned above, The mean of the subsample is given.

3. The method for detecting abnormal data from a wind direction sensor according to claim 2, characterized in that: The rule for generating the segmentation points in step S40000 is as follows: （2） in, A random number between 0 and 1. The maximum value in the subset of the current isolation tree. The minimum value in the subset of the current isolation tree. The skewness of the subsample.

4. The method for detecting abnormal data from a wind direction sensor according to claim 3, characterized in that: The stopping conditions in step S40000 include: the subset has only one data point left, or all the data points have the same value, or the height of the isolation tree exceeds a preset value; The preset value for the height of the isolation tree is: （3） in, The preset value is the height of the isolation tree. The size of the subsample generated by the first sampling without replacement.

5. The method for detecting abnormal data from a wind direction sensor according to claim 1, characterized in that: Step S50000 further includes: Step S50100: Calculate the anomaly score for each data point based on the average path length, and determine whether the data point is an anomaly based on the anomaly score.

Images