A method for online data quality control of a multi-path visibility sensor

Abstract

The application discloses a kind of multi-path visibility sensor online data quality control methods, comprising the following steps: (1) respectively reading the data of three visibility sensors, using Grubbs criterion to eliminate the abnormal value in each group of visibility sensor measurement data;(2) for each group of visibility sensor measurement data after eliminating abnormal value, fitting regression equation and carrying out significance test, correct each group of visibility sensor measurement data;(3) the three groups of visibility sensor measurement data after correction are formed into a group two by two, obtain three groups of measurement data sets;(4) based on multiple evaluation rules, each group of measurement data set is evaluated and the comprehensive evaluation score is calculated;(5) according to the comprehensive evaluation score, the measurement data set with the lowest comprehensive evaluation score is selected, and the median value of the two groups of visibility sensor measurement data in the measurement data set is averaged to obtain the final measurement result.The application can obtain more reliable and accurate visibility sensor data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of meteorological sensor data processing, in particular to a multi-channel visibility sensor online data quality control method. BACKGROUND

[0002] The visibility observation instrument converts optical information into an electrical signal through optical sensing, and then processes the obtained electrical signal to convert it into meteorological optical visibility data through a specific algorithm. Visibility, as an important meteorological observation element, has a wide impact on social production activities and people's lives. In some areas that are greatly affected by visibility, such as wharfs, airports, and highways, higher quality visibility sensor observation data is needed. Traditional sensor data processing methods rarely consider data quality issues before processing data. Even if they do, they only perform simple data cleaning to remove obviously deviating observation data, and there are still a large number of false measurements in the remaining data.

[0003] Patent application No. CN110134676B discloses a sensor data quality monitoring method, which divides abnormal data into abnormal format, abnormal timing, and abnormal value, converts the complex sensor abnormal data quality monitoring problem into abnormal format, abnormal timing, and abnormal value monitoring problem, and integrates detection, isolation, and recovery of abnormal data to realize online and reliable monitoring of sensor data quality. However, for visibility observation data quality control, this method is relatively complex, and processing single-layer data cannot solve the problem of low-quality data itself. SUMMARY

[0004] The present application provides a multi-channel visibility sensor online data quality control method to solve the above technical problems. The infrared scattering data output by the visibility sensor after online data quality control is obtained through a related algorithm to achieve the purpose of outputting one most actual visibility information from the multi-channel visibility sensor data.

[0005] Technical solution: To solve the above problems, the present application discloses a multi-channel visibility sensor online data quality control method, which specifically includes the following steps:

[0006] (1) Set the sampling frequency, read the data of three visibility sensors at each sampling time, and obtain three groups of visibility sensor measurement data; use the Grubbs criterion to remove abnormal values in each group of visibility sensor measurement data;

[0007] (2) Fit a regression equation for each group of visibility sensor measurement data after removing abnormal values and perform significance test to correct each group of visibility sensor measurement data;

[0008] (3) For the three sets of corrected visibility sensor measurement data, form a set of two sets of visibility sensor measurement data, and obtain three sets of measurement data sets;

[0009] (4) Based on a plurality of evaluation rules, each set of measurement data set is comprehensively evaluated and the comprehensive evaluation score is calculated; the plurality of evaluation rules include validity rule, correlation coefficient rule, Euclidean distance rule, and Jacard distance rule;

[0010] (5) According to the comprehensive evaluation score, the measurement data set with the lowest comprehensive evaluation score is selected, and the median value of the two sets of visibility sensor measurement data in the measurement data set is calculated to obtain the final measurement result.

[0011] Further, step (4) specifically includes:

[0012] (4.1) The three sets of measurement data sets are denoted as Data1(Y1,Y2), Data2(Y1,Y3), and Data3(Y2,Y3), wherein Y1, Y2, and Y3 represent a set of visibility sensor measurement data;

[0013] (4.2) For each set of measurement data set, the comprehensive evaluation score is calculated, and the comprehensive evaluation score is specifically calculated as follows:

[0014]

[0015]

[0016] R2=1-r t

[0017] R3=D Et

[0018] R4=D Jt

[0019] In the formula, G t represents the comprehensive evaluation score of the tth set of measurement data set; n represents the number of rules; R i represents the score of the ith rule; R1 represents the score of the validity rule; Data tj represents the jth data in the tth data set, and N represents the total number of data in the tth data set; F(Data tj ) is an effective function, and if Data tj is within the value range defined by the user, the value of F is 1, otherwise the value is 0; R2 represents the correlation coefficient rule score; R3 represents the Euclidean distance rule score; R4 represents the Jacard distance rule; wherein:

[0020]

[0021] wherein r t represents the similarity between two groups of measurement data in the tth data set; Y ik represents the measurement data at the kth sampling time in a group of visibility sensor measurement data in the tth data set, represents the average of a group of visibility sensor measurement data in the tth data set; Y jk represents the measurement data at the kth sampling time in another group of visibility sensor measurement data in the tth data set, represents the average of another group of visibility sensor measurement data in the tth data set; k represents the sampling time, and p represents the number of sampling times;

[0022]

[0023] wherein D Et represents the Euclidean distance between two groups of measurement data in the tth data set;

[0024]

[0025] wherein D Jt represents the Gower distance between two groups of measurement data in the tth data set; Y i represents a group of visibility sensor measurement data in the tth data set, Y j represents another group of visibility sensor measurement data in the tth data set.

[0026] Further, step (2) specifically comprises:

[0027] (2.1) fitting each visibility sensor measurement data after removing outliers to obtain a corresponding fitting regression equation formula:

[0028] Y = a + bX

[0029] wherein Y represents a group of visibility sensor measurement data obtained by a visibility sensor; X represents a sampling time sequence corresponding to the group of visibility sensor measurement data; a and b are both fitting coefficients;

[0030] (2.2) performing significance test on the fitting regression equation corresponding to each group of visibility sensor measurement data after removing outliers to obtain corrected measurement data of each group of visibility sensor measurement data; specifically:

[0031] According to the abnormal value removed from the corresponding visibility sensor measurement data, the abnormal value is corrected and supplemented to the corresponding visibility sensor measurement data; if the fitting regression equation is not significant, the corrected abnormal value is recorded as a missing value; and finally, the corrected measurement data of each group of visibility sensor measurement data is obtained.

[0032] Further, step (5) specifically comprises:

[0033] According to the comprehensive evaluation score, the measurement data set with the highest comprehensive evaluation score is selected, it is judged whether there is a missing value in the two groups of visibility sensor measurement data in the measurement data set, if there is a missing value record, the corrected abnormal value missing value is deleted, and the median values of the two groups of visibility sensor measurement data in the data set are extracted to calculate the average value to obtain the final measurement result; if there is no missing value record, the median values of the two groups of visibility sensor measurement data in the data set are directly extracted to calculate the average value to obtain the final measurement result.

[0034] In addition, the application also provides an embedded system device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned methods when executing the computer program. An embedded system readable storage medium has a computer program stored thereon, wherein the computer program is executed by the processor to implement the steps of any of the above-mentioned methods.

[0035] Beneficial effects: the application provides a multi-channel visibility sensor online data quality control method, which has the following advantages compared with the prior art: 1. By designing three visibility sensors, the data quality control algorithm is used to remove the measurement data with poor quality and large error, and then the evaluation rule is determined to evaluate the data set formed by the measurement data of the three visibility sensors to select the optimal visibility sensor data set, and the average value of the median values of the two groups of data in the optimal data set is taken as the output result, which is more accurate and reliable; 2. The multi-channel data abnormality identification technology is used to timely find the sensor abnormal event, and provide data support for daily maintenance and decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 Fig. 1 shows a working principle diagram of a multi-channel sensor system;

[0037] Figure 2 Fig. 2 shows the online data quality control steps of the application;

[0038] Figure 3 Fig. 3 shows a flowchart of the online data quality control method of the application. DETAILED DESCRIPTION

[0039] The technical solutions of the present application are further described below with reference to the drawings.

[0040] As Figures 2 to 3 shown, the present application provides a multi-channel visibility sensor online data quality control method, which specifically includes the following steps:

[0041] Step one, obtain the data of the multi-channel visibility sensor, and eliminate the outliers in the data.

[0042] As Figure 1 shown, the general multi-channel visibility sensor observation system mainly includes front-end sensors, collectors, data quality control units, power supply systems, etc., and the power supply system provides power supply for other systems. The present application configures three channels of visibility sensors, and performs visibility observation through the three channels of sensors, and the specific steps include:

[0043] (1) Read the real-time measurement data of the three channels of sensors;

[0044] Through sensor timing control, assuming that the sampling frequency is set to f. Specifically, taking the actual measurement data of three groups of infrared scattering sensors on May 3, 2021 as an example, starting from 20 o'clock, collecting once every ten seconds, a total of 13 times, obtaining three groups of 39 data of the first visibility sensor, the second visibility sensor and the third visibility sensor, respectively denoted as:

[0045] Y1[Y1(1),Y1(2),...,Y1(10)]

[0046] Y2[Y2(1),Y2(2),...,Y2(10)]

[0047] Y3[Y3(1),Y3(2),...,Y3(10)]

[0048] Among them, Y1, Y2, Y3 three groups of data are given time sequence X, each sampling time corresponds to a value, which is shown in the following table 1:

[0049] Table 1

[0050] [Y1] [Y1(1)] [Y1(2)] [Y1(3)] [Y1(4)] [Y1(5)] [Y1(6)] [Y1(7)] …… [Y1(12)] [Y1(13)] [Y2] [Y2(1)] [Y2(2)] [Y2(3)] [Y2(4)] [Y2(5)] [Y2(6)] [Y2(7)] …… [Y2(12)] [Y2(13)] [Y3] [Y3(1)] [Y3(2)] [Y3(3)] [Y3(4)] [Y3(5)] [Y3(6)] [Y3(7)] …… [Y3(12)] [Y3(13)] X 1 2 3 4 5 6 7 …… 12 13

[0051] Specifically, the actual three groups of measurement parameter data collected in 1s are shown in table 2:

[0052] Table 2

[0053] [Y1] 15 15 18 16 17 16 14 15 15 15 15 15 16 [Y2] 20 17 17 16 16 20 20 17 20 17 20 23 17 [Y3] 29 29 26 29 34 29 33 29 29 27 26 23 25 X 1 2 3 4 5 6 7 8 9 10 11 12 13

[0054] (2) Adopt Grubbs criterion to eliminate outliers in Y1, Y2 and Y3. Taking the elimination of outliers in Y1 group data by Grubbs criterion as an example, the specific steps include:

[0055] (2.1) Arrange the data; arrange the Y1 group data in Table 1 in ascending order to obtain Y 11 [ Y 11 (1), Y 11 (2),..., Y 11 (13)]. If an individual data deviates far from the average value in a group of measurement data, the data is called a "suspicious value", and it can be determined that the minimum value Y 11 (1) or the maximum value Y 11 (13) is a suspicious value;

[0056] (2.2) Calculate the average value and the standard deviation S of the group of measurement data;

[0057] (2.3) Calculate the deviation value: the difference between the average value and the minimum value Y 11 (1), and the difference between the maximum value Y 11 (13) and the average value ;

[0058] (2.4) Determine the suspicious value: compare the deviation values, if the difference between the maximum value and the average value is greater than the difference between the average value and the minimum value, then the maximum value Y 11 (13) is considered to be a suspicious value; if the difference between the maximum value and the average value is less than the difference between the average value and the minimum value, then the minimum value Y 11 (1) is considered to be a suspicious value; after calculation, the maximum value Y 11 (13) is considered to be a suspicious value here;

[0059] (2.5) Calculate the Gr i value: where i is the arrangement number of the suspicious value; Y 11 (13) is a suspicious value, then calculate

[0060] (2.6) Set the detection level a; if strictness is required, the detection level a can be set smaller; for example, a = 0.01, then the confidence probability P = 1 - a = 0.99; if leniency is required, a can be set larger; for example, a = 0.1, then the confidence probability P = 1 - a = 0.9; the present application adopts a = 0.05, P = 0.95.

[0061] (2.7) Look up the Grubbs table to obtain the critical value; according to the selected P value and the number of measurements n, look up the Grubbs table, and obtain the critical value Gr P (n) by horizontal and vertical intersection; that is, Gr 95 (13) = 2.331 in the present application;

[0062] (2.8) Compare the calculated value Gr iand the critical value Gr P (n), if the calculated value Gr i and greater than the critical value Gr P (n) in the table, it can be judged that the measured data is an outlier, which is rejected;

[0063] (2.9) For the remaining measured data, repeat the above steps to remove all outliers in the group of measured data.

[0064] In specific embodiments, outliers in Y1, Y2, and Y3, which are all sequences lacking some values, are removed according to the above Grubbs criterion. For example, Y1 after removing outliers is:

[0065] Y1[15,15,16,17,16,14,15,15,15,15,15,16]

[0066] X1[1,2,4,5,6,7,8,10,11,12,13]

[0067] Step two, for each group of visibility sensor measurement data after removing outliers, a regression equation is fitted and a significance test is performed to correct each group of visibility sensor measurement data;

[0068] (1) For each group of visibility sensor measurement data after removing outliers, a regression equation is fitted respectively, and the corresponding fitting regression equation formula is:

[0069] Y = a + bX

[0070] where Y represents a group of visibility sensor measurement data obtained by a visibility sensor; X represents a sampling time sequence corresponding to a group of visibility sensor measurement data; a and b are both fitting coefficients;

[0071] (2.2) For each group of visibility sensor measurement data after removing outliers, a significance test is performed on the corresponding fitting regression equation to supplement outliers in each group of visibility sensor measurement data; specifically including:

[0072] According to the obtained fitting regression equation, the removed outliers in the corresponding visibility sensor measurement data are corrected, and the corrected outliers are supplemented to the corresponding visibility sensor measurement data; if the fitting regression equation is not significant, the corrected outliers also need to be recorded as missing values.

[0073] where, for the significance test of the fitting regression equation, the statistical quantity F is calculated According to the F distribution with 1 degree of freedom in the numerator and n-2 degrees of freedom in the denominator, if F>F α , the regression equation is considered significant, otherwise it is not significant.

[0074] In the embodiment, the measured data of the abnormal values in Y1, Y2 and Y3 is corrected, wherein the obtained sequence Y1, X1 of the missing part values is fitted to obtain an equation Y1=15.48-0.02X1, and the equation is not significant through inspection.

[0075] The simulated value 15.42 corresponding to the eliminated time X(3) is calculated by the equation Y1=15.48-0.02X1, and the value is replaced by the original value to obtain a complete sequence:

[0076] Y1[15, 15, 15.42, 16, 17, 16, 14, 15, 15, 15, 15, 15, 16]

[0077] Step three, for the three groups of corrected visibility sensor measured data, the three groups of visibility sensor measured data are formed into a group two by two to obtain three groups of measured data sets; that is, the three groups of measured data sets are respectively Data1(Y1, Y2), Data2(Y1, Y3) and Data3(Y2, Y3).

[0078] Step four, according to the actual situation, the constraint rules are determined, and the measured data sets are comprehensively evaluated; the present application sets multiple evaluation rules including validity rules, correlation coefficient rules, Euclidean distance rules and Jacard distance rules;

[0079] The comprehensive evaluation score is calculated for each group of measured data sets, and the specific calculation formula of the comprehensive evaluation score is:

[0080]

[0081]

[0082] R2=1-r t

[0083] R3=D Et

[0084] R4=D Jt

[0085] In the formula, G t represents the comprehensive evaluation score of the tth group of measured data sets; n represents the number of rules, which is 4; R i represents the score of the ith rule; R1 represents the score of the validity rule; Data tj represents the jth data in the tth data set, and N represents the total number of data in the tth data set; F(Data tj ) is an effective function, and if Data tjF is 1 if the value is within the user-defined range, otherwise 0; R2 represents the correlation coefficient rule score; R3 represents the Euclidean distance rule score; R4 represents the Jaccard distance rule; wherein:

[0086]

[0087] wherein, r t represents the similarity between two groups of measurement data in the tth data set; Y ik represents the measurement data at the kth sampling time in a group of visibility sensor measurement data in the tth data set, represents the average value of a group of visibility sensor measurement data in the tth data set; Y jk represents the measurement data at the kth sampling time in another group of visibility sensor measurement data in the tth data set, represents the average value of another group of visibility sensor measurement data in the tth data set; k represents the sampling time, and p represents the number of sampling times;

[0088]

[0089] wherein, D Et represents the Euclidean distance between two groups of measurement data in the tth data set; represents the distance between two groups of data in the data set, and the greater the distance, the smaller the similarity; the smaller the distance, the greater the similarity;

[0090]

[0091] wherein, D Jt represents the Jaccard distance between two groups of measurement data in the tth data set, represents the similarity and diversity measure of the data set, and is the ratio of the intersection and union of the two groups of data minus 1. The greater the distance, the smaller the similarity; the smaller the distance, the greater the similarity. Y i represents a group of visibility sensor measurement data in the tth data set, Y j represents another group of visibility sensor measurement data in the tth data set.

[0092] In the specific embodiment, for the Data2(Y1, Y3), Data3(Y2, Y3) data sets, the final G1, G2, and G3 scores are: G1 = 8.73, G2 = 14.84, and G3 = 13.53.

[0093] Step five, according to the comprehensive evaluation score, select the measurement data set with the lowest comprehensive evaluation score, and calculate the final measurement result by taking the median value of the two groups of visibility sensor measurement data in the measurement data set as the average value.

[0094] Among them, according to the comprehensive evaluation score, the measurement data set with the lowest comprehensive evaluation score is selected. It is judged whether there is a missing value record in the two groups of visibility sensor measurement data in the measurement data set. If there is a missing value record, the final measurement result is calculated by extracting the median value of the two groups of visibility sensor measurement data in the data set after deleting the corrected abnormal value as the average value. If there is no missing value record, the final measurement result is calculated by directly extracting the median value of the two groups of visibility sensor measurement data in the data set as the average value.

[0095] In a specific embodiment, according to the comprehensive evaluation score, the measurement data set with the lowest comprehensive evaluation score is G1, that is, Data1(Y1, Y2) is the optimal data set. Because the equation is not significant, Y1(3) is removed, and the final output result 16 is obtained.

[0096] Specifically, the data obtained by the other group of sensors in the laboratory is taken as the standard value, the average value of the measurement data of the three groups of sensors is taken as the result obtained by the sensor, and the result obtained after the data quality control (the application) is compared as shown in Table 3.

[0097] Table 3

[0098]

[0099] From the table, it can be seen that compared with the measurement result using the data quality control (the application), the relative error of the average value of the measurement data of the three groups of sensors is 36.36% and 5.05%, respectively. It can be seen that after the data quality control, the data accuracy is obviously improved and is closer to the standard value. From the overall 5-day measurement, before the data quality control, the average relative error of the measurement is 36.86%, and after the data quality control, the average relative error is 4.28%, which decreases by 32.58%. Similarly, it is proved that after the online data quality control algorithm, the data accuracy of the sensor is obviously improved.

Claims

1. A method for online data quality control of multi-channel visibility sensors, characterized in that, Includes the following steps: (1) Set the sampling frequency and read the data of the three visibility sensors at each sampling time to obtain three sets of visibility sensor measurement data; use the Grubbs criterion to remove outliers in each set of visibility sensor measurement data; (2) Fit a regression equation for each group of visibility sensor measurement data after removing outliers and perform a significance test to correct each group of visibility sensor measurement data; including: (2.1) For each visibility sensor measurement data after removing outliers, a fitting is performed to obtain the corresponding fitting regression equation formula: In the formula, This represents a set of visibility sensor measurement data obtained from a single visibility sensor. This represents a sampling time series corresponding to a set of visibility sensor measurement data; , All are fitting coefficients; (2.2) Fitting regression equations for each group of visibility sensor measurement data after removing outliers. Significance tests were performed separately to obtain corrected visibility sensor measurement data for each group; specifically: The outliers removed from the corresponding visibility sensor measurement data are corrected based on the obtained fitted regression equation, and the corrected outliers are added back to the corresponding visibility sensor measurement data. If the fitted regression equation is not significant, the corrected outliers are also recorded as missing values. Finally, the corrected measurement data for each group of visibility sensor measurement data is obtained. (3) For the three sets of visibility sensor measurement data after correction, the three sets of visibility sensor measurement data are combined into a group in pairs to obtain three sets of measurement datasets; (4) Based on multiple evaluation rules, each measurement dataset is comprehensively evaluated and a comprehensive evaluation score is calculated; the multiple evaluation rules include validity rules, correlation coefficient rules, Euclidean distance rules, and Jakehar distance rules; specifically including: (4.1) The three sets of measurement datasets are denoted as follows: ,in , , Each represents a set of visibility sensor measurement data; (4.2) Perform a comprehensive evaluation for each set of measurement datasets and calculate the comprehensive evaluation score. The specific formula for calculating the comprehensive evaluation score is as follows: In the formula, This represents the overall evaluation score of the t-th measurement dataset; Indicates the number of rules; Indicates the first The score for each rule; Indicates the score of the validity rule; Let N represent the j-th data point in the t-th dataset, and let N represent the total number of data points in the t-th dataset. For a function to be valid, if If the value is within the user-defined range, then the value of F is 1; otherwise, the value is 0. This represents the score of the correlation coefficient rule; Indicates the score according to the Euclidean distance rule; Represents the Jäger distance rule; where: In the formula, This represents the degree of similarity between two sets of measurement data in the t-th dataset; This represents the measurement data at the k-th sampling time in a set of visibility sensor measurements within the t-th dataset. This represents the average value of a set of visibility sensor measurements in the t-th dataset; This represents the measurement data at the k-th sampling time in another set of visibility sensor measurements within the t-th dataset. The average value of another set of visibility sensor measurements in the t-th dataset; Indicates the sampling time. Indicates the number of sampling times; In the formula, This represents the Euclidean distance between two sets of measurement data in the t-th dataset; In the formula, This represents the Gerd distance between two sets of measurement data in the t-th dataset; This represents a set of visibility sensor measurements in the t-th dataset. This represents another set of visibility sensor measurement data in the t-th dataset; (5) Select the measurement dataset with the lowest comprehensive evaluation score based on the comprehensive evaluation score, and calculate the final measurement result by averaging the median values ​​of the two sets of visibility sensor measurement data in the measurement dataset.

2. The online data quality control method for multi-channel visibility sensors according to claim 1, characterized in that, Step (5) specifically includes: Based on the comprehensive evaluation score, select the measurement dataset with the lowest comprehensive evaluation score. Determine whether there are missing values ​​in the two sets of visibility sensor measurement data in this dataset. If there are missing value records, delete the corrected outliers and extract the median value of the two sets of visibility sensor measurement data in this dataset to calculate the average value and obtain the final measurement result. If there are no missing value records, directly extract the median value of the two sets of visibility sensor measurement data in this dataset to calculate the average value and obtain the final measurement result.

3. An embedded system readable storage medium, characterized in that, The embedded system readable storage medium includes a stored computer program, wherein, when the embedded system program is executed, it controls the device where the embedded system readable storage medium is located to perform the steps of the method as described in any one of claims 1 to 2.

4. An embedded system device, characterized in that, A memory, a processor, and a program stored in and executable on the memory, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 2.

Images