A method for identifying outliers based on improved Grubbs test

Abstract

An outlier identification method based on an improved Grubbs test method is used for detecting and removing outliers in gas concentration compensation values. The method comprises the following steps: S1, establishing a data model; S2, replacing the mean value by standard model defects; S3, improving the algorithm model, replacing the mean value by introducing a reference value; S4, recalculating the statistical value H i , judge outliers. When outliers account for a large proportion, the present invention can effectively identify and remove outliers, retain the true value, and thus improve the accuracy of the data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an outlier identification method based on an improved Grubbs test method. Background Art

[0002] Gas detection is affected by a variety of environmental factors, and these factors interact with each other, all of which can affect the test results. After designing a corresponding compensation algorithm to offset the impact of external factors on the test results, it is necessary to adopt more efficient data processing methods to resolve abnormal values ​​generated when calculating environmental parameter compensation values.

[0003] The presence of outliers in the compensation values ​​will affect the accuracy and reliability of gas concentrations. Current methods for processing outlier data have significant limitations, such as low accuracy and a limited proportion of outliers. To improve outlier detection accuracy when the proportion of outliers is high, directly applying the Grubbs criterion can effectively handle data with an error rate of less than 10%.

[0004] Current methods for identifying outliers include the Rhineda criterion, the Grubbs criterion, and the Romanovsky criterion. The Rhineda criterion is convenient for identifying outliers by simply looking up a table, but it is extremely inaccurate when the data set is 10 or fewer. The Romanovsky criterion is computationally complex and prone to data redundancy. The Grubbs criterion is suitable for use with smaller data sets. Furthermore, its determination of outliers is independent of the mean and variance of the array, making it easy to control and adjust. However, mathematically, the Grubbs criterion is the best solution when only a small number of outliers are present in a set of measurements. When a large number of outliers are present, it is necessary to first remove the outliers that are considered outliers before applying the Grubbs criterion to identify the remaining smaller outliers.

[0005] When the number of outliers in the data sample is less than 10%, recognition is very stable. However, when the proportion of outliers exceeds 15%, the Grubbs method will directly fail. When there are too many outliers, the standard deviation will increase when outliers are present, resulting in the possibility of recognition failure or omission, reducing the recognition rate. The presence of too many outliers may cause a large deviation in the mean estimate, causing significant changes in the statistical value, resulting in recognition failure or omission of outliers. Summary of the Invention

[0006] In response to the technical problem that traditional outlier identification methods may lead to inaccurate detection results when gas concentration compensation values ​​face a large number of outliers, this technical solution provides an outlier identification method based on the improved Grubbs test method. The Grubbs test method is improved and the improved Grubbs criterion is used to identify and process outliers in the compensation values, thereby improving detection accuracy while solving the problem of being unable to operate due to a large proportion of outliers. It can effectively eliminate the impact of inaccurate results caused by a large proportion of outliers and detect outliers in gas concentration compensation values ​​with higher accuracy, effectively solving the above-mentioned problems.

[0007] The present invention is achieved through the following technical solutions:

[0008] An outlier identification method based on an improved Grubbs test method is used to detect and remove outliers in gas concentration compensation values, the method comprising the steps of:

[0009] S1. Establish data model;

[0010] S2. Establish the Grubbs test standard model;

[0011] S3. Improve the algorithm model and replace the mean with the reference value;

[0012] The mean is replaced by other reference values ​​to realize outlier discrimination. Other reference values ​​are the median, mode, maximum value and minimum value composed of reference values ​​according to different weights, which are used to replace the mean to realize outlier test. The calculation formula (3-1) is

[0013]

[0014] Where P1 and P2 are reference values, x max Represents the maximum value in the selected data sample; x min It represents the minimum value in the selected data sample; N represents the median of the selected data sample; M represents the mode of the selected data sample; a bc d represents the weight of each parameter;

[0015] The reference values ​​P1 and P2 are used instead of the mean to calculate the absolute value of the residual error. The calculation formula is (3-2):

[0016]

[0017] In the above formula, |V i | is expressed as the absolute value of the residual error of the i-th data in the data sample, x i It is represented as the i-th data in the data sample, and P1 and P2 are represented as reference values.

[0018] Replace the mean μ with the reference value to complete the calculation of the standard deviation. The calculation formula is (3-3):

[0019]

[0020] In the above formula, σ' represents the standard deviation calculated after the mean is replaced by the reference value, n represents the total number of data samples, and x k Represented as the kth data sample in the data sample.

[0021] S4. Recalculate the statistical value H i , determine the outliers.

[0022] Furthermore, the establishment of the data model described in step S1 is to detect the gas in the air, and the sensor detection module is used to collect the numerical values ​​of the gas concentration, temperature, humidity and wind speed in the air. Then, the processor calculates the actual concentration of the gas after compensation based on the compensation algorithm for the data sent by the sensor, and finally uses the compensation correction module to process the compensation value to identify and remove abnormal data.

[0023] Furthermore, the specific operation mode of the compensation correction module for performing data processing on the compensation value is as follows:

[0024] Assume that the gas compensation value X is the monitoring data sample set, i = 1, 2···, n, and establish the data model (1-1) with μ as the observation object:

[0025] X~n(μ,σ 2 ) (1-1);

[0026] In the formula, X represents the data set; μ represents the mean of the data set; σ 2 Expressed as the variance of the data set; the compensation value data sample of formula (1-1) conforms to the normal distribution, and the mean μ and variance σ of formula (1-1) are 2 As shown in formula (1-2):

[0027]

[0028] Where x i It represents the i-th data in the detection data sample; n represents the number of data in the sample.

[0029] Furthermore, the specific operation method of establishing the standard model in step S2 is:

[0030] According to formula (1-2), the standard Grubbs test method is established, and its calculation formula is as follows (2-1)

[0031]

[0032] Where H i Expressed as the statistical value of the i-th data in the data sample, |V i| is expressed as the absolute value of the residual error of the i-th data in the data sample.

[0033] Furthermore, the statistical value H is recalculated in step S4. i , to determine the outliers, the specific method is:

[0034] After improving the residual error and standard deviation of the compensation value, the statistical value H is re-calculated i , the calculation formula is (4-1):

[0035]

[0036] Redefine the statistic H' i , so that the interference caused by mean deviation is completely eliminated in the formula. The two reference values ​​are the maximum value, minimum value, median and mode in proportion. The median can effectively represent the middle data in a set of data samples. To a certain extent, it combines the advantages of the average and is representative. The mode focuses on recording the frequency of data occurrence and is related to some data in the data group. It is not affected by extreme values. The maximum and minimum values ​​are the two extreme values ​​in the data group. They are very likely to be outliers, but the combination of the two can eliminate their influence. Calculate the statistical value H' i After that, the recognition work is carried out, and the recognition calculation formula is (4-2):

[0037] H' i ≤G p (n) (4-2);

[0038] In the above formula, G p (n) is the critical value; critical value G p The size of (n) is related to the confidence probability p and the amount of data n. For some data sets, the accuracy needs to be extremely precise, so the confidence probability p must be defined higher. If the statistical value H' of a certain data in the data sample is i Greater than the critical value G p (n), then the data has a large error, delete it, and then use the statistical value H' of the next data i With the critical value G p (n) Compare and repeat the above process until there are no more abnormal values;

[0039] According to the above process, the data after deleting outliers from multiple sample data sets are taken as a group of data, and the mean of this group of data is recalculated to obtain a more accurate compensation value. The calculation formula for the more accurate compensation value is (4-3):

[0040]

[0041] Where m represents the number of compensation values ​​in the data set after deleting outliers, xj is the jth data in the data sample after deleting the outliers; after the improved Grubbs test, the accurate environmental parameter compensation value V for gas detection is obtained 补偿 .

[0042] Beneficial effects

[0043] The present invention proposes an outlier identification method based on an improved Grubbs test method, which has the following beneficial effects compared with the prior art:

[0044] (1) The present invention improves the Grubbs test method and converts the statistical value L i The mean in Replace it. Use the median, mode, maximum, and minimum values ​​according to different weights to form reference values ​​P1 and P2. The median can effectively represent the data in the middle of a set of data samples, and is not affected by the two extreme values ​​of the maximum and minimum values. To a certain extent, it combines the advantages of the average and is representative. The mode focuses on recording the frequency of data occurrence, is related to some data in the data set, and is not affected by extreme values. The maximum and minimum values ​​are the two extreme values ​​that appear in the data set and are very likely to be outliers. However, the combination of the two can eliminate their influence, making the reference value closer to the center of the data set and reducing the influence of extreme values.

[0045] (2) The present invention uses an improved Grubbs criterion to identify and process outliers in the compensation value, improving detection accuracy while resolving the problem of being unable to operate due to a large proportion of outliers. It can effectively eliminate the impact of a large proportion of outliers and detect outliers in the gas concentration compensation value with higher accuracy. When a large number of errors exist, it can reduce the impact on gas concentration detection. This is an innovative point for identifying outliers in the detection results of toxic gas concentration thresholds in chemical plants. It can greatly improve the reliability of gas detection devices, while improving the safety factor of chemical plants and reducing the occurrence of dangerous situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the operation process of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Under the premise of not departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in this field to the technical solutions of the present invention should fall within the scope of protection of the present invention.

[0048] Example 1:

[0049] like Figure 1As shown, an outlier identification method based on an improved Grubbs test method is used for detecting and removing outliers in gas concentration compensation values. The method comprises the steps of: S1, establishing a data model.

[0050] To detect gases in the air, a sensor detection module is used to collect the numerical values ​​of gas concentration, temperature, humidity and wind speed in the air. Then, the processor calculates the true concentration of the gas after compensation based on the compensation algorithm for the data sent by the sensor. Finally, the compensation correction module is used to process the compensation value and identify and remove abnormal data. This is an improvement to the abnormal value detection algorithm in the correction compensation module of the gas detection device.

[0051] The specific operation mode of the compensation correction module for data processing of the compensation value is as follows:

[0052] Assume that the gas compensation value X is the monitoring data sample set, i = 1, 2···, n, and establish the data model (1-1) with μ as the observation object:

[0053] X~n(μ,σ 2 ) (1-1);

[0054] In the formula, X represents the data set; μ represents the mean of the data set; σ 2 Expressed as the variance of the data set; the compensation value data sample of formula (1-1) conforms to the normal distribution, and the mean μ and variance σ of formula (1-1) are 2 As shown in formula (1-2):

[0055]

[0056] Where x i It represents the i-th data in the detection data sample; n represents the number of data in the sample.

[0057] S2. Establish the Grubbs test standard model.

[0058] According to formula (1-2), the standard Grubbs test method is established, and its calculation formula is shown in (2-1):

[0059]

[0060] Where H i Expressed as the statistical value of the i-th data in the data sample, |V i | is expressed as the absolute value of the residual error of the i-th data in the data sample.

[0061] If the standard Grubbs test is used to identify outliers, the compensation value x i The ratio of the absolute value of the difference from the mean μ to the standard deviation σ is defined as H i If the statistic Hi Greater than the critical value G p (n), it indicates that the compensation value is an outlier and can be deleted directly.

[0062] However, when the gas compensation values ​​contain a large number of outliers, the mean of the data sample will deviate significantly, and the standard deviation σ will also be affected by the change in the mean. Assume that the gas compensation value data sample contains a large number of outliers, and their values ​​are large. This directly leads to an increase in the mean and a decrease in the variance in the data model. To reduce the impact of the mean on the Grubbs criterion test for outliers, it is necessary to improve the Grubbs test.

[0063] S3. Improve the algorithm model and replace the mean with a reference value.

[0064] A large number of gas compensation values ​​cannot avoid the influence of outliers, so some gas compensation values ​​are randomly selected. In the standard Grubbs test, the mean is used to measure the center position of a data set. The mean of the data set is used to identify outliers and compare the outlier Z-score (i.e., the number of standard deviations from the mean). Excessive outliers cause the mean deviation to increase, resulting in significant changes in the statistical value. Therefore, the present invention replaces the mean with other reference values ​​to achieve outlier identification.

[0065] Compared with the mean, the median can better represent the center position of the data set and is not easily affected by extreme values ​​in the data. The mode can reflect the central trend of the data set, can represent the overall situation of the data, and is not easily affected by extreme values ​​in the data. The maximum and minimum values ​​are two extreme values ​​in the data set, which can represent the upper and lower limits of the data, but are not necessarily outliers. In this embodiment, the above four values ​​are combined into reference values ​​according to different weights to replace the mean to implement outlier detection. The calculation formula (3-1) is

[0066]

[0067] Where P1 and P2 are reference values, x max Represents the maximum value in the selected data sample; x min It represents the minimum value in the selected data sample; N represents the median of the selected data sample; M represents the mode of the selected data sample; a bc d represents the weight of each parameter.

[0068] The maximum and minimum values ​​constitute the reference value P1 according to different weights, which can effectively prevent the influence of extreme values ​​on both sides of the data set on the test outliers. Assuming that the maximum and minimum values ​​of the data set are both outliers, the reference value P1 obtained after formula (3-1) is close to the center of the data set, reducing the influence of extreme values. The reference value P2 composed of the median and mode can better replace the mean and the center of the data set, and the median and mode are not easily affected by outliers, which can better complete the test. The reference values ​​P1 and P2 replace the mean to complete the calculation of the absolute value of the residual error, which can make the statistical value H i Molecule|V i |The deviation from the mean will not cause the failure or omission of outlier identification. The calculation formula is (3-2):

[0069]

[0070] In the above formula, |V i | is expressed as the absolute value of the residual error of the i-th data in the data sample, x i It is represented as the i-th data in the data sample, and P1 and P2 are represented as reference values.

[0071] By replacing the mean, the interference of the results caused by the mean deviation can be effectively avoided. However, simply replacing the mean cannot completely solve the impact caused by a large number of outliers. It is also necessary to make corresponding adjustments from the standard deviation to fundamentally eliminate the influence of the mean on the statistical value H. i The mean μ is replaced by the reference value to complete the calculation of the standard deviation, and the calculation formula is (3-3):

[0072]

[0073] In the above formula, σ' represents the standard deviation calculated after the mean is replaced by the reference value, n represents the total number of data samples, and x k Represents the kth data in the data sample.

[0074] Strictly speaking, in mathematical terms, the above two processing methods will change the statistical value H i However, from the perspective of practical application, it is believed that the above processing changes only the statistical value H i The value of , has a negligible effect on the distribution.

[0075] S4. Recalculate the statistical value H i , judge the outliers; the specific method is:

[0076] After improving the residual error and standard deviation of the compensation value, the statistical value H is re-calculated i , the calculation formula is (4-1):

[0077]

[0078] Redefine the statistic H' i , so that the formula completely eliminates the interference that may be caused by mean deviation. The two reference values ​​are the maximum value, minimum value, median and mode in proportion. The median can effectively represent the middle data in a set of data samples. To a certain extent, it combines the advantages of the average and is representative. The mode focuses on recording the frequency of data occurrence and is related to some data in the data group. It is not affected by extreme values. The maximum and minimum values ​​are the two extreme values ​​that appear in the data group. They are very likely to be outliers, but the combination of the two can eliminate their influence. Calculate the statistical value H' i After that, the recognition work is carried out, and the recognition calculation formula is (4-2):

[0079] H' i ≤G p (n) (4-2);

[0080] In the above formula, G p (n) is the critical value; critical value G p The size of (n) is related to the confidence probability p and the amount of data n. For some data sets, the accuracy needs to be extremely precise, so the confidence probability p must be defined higher, here it is 0.99.

[0081] This time, 50 data in the data set are selected as data samples. According to the Grubbs criterion table, when n=50, p=0.99, G p (n) = 3.34. If the statistical value H' of a certain data in the data sample i Greater than the critical value G p (n), then the data has a large error, delete it, and then use the statistical value H' of the next data i With the critical value G p (n) and repeat the above process until there are no more abnormal values.

[0082] According to the above process, the data after deleting outliers from the 50 sample data sets are taken as a group of data, and the mean of this group of data is recalculated to obtain a more accurate compensation value. The calculation formula for the more accurate compensation value is (4-3):

[0083]

[0084] Where m represents the number of compensation values ​​in the data set after deleting outliers, x j is the jth data in the data sample after deleting the outliers; after the improved Grubbs test, the accurate environmental parameter compensation value V for gas detection is obtained 补偿 .

[0085] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes, replacements and improvements within the technical scope disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for identifying outliers based on an improved Grubbs test method for detecting and removing outliers in gas concentration compensation values; characterized by: The method comprises the steps of: S1. Establish data model; S2. Establish the Grubbs test standard model; S3. Improve the algorithm model and replace the mean with the reference value; The mean is replaced by other reference values ​​to realize outlier discrimination. Other reference values ​​are the median, mode, maximum value and minimum value composed of reference values ​​according to different weights, which are used to replace the mean to realize outlier test. The calculation formula (3-1) is Where P1 and P2 are reference values, x max Indicates the maximum value among the selected compensation value samples; x min It represents the minimum value of the selected compensation value sample; N represents the median of the selected compensation value sample; M represents the mode of the selected compensation value sample; abcd represents the weight of each parameter; The reference values ​​P1 and P2 are used instead of the mean to calculate the absolute value of the residual error. The calculation formula is (3-2): In the above formula, |V i | is expressed as the absolute value of the residual error of the i-th data in the data sample, x i It is represented as the i-th data in the data sample, and P1 and P2 are represented as reference values; Replace the mean μ with the reference value to complete the calculation of the standard deviation. The calculation formula is (3-3): In the above formula, σ' represents the standard deviation calculated after the mean is replaced by the reference value, n represents the total number of data samples, and x k Represented as the kth data sample in the data sample, S4. Recalculate the statistical value H i , determine the outliers.

2. The outlier identification method based on the improved Grubbs test according to claim 1, characterized in that: The establishment of the data model described in step S1 is to detect the gas in the air, and the sensor detection module is used to collect the numerical values ​​of the gas concentration, temperature, humidity and wind speed in the air. Then the processor calculates the actual concentration of the gas after compensation based on the compensation algorithm for the data sent by the sensor, and finally uses the compensation correction module to process the compensation value to identify and remove abnormal data.

3. The outlier identification method based on the improved Grubbs test according to claim 2, characterized in that: The specific operation mode of the compensation correction module for data processing of the compensation value is as follows: Assume that the gas compensation value X is the monitoring data sample set, i = 1, 2···, n, and establish the data model (1-1) with μ as the observation object: X~n(μ,σ 2 ) (1-1); In the formula, X represents the data set; μ represents the mean of the data set; σ 2 Expressed as the variance of the data set; the compensation value data sample of formula (1-1) conforms to the normal distribution, and the mean μ and variance σ of formula (1-1) are 2 As shown in formula (1-2): Where x i It represents the i-th data in the detection data sample; n represents the number of data in the sample.

4. The outlier identification method based on the improved Grubbs test according to claim 3, characterized in that: The specific operation method of establishing the standard model in step S2 is: According to formula (1-2), the standard Grubbs test method is established, and its calculation formula is as follows (2-1) Where H i Expressed as the statistical value of the i-th data in the data sample, |V i | represents the absolute value of the residual error of the i-th data in the data sample.

5. The outlier identification method based on the improved Grubbs test according to claim 1, characterized in that: The recalculated statistical value H in step S4 i , to determine the outliers, the specific method is: After improving the residual error and standard deviation of the compensation value, the statistical value H is re-calculated i , the calculation formula is (4-1): Redefine the statistic H' i , so that the interference caused by mean deviation is completely eliminated in the formula. The two reference values ​​are the maximum value, minimum value, median and mode in proportion. The median can effectively represent the middle data in a set of data samples. To a certain extent, it combines the advantages of the average and is representative. The mode focuses on recording the frequency of data occurrence and is related to some data in the data group. It is not affected by extreme values. The maximum and minimum values ​​are the two extreme values ​​in the data group. They are very likely to be outliers, but the combination of the two can eliminate their influence. Calculate the statistical value H' i After that, the recognition work is carried out, and the recognition calculation formula is (4-2): H' i ≤G p (n) (4-2); In the above formula, G p (n) is the critical value; critical value G p The size of (n) is related to the confidence probability p and the amount of data n. For some data sets, the accuracy needs to be extremely precise, so the confidence probability p must be defined higher. If the statistical value H' of a certain data in the data sample is i Greater than the critical value G p (n), then the data has a large error, delete it, and then use the statistical value H' of the next data i With the critical value G p (n) Compare and repeat the above process until there are no more abnormal values; According to the above process, the data after deleting outliers from multiple sample data sets are taken as a group of data, and the mean of this group of data is recalculated to obtain a more accurate compensation value. The calculation formula for the more accurate compensation value is (4-3): Where m represents the number of compensation values ​​in the data set after deleting outliers, x j is the jth data in the data sample after deleting the outliers; after the improved Grubbs test, the accurate environmental parameter compensation value V for gas detection is obtained 补偿 .

Images