Bidirectional correction method for abnormity of industrial time series data

Abstract

The invention relates to an industrial time series data abnormity bidirectional correction method, which fully considers data in time windows before and after an abnormal data point, and performs forward and backward data correction on the abnormal data according to a time sequence; performing weighted summation on results obtained by forward correction and backward correction on the abnormal data segments to obtain a final bidirectional correction value, and replacing the original abnormal data; according to the method provided by the invention, the corresponding weight is given according to the specific position of the abnormal segment where the abnormal data point is located for weighted correction, the characteristics of time sequence data are effectively utilized, and the time correlation of data before and after the abnormal segment is utilized, so that errors possibly introduced in the correction process are reduced; the problem of time sequence feature loss caused by adopting a mean value or mode to correct data is effectively avoided, and support is provided for improving data quality and establishing a more accurate prediction model.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for bidirectionally correcting abnormal industrial time series data. Background Art

[0002] In actual production life, due to human operation or natural reasons, errors may occur in the process of data collection, transmission and storage. The existence of these abnormal data will affect the normal operation of industrial production; therefore, correcting the abnormal data in industrial data is of great value to promoting industrial safety and stable production.

[0003] Traditional methods for correcting abnormal data usually include some basic statistical methods and heuristic rules, such as mean filling, median filling, and mode filling. These methods have some significant shortcomings when dealing with data quality issues: (1) Reliance on empirical rules: Many traditional methods rely on pre-set rules or thresholds. These rules may be based on past data or expert experience, but lack flexibility and cannot adapt to trends and changes. (2) Limited generalization ability: Traditional data correction methods are often designed for specific types of data anomalies, such as point anomalies, contextual anomalies, or group anomalies, and cannot identify and correct new types of anomalies or complex data relationships. (3) Lack of automation: Traditional anomaly detection and correction methods often require A large amount of manual intervention, such as manually setting thresholds, rules and parameters, not only consumes a lot of time and manpower, but also increases the complexity of processing large-scale data sets; (4) Ignoring time series characteristics: These methods usually do not consider the time correlation between data points. However, in actual production life, there is a large amount of time series data, and adjacent data points have a high degree of dependence and continuity. Using the mean, mode or median to fill may destroy this time dependence, resulting in a large difference between the corrected data and the actual situation; (5) It will introduce additional noise: When using statistical methods such as the mean or median to fill out outliers, the selected filling value does not represent the actual data pattern near the outlier point, which will introduce illogical noise and cause the overall data to deviate. Summary of the invention

[0004] In view of this, in order to solve the problem that the traditional data correction method ignores the influence of time trends in data with time series characteristics, resulting in the corrected data still containing noise, thereby reducing the generalization ability and accuracy of model training, the present invention provides an industrial time series data anomaly bidirectional correction method, comprising the following steps:

[0005] S1, get a time series based data segment x1, x2, x3, x4, ..., x k , …, x n-1 , x n, where n is a positive integer, indicating the total number of recorded data points;

[0006] S2. Determine the abnormal data segment x k+1 , …, x k+u ;

[0007] S3, performing forward correction processing on the abnormal data in the original data segment to obtain a forward correction processing result;

[0008] S301, starting from the first abnormal data point of the abnormal data segment, calculate the abnormal data point x k+i The first p data points m p , m p-1 , m p-2 ,…, weight corresponding to m1

[0009] S302, calculate abnormal data point x k+i Forward processing value x k+i(+) , the formula is as follows:

[0010]

[0011] S303, repeating step S301 and step S302 until the forward processing values ​​of all abnormal data points are obtained;

[0012] S4, performing a backward correction process on the abnormal data in the original data segment to obtain a backward correction process result;

[0013] S401, starting from the last abnormal data point of the abnormal data segment, calculate the abnormal data point x k+i The next q non-abnormal data points n1, n2, n3, ..., n q The corresponding weight

[0014] S402, calculate abnormal data point x k+i Postprocessing value x k+i(-) , the formula is as follows:

[0015]

[0016] S403, repeating step S401 and step S402 until the backward processing values ​​of all abnormal data points are obtained;

[0017] S5. Perform comprehensive bidirectional correction on the abnormal data fragment to obtain a final bidirectional correction value and replace the original abnormal data.

[0018] Preferably, the forward correction weight and the backward correction weight are calculated as follows:

[0019] Calculate the distance dis(x between each data point and the abnormal data point k+i , x l ), the formula is as follows:

[0020] dis(x k+i , x l )=| position(x k+i )-position(x l )|

[0021] Among them, position() represents the position function, x k+i represents an abnormal data point and i≤u, x l represents a data point and l∈[1,n];

[0022] Calculate the weight of each abnormal data point weight(x k+i , x l ), the formula is as follows:

[0023] weight(x k+i , x l )=1 / dis(x k+i , x l )

[0024] Calculate the forward corrected normalized weight α(x k+i , x l ), the formula is as follows:

[0025]

[0026] Where p represents the number of non-abnormal data selected in the forward direction of the abnormal data segment;

[0027] Calculate the backward corrected normalized weight β(x k+i , x l ), the formula is as follows:

[0028]

[0029] Among them, q represents the number of non-abnormal data backward selected from the abnormal data segment.

[0030] Preferably, the comprehensive two-way correction is obtained by weighted summing the results obtained by the forward correction and the backward correction to obtain the final two-way correction value, and the expression is as follows:

[0031] x k+i =τ i *x k+i (+)+(1-τ i )*x k+i(-)

[0032] Among them, τ i Represents an abnormal data point x k+i The weight of the forward correction processing value.

[0033] Preferably, the weight τ of the forward correction processing value i The abnormal data point x k+i At the location in the abnormal data fragment, the expression is as follows:

[0034]

[0035] When τ i If it is close to 1, it indicates an abnormal data point x k+i Close to the beginning of the abnormal data segment;

[0036] When τ i If it is close to 0, it indicates an abnormal data point x k+i Near the end of the anomalous data segment;

[0037] When τ i If it is close to 0.5, it indicates an abnormal data point x k+i Near the middle of the abnormal data segment

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention fully considers the time characteristics of data in industrial production, performs bidirectional correction based on the window data before and after the moment when the abnormal data point is located, takes into account the relative position and trend of the data point in the time series, and assigns corresponding weights according to the position of the abnormal data point in the abnormal segment for weighted correction. It fully considers the time correlation with the data before and after the abnormal segment, reduces the errors that may be introduced in the correction process, and effectively avoids the problem of loss of time series characteristics caused by taking the mean or mode to correct the data, providing support for improving data quality and establishing a more accurate prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A fragment diagram of the original data obtained for the present invention;

[0041] Figure 2 This is a schematic diagram of the forward correction of abnormal fragments of the present invention;

[0042] Figure 3 This is a schematic diagram of the backward correction of abnormal fragments of the present invention;

[0043] Figure 4 This is a schematic diagram of bidirectional correction of abnormal fragments according to the present invention;

[0044] Figure 5 This is a visualization diagram of abnormal data in an embodiment of the present invention;

[0045] Figure 6 This is a visualization diagram of the bidirectional correction results of abnormal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0047] Example 1

[0048] The present invention provides a method for bidirectionally correcting abnormal industrial time series data, comprising the following steps:

[0049] S1, get a time series based data segment x1, x2, x3, x4, ..., x k , …, x n-1 , x n , where n is a positive integer, indicating the total number of recorded data points;

[0050] S2. Determine the abnormal data segment x k+1 , …, x k+u ,like Figure 1 As shown;

[0051] S3, perform forward correction processing on the abnormal data in the original data segment to obtain the forward correction processing result, such as Figure 2 As shown;

[0052] S301, starting from the first abnormal data point of the abnormal data segment, calculate the abnormal data point x k+i The first p data points m p , m p-1 , m p-2 ,…, weight corresponding to m1 These p data points contain the original outliers, which ensures that these forward data can be effectively used for subsequent outlier processing;

[0053] S302, calculate abnormal data point x k+i Forward processing value x k+i(+) , the formula is as follows:

[0054]

[0055] S303, repeating step S301 and step S302 until the forward processing values ​​of all abnormal data points are obtained;

[0056] S4, perform back-correction processing on the abnormal data in the original data segment to obtain the back-correction processing result, such as Figure 3 As shown;

[0057] S401. Different from the forward correction, the backward correction starts from the last data point of the abnormal data segment and calculates the abnormal data point x k+i The next q non-abnormal data points n1, n2, n3, ..., n q The corresponding weight

[0058] S402, calculate abnormal data point x k+i Postprocessing value x k+i(-) , the formula is as follows:

[0059]

[0060] S403, repeating step S401 and step S402 until the backward processing values ​​of all abnormal data points are obtained;

[0061] The forward correction weight α and the backward correction weight β are determined by the data point x l and the outlier point x k+i The distance between is determined by: distance x k+i The closer, the closer to x k+i The closer the true value is, the greater the weight is, and the distance x k+i The farther away, the closer to x k+i The larger the difference between the true values, the smaller the corresponding weight; the calculation method of the forward correction weight α and the backward correction weight β is as follows:

[0062] Calculate the distance dis(x between each data point and the abnormal data point k+i , x l ), the formula is as follows:

[0063] dis(x k+i , x l )=| position(x k+i )-position(x l )|

[0064] Among them, position() represents the position function, x k+i represents an abnormal data point and i≤u, x l represents a data point and l∈[1,n];

[0065] Calculate the weight of each abnormal data point weight(x k+i , x l ), the formula is as follows:

[0066] weight(x k+i , x l )=1 / dis(x k+i , x l )

[0067] Calculate the forward corrected normalized weight α(x k+i , x l ), the formula is as follows:

[0068]

[0069] Where p represents the number of non-abnormal data selected in the forward direction of the abnormal data segment;

[0070] Calculate the backward corrected normalized weight β(x k+i , x l ), the formula is as follows:

[0071]

[0072] Among them, q represents the number of non-abnormal data backward selected from the abnormal data segment;

[0073] The values ​​of p and q are determined by the number of continuous normal data in their respective directions, while ensuring that the number of non-abnormal data is greater than or equal to one;

[0074] S5. Perform comprehensive bidirectional correction on the abnormal data fragment to obtain the final bidirectional correction value and replace the original abnormal data, such as Figure 4 As shown;

[0075] The comprehensive two-way correction is obtained by weighted summing the results of the forward correction and the backward correction to obtain the final two-way correction value. The expression is as follows:

[0076] x k+i =τ i *x k+i(+) +(1-τ i )*x k+i(-)

[0077] Among them, τ i Represents an abnormal data point x k+i The weight of the forward correction processing value;

[0078] The weight of the forward correction processing value τ i The abnormal data point x k+i At the location in the abnormal data fragment, the expression is as follows:

[0079]

[0080] When τ i If it is close to 1, it indicates an abnormal data point x k+i Close to the beginning of the abnormal data segment;

[0081] When τ iIf it is close to 0, it indicates an abnormal data point x k+i Near the end of the anomalous data segment;

[0082] When τ i If it is close to 0.5, it indicates an abnormal data point x k+i Near the middle of the anomalous data segment.

[0083] Example 2

[0084] This embodiment is to correct abnormal data of strip steel production data.

[0085] Strip steel production has the characteristics of a multi-link continuous process. In each production link, relevant production data is collected by using sensors or manual recording. Due to the harsh environment of the steel production site, data collection is difficult, resulting in a small number of strip steel data samples, and sensor failures or human recording errors may occur during the collection process, reducing the quality of the data. Therefore, the collected abnormal data needs to be corrected.

[0086] In the data set of strip steel grades, element characteristics are added in proportion to production quality standards without special treatment. Indicators such as production line status and equipment working conditions play a key role in monitoring and optimizing the production process and are essential to maintaining the performance of strip steel production; abnormal data results of some unit parameters are as follows Figure 5 As shown in the figure, through graphical analysis, abnormal data points mainly appear at the extreme values ​​of numerical mutations; the differences between these abnormal points and the previous and subsequent data values ​​are usually large, which does not conform to the continuity characteristics of time series data; in the data distribution of the tension of the straightening machine, not only outlier abnormal points are identified, but also some abnormal points are found to show a clustering trend, which indicates that the production line may have operating failures for a period of time.

[0087] In order to preserve the data distribution characteristics as much as possible, a bidirectional correction method for abnormal industrial time series data provided by the present invention is used to correct the data. The data correction result is as follows: Figure 6 As shown in the figure; at the extreme mutation position, the data corrected by the two-way correction method is basically consistent with the trend of the previous and next data values, and the fluctuation range in most areas is reduced, indicating that while reducing the outlier noise, it also smoothes some abnormal fluctuations, retaining the main features of the data while removing extreme values.

[0088] At the same time, the original data of the strip steel, the median-corrected data, the average-corrected data, and the data corrected by the present embodiment are used to train a variety of machine learning models to predict the mechanical properties of the strip steel production, and the root mean square error RMSE of the prediction results is calculated. Each model is trained using default parameters; tensile strength and yield strength are used as mechanical property prediction indicators. The results are shown in Tables 1 and 2, which describe ten machine learning fitting feature variables and the prediction errors of tensile strength and yield strength;

[0089] Table 1 Comparison of RMSE of tensile strength predicted by different machine learning models

[0090]

[0091] Table 2 Comparison of RMSE of yield strength predicted by different machine learning models

[0092]

[0093] In terms of the tensile strength index, most of the machine learning models achieved high prediction accuracy on the data set after correction in this embodiment. In terms of the yield strength index, the root mean square error RMSE of the six models on the data set after correction in this embodiment reached the lowest, achieving high prediction accuracy. Therefore, the prediction performance of the data set after correction in this embodiment on most models was optimally improved, verifying the effectiveness of the bidirectional correction method provided by the present invention.

[0094] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A bidirectional correction method for abnormal industrial time series data, characterized in that: The following steps are involved: S1, get a time series based data segment x1, x2, x3, x4, ..., x k , ..., x n-1 , x n , where n is a positive integer, indicating the total number of recorded data points; S2. Determine the abnormal data segment x k+1 , …, x k+u ; S3, performing forward correction processing on the abnormal data in the original data segment to obtain a forward correction processing result; S301, starting from the first abnormal data point of the abnormal data segment, calculate the abnormal data point x k+i The first p data points m p , m p-1 , m p-2 ,…, weight corresponding to m1 S302, calculate abnormal data point x k+i Forward processing value x k+i(+) , the formula is as follows: S303, repeating step S301 and step S302 until the forward processing values ​​of all abnormal data points are obtained; S4, performing a backward correction process on the abnormal data in the original data segment to obtain a backward correction process result; S401, starting from the last abnormal data point of the abnormal data segment, calculate the abnormal data point x k+i The next q non-abnormal data points n1, n2, n3, ..., n q The corresponding weight S402, calculate abnormal data point x k+i Postprocessing value x k+i(-) , the formula is as follows: S403, repeating step S401 and step S402 until the backward processing values ​​of all abnormal data points are obtained; S5. Perform comprehensive bidirectional correction on the abnormal data fragment to obtain a final bidirectional correction value and replace the original abnormal data.

2. The method for bidirectional correction of abnormal industrial time series data according to claim 1, characterized in that: The forward correction weight and backward correction weight are calculated as follows: Calculate the distance dis(x between each data point and the abnormal data point k+i , x l ), the formula is as follows: dis(x k+i ,x l )=|position(x k+i )-position(x l )| Among them, position() represents the position function, x k+i represents an abnormal data point and i≤u, x l represents a data point and l∈[1,n]; Calculate the weight of each abnormal data point weight(x k+i , x l ), the formula is as follows: weight(x k+i ,x l )=1 / dis(x k+i ,x l ) Calculate the forward corrected normalized weight α(x k+i , x l ), the formula is as follows: Where p represents the number of non-abnormal data selected in the forward direction of the abnormal data segment; Calculate the backward corrected normalized weight β(x k+i , x l ), the formula is as follows: Among them, q represents the number of non-abnormal data backward selected from the abnormal data segment.

3. The method for bidirectional correction of abnormal industrial time series data according to claim 1, characterized in that: The comprehensive two-way correction is obtained by weighted summing the results of the forward correction and the backward correction to obtain the final two-way correction value, and the expression is as follows: Among them, τ i Represents an abnormal data point x k+i The weight of the forward correction processing value.

4. The method for bidirectional correction of abnormal industrial time series data according to claim 3, characterized in that: The weight of the forward correction processing value τ i The abnormal data point x k+i At the location in the abnormal data fragment, the expression is as follows: When τ i If it is close to 1, it indicates an abnormal data point x k+i Close to the beginning of the abnormal data segment; When τ i If it is close to 0, it indicates an abnormal data point x k+i Near the end of the anomalous data segment; When τ i If it is close to 0.5, it indicates an abnormal data point x k+i Near the middle of the anomalous data segment.

Images