Abnormal data real-time processing method and system

Abstract

The invention provides an abnormal data real-time processing method and system. The method comprises the following steps: receiving an equipment acquisition value in real time; setting a sliding window, and performing sliding selection on the acquired value; calculating a fluctuation value according to the collected value selected in the sliding window; performing real-time data value updating on the current acquisition value according to the fluctuation value of the sliding window at the current moment and the fluctuation value of the sliding window at the previous moment; and updating the fluctuation value at the current moment by using the updated data value so as to update the subsequently received acquisition value in real time. According to the invention, equipment monitoring abnormal data can be processed in real time, data stability and real change tendency are ensured, and monitoring false alarms are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a real-time processing method and system for abnormal data, and in particular to a real-time processing method for abnormal data of early warning equipment suitable for the field of geological disasters, which can reduce equipment false alarms and improve the accuracy of geological disaster early warnings. Background Art

[0002] Geological disasters occur frequently in my country, threatening people's lives and property. The life cycle of geological disasters includes the incubation period, slow change period, acceleration period, collapse period and other processes. The use of automated early warning equipment to monitor geological disasters is an important means of geological disaster early warning and forecasting. The automated early warning equipment for geological disasters uses contact means to collect a variety of physical characteristic data (such as gravity acceleration, displacement and other characteristics) of geological disaster targets throughout their life cycle, and monitor the trend changes of geological disaster characteristic parameters in the vertical or horizontal direction. When the cumulative trend change exceeds the critical value, the geological disaster early warning forecast is triggered.

[0003] Due to the complex geological disaster environment, the data collection process of early warning equipment is frequently disturbed by the field environment, resulting in abnormal physical feature data collected in real time. Abnormal data are mostly values ​​far exceeding the critical point of early warning, causing false alarms and a high false alarm rate for geological disaster monitoring. Therefore, solving abnormal data interference in real time, reducing false alarms, and retaining the true trend changes of data are the key points to improve the accuracy of geological disaster monitoring and early warning. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for real-time processing of abnormal data, so as to monitor abnormal data in real time, ensure data stability and true change trend, and reduce false alarms in monitoring.

[0005] In order to achieve the above object, the present invention provides a method for real-time processing of abnormal data, comprising the following steps:

[0006] Receive device collection values ​​in real time;

[0007] Set the sliding window to slide and select the collected values;

[0008] Calculate the fluctuation value based on the selected collected values ​​in the sliding window;

[0009] The current collected value is updated in real time according to the fluctuation value of the sliding window at the current moment and the fluctuation value of the previous moment;

[0010] The updated data value is used to update the fluctuation value at the current moment, so as to update the subsequently received collected values ​​in real time.

[0011] Furthermore, the method for sliding selection of the collected values ​​is: sliding selection of the received collected values ​​is performed with a sliding window of a set step length, and each time a collected value is received, the sliding window slides one step to select it.

[0012] Furthermore, the sliding window length includes 4 to 6 collection values.

[0013] Furthermore, the calculation method of the fluctuation value of the sliding window at the current moment is:

[0014]

[0015] The calculation method of the fluctuation value of the sliding window at the previous moment is:

[0016]

[0017] Where W is the sliding window length, V i is the i-th collected value in the sliding window, V 基准采集值 is the reference acquisition value, V 当前采集值 is the current collection value.

[0018] Furthermore, the method for updating the real-time data value of the current collected value is as follows:

[0019] Set the weight coefficient of the current collection value

[0020]

[0021] Update the fluctuation value of the sliding window at the current moment

[0022]

[0023] Update the weight coefficient of the current collection value;

[0024]

[0025] Update the current collection value to:

[0026] V 更新值 =(1-K1)*V 基准采集值 +K1*V 当前采集值 .

[0028] Furthermore, it is used for real-time processing of abnormal data of geological disaster early warning equipment.

[0029] The present invention also provides a real-time abnormal data processing system, which adopts the above-mentioned real-time abnormal data processing method. The real-time abnormal data processing system includes:

[0030] A data acquisition module, used for receiving the collected values ​​output by the device;

[0031] A sliding window control module, used for sliding selected collection values;

[0032] A fluctuation value calculation module is used to calculate the fluctuation value of the collected value in the sliding window;

[0033] A collection value update module, used to update the current collection value;

[0034] The storage module is used to store the collected values ​​at each moment, as well as the updated collected values ​​and fluctuation values.

[0035] Furthermore, it is used for real-time processing of abnormal data of geological disaster early warning equipment.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The present invention provides a real-time processing method for abnormal data, which counts the data fluctuation value of the sliding window at the previous moment and the data fluctuation value at the current moment, assigns different weight coefficients to the baseline collection value and the current collection value according to the change in the previous and next data fluctuation values, and dynamically updates the weight coefficients to update the current moment data. This method can process the abnormal data monitored by the equipment in real time, ensure the data stability and the real change trend, and reduce false alarms of monitoring.

[0038] The present invention is sensitive to rapidly changing measurement parameters (as long as two trend change points in the sliding window will cause the fluctuation value to change, and then cause the predicted data and the updated value to change), and can timely predict data changes and reduce data change delays.

[0039] The present invention adaptively gives a suitable weight ratio according to the degree of data anomaly, so that the variation range of time series data is more convergent and controllable. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a method for real-time processing of abnormal data provided by a specific embodiment of the present invention;

[0042] Figure 2 A flow chart of a method for real-time processing of abnormal data provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below through implementation modes in conjunction with the accompanying drawings.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. Unless otherwise specifically stated, the relative arrangement of the components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0045] As one aspect of the present invention, a method for real-time processing of abnormal data is proposed, comprising the following steps of receiving device collection values ​​in real time;

[0046] Set the sliding window to slide and select the collected values;

[0047] Calculate the fluctuation value based on the selected collected values ​​in the sliding window;

[0048] The current collected value is updated in real time according to the fluctuation value of the sliding window at the current moment and the fluctuation value of the previous moment;

[0049] The updated data value is used to update the fluctuation value at the current moment, so as to update the subsequently received collected values ​​in real time.

[0050] The method for real-time processing of abnormal data can process abnormal data monitored by the equipment in real time, ensure data stability and true change trend, and reduce false alarms of monitoring.

[0051] Furthermore, the method for sliding selection of the collected values ​​is: sliding selection of the received collected values ​​with a sliding window of a set step length, and each time a collected value is received, the sliding window slides one step to select it. To ensure correctness, the sliding window length includes multiple collected values, such as 4 to 10.

[0052] Furthermore, the calculation method of the fluctuation value of the sliding window at the current moment is:

[0053]

[0054] The calculation method of the fluctuation value of the sliding window at the previous moment is:

[0055]

[0056] Where W is the sliding window length, V i is the i-th collected value in the sliding window, V 基准采集值 is the reference acquisition value (i.e., the data value after the acquisition value at the last moment is updated and calculated by the method of the present invention), V 当前采集值 is the current collection value.

[0057] Furthermore, the method for updating the real-time data value of the current collected value is as follows:

[0058] Set the weight coefficient of the current collection value

[0059]

[0060] Update the fluctuation value of the sliding window at the current moment

[0061]

[0062] Update the weight coefficient of the current collection value;

[0063]

[0064] Update the current collection value to:

[0065] V 更新值 =(1-K1)*V 基准采集值 +K1*V 当前采集值

[0066] The updated data value is also used as the benchmark collection value at the next moment.

[0067] As another aspect of the present invention, a real-time abnormal data processing system is also proposed, comprising a data acquisition module for receiving a collection value output by a device;

[0068] A sliding window control module, used for sliding selected collection values;

[0069] A fluctuation value calculation module is used to calculate the fluctuation value of the collected value in the sliding window;

[0070] A collection value update module, used to update the current collection value;

[0071] The storage module is used to store the collected values ​​at each moment, as well as the updated collected values ​​and fluctuation values.

[0072] By adopting the above-mentioned real-time abnormal data processing system, abnormal data monitored by the equipment can be processed in real time, ensuring data stability and true change trends, and reducing false monitoring alarms.

[0073] Taking the data processing of geological disaster equipment monitoring as a specific embodiment, a real-time processing method for abnormal data is provided below, which processes abnormal data of geological disaster equipment monitoring in real time, ensures data stability and true change trend, and reduces monitoring false alarms. The real-time processing method is based on the premise that when the disaster body does not undergo trend changes or changes slowly, the change in the data values ​​collected at two adjacent moments is very small. Set the data at the previous moment as the benchmark collection value, which is the data at the previous moment obtained by processing the historical collection value by the method described in the present invention, which is close to the real data and is used to estimate the data at the current moment. The current collection value is the real-time collection data. The data fluctuation value in the sliding window with a time step of W at the previous moment (including the benchmark collection value) and the data fluctuation value in the sliding window with a time step of W at the current moment (including the current collection value) are counted. According to the change in the fluctuation value of the previous and subsequent data, different weight coefficients are assigned to the benchmark collection value and the current collection value, and the weight coefficient is dynamically updated. If the current collection value is an abnormal value, a smaller weight is obtained and the current collection value is updated. Specifically, Figure 1 shown.

[0074] A flow chart of a method for real-time processing of abnormal data in this embodiment is as follows Figure 2 As shown, the specific implementation steps are as follows:

[0075] Step 1: Set the sliding window step size W

[0076] The sliding window mainly counts the fluctuations of the benchmark collection value and the current collection value, and assigns the corresponding weight coefficients to the benchmark collection value and the current collection value according to the fluctuations. The sliding window W value should not be too small. When the number of consecutive abnormal points is greater than the sliding window length, it affects the weight of the abnormal data and cannot remove the abnormal values. However, a sliding window length that is too long will cause a delay in the real trend change and affect the timeliness of the real alarm.

[0077] In this embodiment, based on a large amount of geological disaster warning data, the influence of different sliding window sizes on the removal effect of abnormal data and the timeliness of real trend changes is comprehensively considered, and the preferred sliding window length W is selected to be 4 to 6 collected values.

[0078] In other embodiments, the sliding window length W is selected according to the actual use environment of the safety monitoring equipment.

[0079] Step 2: Calculate the fluctuation value of the collected value in the sliding window

[0080] The fluctuation value (Var1) of the data in the sliding window at the previous moment (including the benchmark collection value) and the fluctuation value (Var2) of the data in the window at the current moment (including the current collection value) are counted with a window with a step size of W, as shown in Formula 1 and Formula 2.

[0081]

[0082]

[0083] Where W is the sliding window length, V i is the i-th collected value in the sliding window, V 基准采集值 is the reference acquisition value, V 当前采集值 is the current collection value.

[0084] Step 3: Calculate the weight coefficient

[0085] Combining Var1 and Var2, the weight coefficients of the benchmark acquisition value and the current acquisition value are calculated, as shown in Formula 3. K is the weight coefficient of the current acquisition value, and 1-K is the weight coefficient of the benchmark acquisition value.

[0086]

[0087] In order to update the abnormal collection value to be close to the benchmark collection value, it is necessary to assign a smaller weight to the abnormal collection value, that is, the larger the abnormal data value, the smaller the weight coefficient K1 is assigned. Therefore, it is necessary to amplify the fluctuation value of the sliding window data where the abnormal collection value is located, as shown in Formula 4.

[0088]

[0089] Therefore, the current acquisition value weight coefficient is updated to formula (5)

[0090]

[0091] Step 4: Update current collection value

[0092] Substitute the current acquisition value weight update coefficient K1 and the estimated acquisition value weight coefficient 1-K1 obtained above into formula (6), update the current acquisition value, and dynamically update NewVar2 and Var1 in real time.

[0093] V 更新值 =(1-K1)*V 预估采集值 +K1*V 当前采集值 (6)

[0094] V calculated by formula (6) 更新值 That is, the reference collection value corresponding to the collection value at the next moment.

[0095] Compared with the existing processing technology that uses historical data to predict the next moment, the present invention has the following advantages: 1. The present invention is sensitive to rapidly changing measurement parameters and can predict data changes in a timely manner, reducing data change delays; 2. The present invention adaptively gives a suitable weight ratio according to the degree of data anomaly, making the range of time series data changes more convergent and controllable.

[0096] The present invention is suitable for real-time processing of abnormal data of geological disaster early warning equipment and other safety monitoring equipment.

[0097] Features described and / or illustrated above for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or used in place of features in other embodiments.

[0098] It should be emphasized that the term "include / comprises" when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps, components or combinations thereof.

[0099] The many features and advantages of these embodiments are apparent from this detailed description, and thus the appended claims are intended to cover all such features and advantages of these embodiments that fall within their true spirit and scope. Furthermore, since numerous modifications and changes will readily occur to those skilled in the art, it is not intended that the embodiments of the invention be limited to the exact construction and operation illustrated and described, but rather all suitable modifications and equivalents falling within the scope thereof are intended to be covered.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0101] Parts of the present invention that are not described in detail are well known to those skilled in the art.

Claims

1. A method for real-time processing of abnormal data, characterized in that: The steps include: Receive device collection values ​​in real time; Set the sliding window to slide and select the collected values; Calculate the fluctuation value based on the selected collected values ​​in the sliding window; The current collected value is updated in real time according to the fluctuation value of the sliding window at the current moment and the fluctuation value of the previous moment; The updated data value is used to update the fluctuation value at the current moment, so as to update the subsequently received collected values ​​in real time.

2. The method for real-time processing of abnormal data according to claim 1, characterized in that: The method for sliding selection of the collected values ​​is: sliding selection of the received collected values ​​is performed with a sliding window of a set step length, and each time a collected value is received, the sliding window slides one step to select it.

3. The method for real-time processing of abnormal data according to claim 2, characterized in that: The sliding window length includes 4 to 6 acquisition values.

4. The method for real-time processing of abnormal data according to claim 1, characterized in that: The calculation method of the fluctuation value of the sliding window at the current moment is: The calculation method of the fluctuation value of the sliding window at the previous moment is: Where W is the sliding window length, V i is the i-th collected value in the sliding window, V 基准采集值 is the reference acquisition value, V 当前采集值 is the current collection value.

5. The method for real-time processing of abnormal data according to claim 4, characterized in that: The method for updating the real-time data value of the current collection value is as follows: Set the weight coefficient of the current collection value Update the fluctuation value of the sliding window at the current moment Update the weight coefficient of the current collection value; Update the current collection value to: In 更新值 =(1-K1)*V 基准采集值 +K1*V 当前采集值 。 6. The method for real-time processing of abnormal data according to any one of claims 1 to 5, characterized in that: Used for real-time processing of abnormal data of geological disaster early warning equipment.

7. A real-time abnormal data processing system, characterized in that: The method for real-time processing of abnormal data according to any one of claims 1 to 5 is adopted, wherein the real-time processing system for abnormal data comprises A data acquisition module, used for receiving the collected values ​​output by the device; A sliding window control module, used for sliding selected collection values; A fluctuation value calculation module is used to calculate the fluctuation value of the collected value in the sliding window; A collection value update module, used to update the current collection value; The storage module is used to store the collected values ​​at each moment, as well as the updated collected values ​​and fluctuation values.

8. The real-time abnormal data processing system according to claim 7, characterized in that: Used for real-time processing of abnormal data of geological disaster early warning equipment.

Images