Data anomaly detection and correction method and device

Through the autoregressive prediction model, the abnormalities in energy data are corrected in a timely manner, and the problem of data anomaly detection and correction delay in the prior art is solved, which improves the efficiency and accuracy of data processing.

CN119025306APending Publication Date: 2024-11-26SCHNEIDER SMART TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310604649.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art data abnormality detection and correction process, there is a delay in the correction link, and the abnormal data cannot be corrected in time, which affects the efficiency and accuracy of the data link and subsequent data processing.

Method used

The autoregressive prediction model is used to obtain the prediction data of the variables to be measured, and the real data is detected and corrected based on the detection rules to correct the abnormal data in a timely manner.

Benefits of technology

Improve the efficiency and accuracy of data processing, correct abnormal data in a timely manner, and avoid abnormal effects in the data link.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025306A_ABST
    Figure CN119025306A_ABST
Patent Text Reader

Abstract

The invention provides a data anomaly detection and correction method and device, and the method comprises the steps: obtaining prediction data of a to-be-detected variable at a first moment based on an autoregression prediction model; performing anomaly detection on the real data of the to-be-detected variable at the first moment based on the first detection rule; and if the result of the anomaly detection is abnormal, correcting the real data into predicted data. By means of the method, when it is detected that the real data of the to-be-detected variable at the first moment is abnormal data, the real data of the to-be-detected variable at the first moment can be corrected into the prediction data of the to-be-detected variable at the first moment in time. Data processing can be carried out in time subsequently according to the prediction data, the efficiency of subsequent data processing is higher, and the accuracy is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for detecting and correcting data anomalies. Background Art

[0002] With the development of digital energy technology, the quality of energy data is becoming more and more important. However, in the energy field, the data link is long, including sensor data collection, data aggregation, data transmission, data conversion, and data basic calculation. Any abnormality in any of these links may lead to abnormal phenomena such as data loss, value mutation, and data discreteness. Based on this, abnormal detection and correction of energy data is crucial.

[0003] At present, in most methods of data anomaly detection and correction, data anomaly detection and correction of abnormal data are divided into two independent processing processes. Among them, the correction link is a passive repair, which needs to rely on the latest data. There is a certain delay in the correction process, and the abnormal data cannot be corrected in time.

[0004] Based on this, in the process of data anomaly detection and correction, how to correct abnormal data in a timely manner has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In order to solve the problem of how to correct abnormal data in a timely manner during the process of data anomaly detection and correction, the embodiment of the present application provides a data anomaly detection and correction method and device.

[0006] In a first aspect, an embodiment of the present application provides a method for data anomaly detection and correction, the method comprising: obtaining predicted data of a variable to be measured at a first moment based on an autoregressive prediction model; performing anomaly detection on real data of the variable to be measured at the first moment based on a first detection rule; if the result of the anomaly detection is abnormal, correcting the real data to the predicted data.

[0007] In a possible implementation, the variable to be measured is an energy data variable, and the energy data variable includes electricity consumption, water consumption or gas consumption; the method also includes: obtaining the value of the energy data variable of the first load device in the target building at the first moment through a sensing device; the sensing device is an electric meter, the first load device includes lighting equipment and / or elevator equipment, and the energy data variable is electricity consumption; or, the sensing device is a water meter, the first load device includes water-using equipment, and the energy data variable is water consumption; or, the sensing device is a gas meter or a gas meter, the first load device includes gas-using equipment, and the energy data variable is gas consumption; determine the value of the energy data variable at the first moment as the real data.

[0008] In one possible implementation, the variable to be measured is operating power; the method also includes: obtaining the value of the operating power of a second load device in the power grid management platform at the first moment; the second load device includes a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device; and determining the value of the operating power at the first moment as the real data.

[0009] In a possible implementation, the variable to be measured is an operating data variable, and the operating data variable includes a start-up frequency, an instantaneous temperature, or a pressure; the method further includes: obtaining the value of the operating data variable of the Internet of Things device at the first moment; and determining that the value of the operating data variable at the first moment is the real data.

[0010] In one possible implementation, before performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule, the method also includes: determining an upper detection threshold and a lower detection threshold; and determining the first detection rule based on the upper detection threshold and the lower detection threshold.

[0011] In a possible implementation, the obtaining of predicted data of the variable to be measured at the first moment based on the autoregressive prediction model includes: obtaining the predicted data, predicted upper limit value and predicted lower limit value of the variable to be measured at the first moment based on the autoregressive prediction model; the predicted data is greater than or equal to the predicted lower limit value, and less than or equal to the predicted upper limit value; the determining of the detection upper limit threshold and the detection lower limit threshold includes: respectively determining the prediction upper limit value and the prediction lower limit value as the detection upper limit threshold and the detection lower limit threshold.

[0012] In one possible implementation, the determining of the detection upper limit threshold and the detection lower limit threshold includes: determining a first upper limit threshold and a first lower limit threshold based on the predicted data; and determining the first upper limit threshold and the first lower limit threshold as the detection upper limit threshold and the detection lower limit threshold, respectively.

[0013] In one possible implementation, the determination of the detection upper limit threshold and the detection lower limit threshold includes: determining a first upper limit threshold and a first lower limit threshold based on the predicted data; determining a second upper limit threshold and a second lower limit threshold based on the first historical data of the variable to be measured before the first moment; determining a third upper limit threshold and a third lower limit threshold, the third upper limit threshold being a value obtained by merging the first upper limit threshold and the second upper limit threshold, and the third lower limit threshold being a value obtained by merging the first lower limit threshold and the second lower limit threshold; and determining the third upper limit threshold and the third lower limit threshold as the detection upper limit threshold and the detection lower limit threshold, respectively.

[0014] In a possible implementation, the second upper threshold and the second lower threshold are determined based on the first historical data of the variable to be measured before the first moment, including: obtaining the first historical data of the variable to be measured before the first moment; according to a preset quantile rule, respectively obtaining the high quantile data and the low quantile data included in the first historical data; and respectively determining the high quantile data and the low quantile data as the second upper threshold and the second lower threshold.

[0015] In one possible implementation, the second upper threshold and the second lower threshold are determined based on the first historical data of the variable to be measured before the first moment, including: obtaining the historical maximum data and the historical minimum data included in the first historical data of the variable to be measured before the first moment; determining the second upper threshold based on the historical maximum data; determining the second lower threshold based on the historical minimum data.

[0016] In one possible implementation, based on the first detection rule, anomaly detection is performed on the real data of the variable to be measured at the first moment, including: determining the size relationship between the real data and the detection upper limit threshold and the detection lower limit threshold; if the real data is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the result of the anomaly detection is normal; or, if the real data is less than the detection lower limit threshold, or, greater than the detection upper limit threshold, then the result of the anomaly detection is abnormal.

[0017] In a possible implementation, the method further includes: training and generating the autoregressive prediction model based on second historical data of the variable to be measured before the first moment.

[0018] In the second aspect, an embodiment of the present application also provides a data anomaly detection and correction device, which includes: a first acquisition module, used to obtain the predicted data of the variable to be measured at the first moment based on an autoregressive prediction model; an anomaly detection module, used to perform anomaly detection on the real data of the variable to be measured at the first moment based on a first detection rule; and a correction module, used to correct the real data to the predicted data if the result of the anomaly detection is abnormal.

[0019] In one possible implementation, the variable to be measured is an energy data variable, and the energy data variable includes electricity consumption, water consumption or gas consumption; the device also includes: a second acquisition module, used to obtain the value of the energy data variable of the first load device in the target building at the first moment through a sensing device; the sensing device is an electric meter, the first load device includes lighting equipment and / or elevator equipment, and the energy data variable is electricity consumption; or, the sensing device is a water meter, the first load device includes water-using equipment, and the energy data variable is water consumption; or, the sensing device is a gas meter or a gas meter, the first load device includes gas-using equipment, and the energy data variable is gas consumption; a first determination module, used to determine that the value of the energy data variable at the first moment is the real data.

[0020] In one possible implementation, the variable to be measured is operating power; the device also includes: a third acquisition module, used to obtain the value of the operating power of a second load device in the power grid management platform at the first moment; the second load device includes a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device; a second determination module, used to determine that the value of the operating power at the first moment is the real data.

[0021] In one possible implementation, the variable to be measured is an operating data variable, and the operating data variable includes a start-up frequency, an instantaneous temperature or a pressure; the device also includes: a fourth acquisition module, used to obtain the value of the operating data variable of the Internet of Things device at the first moment; and a third determination module, used to determine that the value of the operating data variable at the first moment is the real data.

[0022] In one possible implementation, the device also includes: a fourth determination module, used to determine a detection upper limit threshold and a detection lower limit threshold before performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule; and a fifth determination module, used to determine the first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0023] In one possible implementation, the first acquisition module is used to obtain the predicted data of the variable to be measured at the first moment based on the autoregressive prediction model, specifically: the first acquisition module is used to obtain the predicted data, predicted upper limit value and predicted lower limit value of the variable to be measured at the first moment based on the autoregressive prediction model; the predicted data is greater than or equal to the predicted lower limit value, and less than or equal to the predicted upper limit value; the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to respectively determine the prediction upper limit value and the prediction lower limit value as the detection upper limit threshold and the detection lower limit threshold.

[0024] In one possible implementation, the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to: determine a first upper limit threshold and a first lower limit threshold based on the predicted data; and determine the first upper limit threshold and the first lower limit threshold as the detection upper limit threshold and the detection lower limit threshold, respectively.

[0025] In one possible implementation, the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to: determine a first upper limit threshold and a first lower limit threshold based on the predicted data; determine a second upper limit threshold and a second lower limit threshold based on the first historical data of the variable to be measured before the first moment; determine a third upper limit threshold and a third lower limit threshold, the third upper limit threshold being a value obtained by merging the first upper limit threshold and the second upper limit threshold, and the third lower limit threshold being a value obtained by merging the first lower limit threshold and the second lower limit threshold; respectively determine the third upper limit threshold and the third lower limit threshold as the detection upper limit threshold and the detection lower limit threshold.

[0026] In one possible implementation, the fourth determination module is used to determine the second upper threshold and the second lower threshold based on the first historical data of the variable to be measured before the first moment, specifically: the fourth determination module is used to: obtain the first historical data of the variable to be measured before the first moment; according to a preset quantile rule, respectively obtain the high quantile data and the low quantile data included in the first historical data; respectively determine the high quantile data and the low quantile data as the second upper threshold and the second lower threshold.

[0027] In one possible implementation, the fourth determination module is used to determine the second upper limit threshold and the second lower limit threshold based on the first historical data of the variable to be measured before the first moment, specifically: the fourth determination module is used to: obtain the first historical data of the variable to be measured before the first moment including the historical maximum data and the historical minimum data; determine the second upper limit threshold based on the historical maximum data; determine the second lower limit threshold based on the historical minimum data.

[0028] In one possible implementation, the anomaly detection module is used to perform anomaly detection on the real data of the variable to be measured at the first moment based on a first detection rule, specifically: the anomaly detection module is used to: determine the size relationship between the real data and the detection upper limit threshold and the detection lower limit threshold; if the real data is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the result of the anomaly detection is normal; or, if the real data is less than the detection lower limit threshold, or, greater than the detection upper limit threshold, then the result of the anomaly detection is abnormal.

[0029] In a possible implementation, the device further includes: a training module, configured to train and generate the autoregressive prediction model based on second historical data of the variable to be measured before the first moment.

[0030] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the data anomaly detection and correction method of the first aspect mentioned above is implemented.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the data anomaly detection and correction method of the first aspect described above is implemented.

[0032] In summary, in order to solve the problem of how to correct abnormal data in a timely manner during data anomaly detection and correction, an embodiment of the present application provides a data anomaly detection and correction method and device. In this method, the data of the measured variable at the first moment can be predicted based on the autoregressive prediction model to obtain the predicted data of the measured variable at the first moment. In addition, based on the first detection rule, anomaly detection can be performed on the real data of the measured variable at the first moment. When it is detected that the real data of the measured variable at the first moment is abnormal data, the real data can be corrected to the predicted data in a timely manner. Subsequently, data processing can be performed in a timely manner according to the predicted data, and the efficiency of subsequent data processing is higher and the accuracy is also higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A flowchart of a data anomaly detection and correction method provided in an embodiment of the present application;

[0035] Figure 2 A flowchart of another data anomaly detection and correction method provided in an embodiment of the present application;

[0036] Figure 3 A structural block diagram of a data anomaly detection and correction device provided in an embodiment of the present application;

[0037] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0039] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0040] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0041] To facilitate understanding, the application scenarios of the technical solutions provided in this application are first described below.

[0042] With the development of digital energy technology, the quality of energy data is becoming more and more important. Anomaly detection and correction of energy data has become the focus of the intersection of energy, artificial intelligence and big data.

[0043] Among them, most energy data have the following characteristics:

[0044] 1. The original value of energy data increases monotonically.

[0045] For example, energy data may be water consumption data or electricity consumption data collected by sensors connected to water meters or electricity meters. The original values ​​of water consumption data and electricity consumption data usually show a monotonically increasing trend over time.

[0046] 2. The difference in energy data has upper and lower limits.

[0047] For example, for power consumption data, assuming that the original value detected at time t1 (or energy consumption value) is f(t1), and the original value detected at time t1-1 is f(t1-1), then the total power consumption at time t1 (or the unit energy consumption value at time t1) V(t1) = f(t1)-f(t1-1), with upper and lower limits. The upper and lower limits of V(t1) can be expressed in the numerical interval [0, V 额定 ] means that the value range of V(t1) is [0, V 额定 ].

[0048] 3. The difference in energy data has seasonal periodicity.

[0049] For example, in actual applications, the difference in electricity consumption data is strongly correlated with time. If we use days, weeks, months, quarters, years, etc. as the time granularity and observe the trend of the difference in electricity consumption data, we can find that the trend of the difference in electricity consumption data is periodic.

[0050] 4. The difference in energy data has characteristics of special events and holidays.

[0051] For example, the difference in electricity consumption data will increase significantly during holidays or special events. In other words, holidays or special events will have a significant impact on the difference in electricity consumption data.

[0052] 5. The difference of energy data has the characteristics of relatively stable mean and variance.

[0053] For example, the difference in electricity consumption data has the characteristics of stable mean and variance within the global or local time span.

[0054] In addition, abnormal situations of energy data usually include the following:

[0055] 1. Illegal value

[0056] Illegal values ​​refer to missing data such as null values ​​and broken numbers that may appear during data collection or data transmission.

[0057] 2. Value jump

[0058] Value jump refers to abnormal changes in energy data or the difference in energy data, exceeding the upper and lower limits, and experiencing a sudden rise or fall, or it can also be called a positive jump or a negative jump. For example, the difference in power consumption data exceeds the maximum rated power consumption.

[0059] 3. Abnormal zero value

[0060] Abnormal zero value refers to the situation where the detected energy data is zero when the hardware device stops abnormally. For example, when the hardware device used to obtain the meter data stops abnormally, the detected power consumption data is zero. In this case, the zero value is an abnormal zero value.

[0061] 4. Abnormal amplitude

[0062] Amplitude anomaly refers to the situation where energy data or its difference may be abnormal data even if it is within the upper and lower limits.

[0063] At present, in most data anomaly detection and correction methods, data anomaly detection and abnormal data correction are divided into two independent processing processes. Among them, the correction link is a passive repair, which needs to rely on the latest data. There is a certain delay in the correction process, and the abnormal data cannot be corrected in time.

[0064] Combining the above characteristics of energy data and several abnormal situations that may occur in energy data, it can be seen that if the abnormal data cannot be corrected in time, the subsequently acquired energy data and / or the difference in energy data are very likely to be abnormal, which may affect the implementation of other links in the data link and reduce the efficiency of subsequent data processing.

[0065] Based on this, in order to be able to correct abnormal data more promptly, the present application provides a data anomaly detection and correction method. In the method provided by the present application, the data of the measured variable at the first moment can be predicted based on the autoregressive prediction model to obtain the predicted data of the measured variable at the first moment. In addition, based on the first detection rule, the real data of the measured variable at the first moment can be detected for anomaly. When it is detected that the real data of the measured variable at the first moment is abnormal data, the real data can be corrected to the predicted data in time. Data processing can be performed in time according to the predicted data later, and the efficiency of subsequent data processing is higher and the accuracy is also higher.

[0066] The following describes an embodiment of the data anomaly detection and correction method provided by the present application in conjunction with the accompanying drawings.

[0067] See also Figure 1 , Figure 1 A flowchart of a method for detecting and correcting data anomalies provided in an embodiment of the present application. The method can be applied to terminal devices, servers, or other devices with data processing functions (such as control devices installed in buildings, control devices installed in power grid management platforms, control devices installed in the Internet of Things, etc.), and the present application does not limit this. The following uses terminal devices as an example to introduce the embodiments of the present application. Figure 1 As shown, the method may include the following steps:

[0068] Step S101: Based on the autoregressive prediction model, obtain the prediction data of the variable to be measured at the first moment.

[0069] The variable to be measured may be any variable to be subjected to data anomaly detection and correction. Exemplarily, the variable to be measured may be an energy data variable. For example, the energy data variable may include power consumption, water consumption, gas consumption (e.g., coal gas consumption or natural gas consumption), etc. Exemplarily, the variable to be measured may also be an operation data variable. For example, the operation data variable may include operating power, opening frequency, instantaneous temperature or pressure, etc.

[0070] It should be noted that, as time goes by, multiple specific values ​​of the measured variable can be obtained. And, for each specific value of the measured variable obtained, the data anomaly detection and correction method provided by the present application can be used to perform data anomaly detection and correction. In other words, using the data anomaly detection and correction method provided by the present application, data anomaly detection and correction can be performed for any specific value of the measured variable. In addition, the specific value of the measured variable is referred to as the real data of the measured variable in the subsequent application. Based on this, since the method for performing data anomaly detection and correction for any real data of the measured variable is the same, in the embodiment of the present application, taking the data anomaly detection and correction of the real data of the measured variable at the first moment as an example, the embodiment of the data anomaly detection and correction method provided by the present application is described.

[0071] It can be seen from this that the first moment can be the moment when any real data of the variable to be measured is generated or produced, that is, the moment when any real data of the variable to be measured is generated or produced can be used as the first moment.

[0072] In a possible implementation, before executing step S101, an autoregressive prediction model may be trained and generated based on historical data of the variable to be measured before the first moment (hereinafter referred to as second historical data) according to the needs of the actual application scenario. The second historical data may be all historical data of the variable to be measured before the first moment, or may be part of the historical data of the variable to be measured before the first moment, and this application does not limit this.

[0073] Optionally, in order to reduce the amount of calculation and increase the accuracy of the model, part of the historical data of the variable to be measured before the first moment is usually selected as the second historical data. The amount of the second historical data can be set according to the requirements of the actual application scenario.

[0074] In a possible implementation, training and generating an autoregressive prediction model based on the second historical data can be implemented in the following manner:

[0075] The first step is to perform data preprocessing on the second historical data to obtain preprocessed data.

[0076] Optionally, data preprocessing of the second historical data may include at least one of the following: filtering out noise data such as null values ​​in the second historical data; correcting noise data such as null values ​​in the second historical data; filtering the second historical data according to a preset time span and a preset filtering interval, wherein the preset time span and the preset filtering interval may be set according to the needs of the actual application scenario. For example, the preset time span may be three years, two years, one year, six months, etc. The preset filtering interval may be one day, one hour, 30 minutes, 10 minutes, etc. Exemplarily, one second historical data may be filtered out every 30 minutes in the second historical data in the year before the first moment. Filtering may also be performed according to other filtering methods, which are not listed here one by one.

[0077] In the present application, data obtained after data preprocessing is performed on the second historical data is recorded as preprocessed data.

[0078] The second step is to establish the autoregressive prediction model to be trained.

[0079] The autoregressive prediction model to be trained can be established according to the needs of the actual application scenario. Optionally, the autoregressive prediction model to be trained can be a Neural Prophet algorithm model, which is an upgraded algorithm model based on neural networks and Prophet (prophet algorithm). Optionally, the autoregressive prediction model to be trained can also be other similar algorithm models of the Neural Prophet algorithm model. Optionally, the autoregressive prediction model to be trained can also be an algorithm model obtained by merging the Neural Prophet algorithm model with other similar algorithm models.

[0080] Exemplarily, when the variable to be measured is an energy data variable, for example, when the variable to be measured is an energy data variable such as electricity consumption, water consumption, gas consumption, etc., the following autoregressive prediction model to be trained can be established:

[0081] y^(t)=T(t)+S(t)+E(t)+F(t)+A(t)+L();

[0082] Among them, y^(t) represents the predicted data of the variable to be measured at time t; T(t) represents the time trend component at time t, which is used to indicate the impact of the time change trend on the predicted data of the variable to be measured at time t; S(t) represents the seasonal component at time t, which is used to indicate the impact of seasonal changes on the predicted data of the variable to be measured at time t; E(t) represents the event and holiday component at time t, which is used to indicate the impact of special events and / or holiday characteristics on the predicted data of the variable to be measured at time t; F(t) represents the regression component of future known exogenous variables at time t, which is used to indicate the impact of known exogenous variables after time t (such as weather variables after time t) on the predicted data at time t; A(t) represents the autoregressive component based on past observations at time t, which is used to indicate the impact of real data before time t on the predicted data at time t; L(t) represents the regression component of the lagged observation of the exogenous variable at time t, which is used to indicate the impact of the exogenous variable before time t on the predicted data at time t.

[0083] It should be noted that, in the autoregressive prediction model to be trained, one or more other components may be added on the basis of the above model according to the needs of the actual application scenario. Alternatively, one or more components may be deleted on the basis of the above model according to the needs of the actual application scenario. This application does not limit this.

[0084] The third step is to use the preprocessed data to train the autoregressive prediction model to be trained to generate an autoregressive prediction model.

[0085] In one possible implementation, the autoregressive prediction model is trained and generated based on the second historical data. It can also be implemented in the following manner: the autoregressive prediction model that has been trained and generated before (hereinafter referred to as the initial autoregressive prediction model) is updated using the second historical data or the preprocessed data obtained after data preprocessing the second historical data to obtain the autoregressive prediction model required to execute step S101.

[0086] After the terminal device trains and generates the autoregressive prediction model based on the second historical data, the first moment can be used as the value of t and input into the autoregressive prediction model to obtain the output data of the autoregressive prediction model. The output data of the autoregressive prediction model may include a prediction upper limit, a prediction lower limit, and a prediction value, wherein the prediction value is greater than or equal to the prediction lower limit and less than or equal to the prediction upper limit. The terminal device may determine the prediction value included in the output data as the prediction data of the variable to be measured at the first moment, and determine the prediction upper limit and prediction lower limit included in the output data as the prediction upper limit value and prediction lower limit value of the variable to be measured at the first moment.

[0087] Step S102: Based on a first detection rule, perform anomaly detection on real data of the variable to be measured at the first moment.

[0088] The real data of the variable to be measured at the first moment refers to the actual detection data obtained by detecting the variable to be measured at the first moment through a detection device for the variable to be measured (such as an electric meter, a water meter, etc.).

[0089] Optionally, the terminal device may obtain the real data of the variable to be measured at the first moment by reading the actual detection data in the variable to be measured detection device in real time. Optionally, the terminal device may also first read the actual detection data of the variable to be measured at each detection moment from the variable to be measured detection device, and then store the read actual detection data together with the detection time in the terminal device accordingly. When executing step S102, the actual detection data at the first moment is directly read from the terminal device, and the actual detection data is determined as the real data of the variable to be measured at the first moment.

[0090] The following describes a process of obtaining real data of the variable to be measured at the first moment by taking a specific application scenario as an example.

[0091] The first application scenario may be an application scenario of a building equipment management system or an energy management system of a target building (or a building to be managed). The variable to be measured may be an energy data variable, which may include electricity consumption, water consumption, or gas consumption.

[0092] In this application scenario, obtaining the real data of the variable to be measured at the first moment can be achieved in the following way:

[0093] The first step is to obtain the value of the energy data variable of the first load device in the target building at the first moment through a sensing device (or a device for detecting the variable to be measured).

[0094] The sensing device may be an electric meter, the first load device may include a lighting device and / or an elevator device, and the energy data variable may be electricity consumption; or,

[0095] The sensing device may be a water meter, the first load device may include a water-using device, and the energy data variable may be water consumption; or,

[0096] The sensing device may be a coal gas meter or a gas meter, the first load device may include a gas-consuming device, and the energy data variable may be gas consumption.

[0097] The second step is to determine that the value of the energy data variable at the first moment is the real data of the variable to be measured at the first moment.

[0098] The second application scenario may be an application scenario of a power grid management platform, and the variable to be measured may be operating power.

[0099] In this application scenario, obtaining the real data of the variable to be measured at the first moment can be achieved in the following way:

[0100] The first step is to obtain the value of the operating power of the second load device in the power grid management platform at the first moment.

[0101] The second load device may include a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device, etc.

[0102] The second step is to determine that the value of the operating power at the first moment is the real data of the variable to be measured at the first moment.

[0103] The third application scenario may be an Internet of Things application scenario. The variable to be measured may be an operation data variable, which may include a start frequency, an instantaneous temperature or a pressure, and the like.

[0104] In this application scenario, obtaining the real data of the variable to be measured at the first moment can be achieved in the following way:

[0105] The first step is to obtain the value of the operating data variable of the IoT device at the first moment.

[0106] The second step is to determine that the value of the operating data variable at the first moment is the real data of the variable to be measured at the first moment.

[0107] In a possible implementation, before performing anomaly detection on real data of the variable to be measured at the first moment based on the first detection rule, the method may further include: acquiring the first detection rule.

[0108] Obtaining the first detection rule can be achieved in many ways, for example:

[0109] In a first implementation, obtaining the first detection rule may be implemented in the following manner: obtaining the first detection rule from a preset rule table. The preset rule table may include one or more preset detection rules. Optionally, the preset detection rules in the preset rule table may be set based on experience.

[0110] Optionally, the terminal device may select any preset detection rule from the preset rule table as the first detection rule.

[0111] Optionally, the terminal device may also select multiple preset detection rules from the preset rule table, and then use the fusion result of the multiple preset detection rules as the first detection rule. For example, the weighted average result of the multiple preset detection rules is used as the first detection rule. It is understandable that other fusion results of the multiple preset detection rules may also be used as the first detection rule, which are not listed here one by one.

[0112] In the second implementation, obtaining the first detection rule can also be implemented in the following manner:

[0113] Step 1: Determine a first upper threshold and a first lower threshold based on the predicted data of the variable to be measured at the first moment.

[0114] Optionally, the first upper threshold value can be expressed by the following relationship: M1=yhat+S uo , calculated. Among them, M1 is the first upper threshold; yhat is the predicted data of the variable to be measured at the first moment; S up For the prediction upper bound, it can be set according to the requirements of the actual application scenario, for example, it can be set to 5%, 10% or 15% of the predicted data.

[0115] Optionally, the first lower threshold can be expressed by the following relation: N1=yhat-S low , calculated. Where N1 is the first lower limit threshold; yhat is the predicted data of the variable to be measured at the first moment; S low To predict the lower bound, it can be set according to the needs of the actual application scenario, for example, it can be set to 5%, 10% or 15% of the predicted data.

[0116] It should be noted that the prediction upper bound and the prediction lower bound can also be set to other values. For example, the prediction upper bound and the prediction lower bound can be set to the upper and lower bounds of reference factors such as time trend component, seasonal component, event and holiday component, etc. For another example, the prediction upper bound and the prediction lower bound can also be set to 3σ. In addition, the prediction upper bound and the prediction lower bound can be the same or different. This application does not limit this.

[0117] Step 2: respectively determine the first upper limit threshold and the first lower limit threshold as the detection upper limit threshold and the detection lower limit threshold of the first detection rule.

[0118] After the terminal device determines the first upper threshold and the first lower threshold, the first upper threshold may be determined as the detection upper threshold of the first detection rule, and the first lower threshold may be determined as the detection lower threshold of the first detection rule.

[0119] Step 3: Determine a first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0120] After the terminal device determines the detection upper limit threshold and the detection lower limit threshold of the first detection rule, the first detection rule can be determined. Optionally, the first detection rule can be: if the data to be detected is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be normal; or, if the data to be detected is less than the detection lower limit threshold, or the data to be detected is greater than the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be abnormal.

[0121] In the third implementation, obtaining the first detection rule may also be implemented in the following manner:

[0122] Step 1: Determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment.

[0123] The first historical data of the variable to be measured before the first moment may be all the historical data of the variable to be measured before the first moment, or may be part of the historical data of the variable to be measured before the first moment, and the present application does not impose any limitation on this.

[0124] Optionally, in order to reduce the amount of calculation and increase the accuracy of the rule, part of the historical data of the variable to be measured before the first moment is usually selected as the first historical data. The amount of the first historical data can be set according to the requirements of the actual application scenario.

[0125] It should be noted that the second historical data may be the same as the first historical data or may be different from the first historical data, and this application does not impose any limitation on this.

[0126] In one possible implementation, based on the first historical data of the variable to be measured before the first moment, the second upper limit threshold and the second lower limit threshold are determined, which can be implemented in the following manner: obtain the first historical data of the variable to be measured before the first moment; according to a preset quantile rule, respectively obtain the high quantile data and the low quantile data included in the first historical data; respectively determine the high quantile data and the low quantile data as the second upper limit threshold and the second lower limit threshold.

[0127] The preset quantile rule can be set according to the needs of the actual application scenario. For example, the preset quantile rule can be that the high quantile data (or high quantile threshold) is the value of the 98% quantile, and the low quantile data (or low quantile threshold) is the value of the 5% quantile. Then, the high quantile data included in the first historical data can be the value at the 98% quantile in the first historical data, and the low quantile data included in the first historical data can be the value at the 5% quantile in the first historical data.

[0128] After the terminal device obtains the high percentile data and the low percentile data included in the first historical data, the high percentile data can be determined as the second upper limit threshold and the low percentile data can be determined as the second lower limit threshold.

[0129] In one possible implementation, based on the first historical data of the variable to be measured before the first moment, the second upper threshold and the second lower threshold are determined. It can also be implemented in the following manner: obtain the historical maximum data and the historical minimum data included in the first historical data of the variable to be measured before the first moment; determine the second upper threshold based on the historical maximum data; determine the second lower threshold based on the historical minimum data.

[0130] The historical maximum data included in the first historical data of the measured variable before the first moment refers to the maximum value in the first historical data, and the historical minimum data included in the first historical data of the measured variable before the first moment refers to the minimum value in the first historical data.

[0131] The second upper limit threshold is determined based on the historical maximum data, and can be determined according to the following relationship M2=X max -′ up , calculated. Where M2 is the second upper limit threshold; X max is the maximum historical data; S′ up It is a lower floating value and can be set according to the needs of the actual application scenario. For example, the lower floating value can be 5%, 4%, 2% and so on of the historical maximum data.

[0132] The second lower limit threshold is determined based on the historical minimum data, and can be determined according to the following relationship N2=X min +′ low , calculated. Where N2 is the second lower threshold; X min is the historical minimum data; S′ low It is an upward value and can be set according to the needs of the actual application scenario. For example, the upward value can be 1% of the historical minimum data.

[0133] Step 2: respectively determine the second upper threshold and the second lower threshold as the detection upper threshold and the detection lower threshold of the first detection rule.

[0134] After the terminal device determines the second upper threshold and the second lower threshold, the second upper threshold can be determined as the detection upper threshold of the first detection rule, and the second lower threshold can be determined as the detection lower threshold of the first detection rule.

[0135] Step three: determine a first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0136] After the terminal device determines the detection upper limit threshold and the detection lower limit threshold of the first detection rule, the first detection rule can be determined. Optionally, the first detection rule can be: if the data to be detected is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be normal; or, if the data to be detected is less than the detection lower limit threshold, or the data to be detected is greater than the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be abnormal.

[0137] In the fourth implementation, obtaining the first detection rule may also be implemented in the following manner:

[0138] Step 1: Determine a first upper threshold and a first lower threshold based on the predicted data of the variable to be measured at the first moment.

[0139] The specific implementation method of step one can refer to the implementation method of step one in the second implementation method, which will not be repeated here.

[0140] Step 2: Determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment.

[0141] The specific implementation method of step 2 can refer to the implementation method of step 1 in the third implementation method, which will not be repeated here.

[0142] Step 3: Determine a third upper threshold and a third lower threshold.

[0143] The third upper limit threshold is a value obtained by fusing the first upper limit threshold with the second upper limit threshold. Optionally, the third upper limit threshold may be a value obtained by fusing the first upper limit threshold with the second upper limit threshold obtained in any one of the third implementations. Optionally, the third upper limit threshold may also be a value obtained by fusing the first upper limit threshold with two second upper limit thresholds obtained in two of the third implementations.

[0144] Similarly, the third lower threshold is a value obtained by merging the first lower threshold with the second lower threshold. Optionally, the third lower threshold can be a value obtained by merging the first lower threshold with the second lower threshold obtained in any one of the third implementations. Optionally, the third lower threshold can also be a value obtained by merging the first lower threshold with two second lower thresholds obtained in two of the third implementations.

[0145] Step 4: respectively determine the third upper threshold and the third lower threshold as the detection upper threshold and the detection lower threshold of the first detection rule.

[0146] After the terminal device determines the third upper threshold and the third lower threshold, the third upper threshold can be determined as the detection upper threshold of the first detection rule, and the third lower threshold can be determined as the detection lower threshold of the first detection rule.

[0147] Step 5: Determine a first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0148] After the terminal device determines the detection upper limit threshold and the detection lower limit threshold of the first detection rule, the first detection rule can be determined. Optionally, the first detection rule can be: if the data to be detected is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be normal; or, if the data to be detected is less than the detection lower limit threshold, or the data to be detected is greater than the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be abnormal.

[0149] In the fifth implementation, obtaining the first detection rule may also be implemented in the following manner:

[0150] Step 1: respectively determine the predicted upper limit value and the predicted lower limit value of the variable to be measured at the first moment as the detection upper limit threshold and the detection lower limit threshold of the first detection rule.

[0151] That is to say, after obtaining the predicted upper limit value and predicted lower limit value of the variable to be measured at the first moment through the autoregressive prediction model, the predicted upper limit value can be determined as the detection upper limit threshold of the first detection rule, and the predicted lower limit value can be determined as the detection lower limit threshold of the first detection rule.

[0152] Step 2: Determine a first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0153] After the terminal device determines the detection upper limit threshold and the detection lower limit threshold of the first detection rule, the first detection rule can be determined. Optionally, the first detection rule can be: if the data to be detected is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be normal; or, if the data to be detected is less than the detection lower limit threshold, or the data to be detected is greater than the detection upper limit threshold, then the abnormal detection result of the data to be detected is determined to be abnormal.

[0154] It should be noted that the detection upper limit threshold of the first detection rule can also be obtained by integrating the first upper limit threshold, the second upper limit threshold, the predicted upper limit value and the preset upper limit threshold in the preset detection rule. Similarly, the detection lower limit threshold of the first detection rule can also be obtained by integrating the first lower limit threshold, the second lower limit threshold, the predicted lower limit value and the preset lower limit threshold in the preset detection rule. This application does not limit this.

[0155] In one possible implementation, after the terminal device obtains the first detection rule, based on the first detection rule, anomaly detection is performed on the real data of the measured variable at the first moment, which can be implemented in the following manner: determine the size relationship between the real data and the detection upper limit threshold and the detection lower limit threshold; if the real data is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, the result of the anomaly detection is normal; or, if the real data is less than the detection lower limit threshold, or greater than the detection upper limit threshold, the result of the anomaly detection is abnormal.

[0156] That is to say, if the real data of the variable to be measured at the first moment is greater than or equal to the detection lower limit threshold of the first detection rule, and less than or equal to the detection upper limit threshold of the first detection rule, then the real data of the variable to be measured at the first moment is determined to be normal data. Alternatively, if the real data of the variable to be measured at the first moment is less than the detection lower limit threshold of the first detection rule, or the real data of the variable to be measured at the first moment is greater than the detection upper limit threshold of the first detection rule, then the real data of the variable to be measured at the first moment is determined to be abnormal data.

[0157] Step S103: If the result of the abnormality detection is abnormal, the real data is corrected to the predicted data.

[0158] In one possible implementation, if the result of anomaly detection is abnormal, that is, the real data of the variable to be measured at the first moment is abnormal data, then the real data of the variable to be measured at the first moment is corrected to the predicted data of the variable to be measured at the first moment, that is, the predicted data of the variable to be measured at the first moment is used as the real data of the variable to be measured at the first moment, and then, the predicted data of the variable to be measured at the first moment can be stored as the real data of the variable to be measured at the first moment.

[0159] In a possible implementation, if the result of the abnormality detection is normal, that is, the real data of the variable to be measured at the first moment is normal data, the real data of the variable to be measured at the first moment can be stored.

[0160] In the data anomaly detection and correction method provided by the present application, the data of the measured variable at the first moment can be predicted based on the autoregressive prediction model to obtain the predicted data of the measured variable at the first moment. In addition, the real data of the measured variable at the first moment can be detected for anomaly based on the first detection rule. When it is detected that the real data of the measured variable at the first moment is abnormal data, the real data can be corrected to the predicted data in a timely manner. Data processing can be performed in a timely manner according to the predicted data, and the efficiency of subsequent data processing is higher and the accuracy is also higher.

[0161] See also Figure 2 , Figure 2 A flowchart of another data anomaly detection and correction method provided in an embodiment of the present application. Figure 2 As shown, the method can be implemented according to the following steps:

[0162] The first step is regular scheduling.

[0163] In the first step, the terminal device can periodically retrieve the historical data of the variables to be measured according to a preset model update cycle in the batch task process.

[0164] The second step is tag set and data acquisition.

[0165] In the second step, the terminal device can label the acquired historical data as normal or abnormal. Then the historical data labeled as normal or abnormal is divided into a training set and a test set. After that, the training set is used to train the autoregressive prediction model to be trained to obtain a trained autoregressive prediction model; or, the trained autoregressive prediction model is updated (or optimized) to obtain an updated autoregressive prediction model.

[0166] The third step is to make numerical predictions for future periods.

[0167] In the third step, the terminal device can use the trained or updated autoregressive prediction model to predict the value of the variable to be measured at each detection moment, and obtain the predicted data (or predicted value) of the variable to be measured at each detection moment (such as the first moment).

[0168] Step 4: Prediction value storage.

[0169] In the fourth step, after the terminal device predicts the predicted data of the variable to be measured at each detection moment, the predicted data can be stored in correspondence with the detection moment.

[0170] Step 5: Generate rules.

[0171] In the fifth step, the terminal device can generate a detection rule corresponding to the detection moment based on the prediction upper limit and prediction lower limit of each detection moment output by the autoregressive prediction model. Alternatively, the terminal device can also generate a detection rule corresponding to the detection moment based on the prediction data of the variable to be measured at each detection moment.

[0172] Step 6: Get the rules.

[0173] In the sixth step, the terminal device can obtain the prediction upper limit and prediction lower limit output by the autoregressive prediction model corresponding to each detection moment in the process of the flow task, and then generate the detection rule used at the detection moment based on the prediction upper limit and prediction lower limit. Alternatively, the terminal device can also obtain the detection rule generated based on the prediction data corresponding to each detection moment, and use the detection rule as the detection rule used at the detection moment (for example, the first detection rule). Alternatively, the terminal device can also obtain the preset detection rule in the preset rule table, and use the preset detection rule as the detection rule used at each detection moment. Alternatively, the terminal device can also generate the detection rule used at the detection moment based on the detection rule generated based on the prediction data corresponding to each detection moment and the preset detection rule in the preset rule table.

[0174] Step 7: Anomaly detection.

[0175] In the seventh step, the terminal device may receive the real data of the variable to be measured at each detection moment from the entrance, and then perform anomaly detection on the real data of the variable to be measured at the detection moment based on the detection rule used at the detection moment.

[0176] Step 8. Repair the abnormality.

[0177] If the abnormal detection result of the real data of the measured variable at any detection time is abnormal, the terminal device can correct the real data of the measured variable at the detection time to the predicted value of the measured variable at the detection time, and then store the predicted value at the detection time as the real data.

[0178] Step 9: Data storage.

[0179] In the ninth step, if the abnormality detection result of the real data of the measured variable at any detection time is normal, the terminal device can store the real data of the measured variable at the detection time. Afterwards, the stored real data can be output from the export to other data processing links.

[0180] In the data anomaly detection and correction method provided by the present application, the numerical value of the measured variable at any detection time can be predicted based on the autoregressive prediction model to obtain the predicted data of the measured variable at the detection time. In addition, the real data of the measured variable at the detection time can be detected based on the detection rules used at any detection time. When it is detected that the real data of the measured variable at any detection time is abnormal data, the real data of the measured variable at the detection time can be corrected to the predicted data of the measured variable at the detection time in a timely manner. Data processing can be performed in a timely manner according to the predicted data, and the efficiency of subsequent data processing is higher and the accuracy is also higher.

[0181] It can be understood that the above embodiments are only examples and can be modified in actual implementation. Those skilled in the art can understand that the modification methods of the above embodiments without creative labor all fall within the protection scope of the present application and will not be described in detail in the embodiments.

[0182] Based on the same inventive concept, the embodiment of the present application also provides a data anomaly detection and correction device. Since the principle of solving the problem by the data anomaly detection and correction device is similar to that of the aforementioned data anomaly detection and correction method, the implementation of the data anomaly detection and correction device can refer to the implementation of the aforementioned data anomaly detection and correction method, and the repeated parts will not be repeated.

[0183] See also Figure 3 , Figure 3 This is a structural block diagram of a data anomaly detection and correction device provided in an embodiment of the present application. Figure 3 As shown, the data anomaly detection and correction device 300 may include: a first acquisition module 301, an anomaly detection module 302 and a correction module 303.

[0184] The first acquisition module 301 may be used to acquire prediction data of the variable to be measured at the first moment based on an autoregressive prediction model.

[0185] The anomaly detection module 302 may be configured to perform anomaly detection on real data of the variable to be measured at the first moment based on a first detection rule.

[0186] The correction module 303 may be used to correct the real data to the predicted data if the result of the abnormality detection is abnormal.

[0187] In a possible implementation, the variable to be measured is an energy data variable, and the energy data variable includes electricity consumption, water consumption or gas consumption; the data anomaly detection and correction device 300 may also include: a second acquisition module, used to obtain the value of the energy data variable of the first load device in the target building at the first moment through a sensing device; the sensing device is an electric meter, the first load device includes lighting equipment and / or elevator equipment, and the energy data variable is electricity consumption; or, the sensing device is a water meter, the first load device includes water consumption equipment, and the energy data variable is water consumption; or, the sensing device is a gas meter or a gas meter, the first load device includes gas consumption equipment, and the energy data variable is gas consumption; a first determination module, used to determine that the value of the energy data variable at the first moment is the real data.

[0188] In one possible implementation, the variable to be measured is operating power; the data anomaly detection and correction device 300 may also include: a third acquisition module, used to obtain the value of the operating power of a second load device in the power grid management platform at the first moment; the second load device includes a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device; a second determination module, used to determine that the value of the operating power at the first moment is the real data.

[0189] In one possible implementation, the variable to be measured is an operating data variable, and the operating data variable includes a start-up frequency, an instantaneous temperature or a pressure; the data anomaly detection and correction device 300 may also include: a fourth acquisition module, used to obtain the value of the operating data variable of the Internet of Things device at the first moment; a third determination module, used to determine that the value of the operating data variable at the first moment is the real data.

[0190] In a possible implementation, the data anomaly detection and correction device 300 may also include: a fourth determination module, used to determine a detection upper limit threshold and a detection lower limit threshold before performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule; a fifth determination module, used to determine the first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

[0191] In one possible implementation, the first acquisition module 301 is used to obtain the predicted data of the variable to be measured at the first moment based on the autoregressive prediction model, specifically: the first acquisition module 301 is used to obtain the predicted data, predicted upper limit value and predicted lower limit value of the variable to be measured at the first moment based on the autoregressive prediction model; the predicted data is greater than or equal to the predicted lower limit value, and less than or equal to the predicted upper limit value; the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to respectively determine the prediction upper limit value and the prediction lower limit value as the detection upper limit threshold and the detection lower limit threshold.

[0192] In one possible implementation, the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to: determine a first upper limit threshold and a first lower limit threshold based on the predicted data; and determine the first upper limit threshold and the first lower limit threshold as the detection upper limit threshold and the detection lower limit threshold, respectively.

[0193] In one possible implementation, the fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: the fourth determination module is used to: determine a first upper limit threshold and a first lower limit threshold based on the predicted data; determine a second upper limit threshold and a second lower limit threshold based on the first historical data of the variable to be measured before the first moment; determine a third upper limit threshold and a third lower limit threshold, the third upper limit threshold being a value obtained by merging the first upper limit threshold and the second upper limit threshold, and the third lower limit threshold being a value obtained by merging the first lower limit threshold and the second lower limit threshold; respectively determine the third upper limit threshold and the third lower limit threshold as the detection upper limit threshold and the detection lower limit threshold.

[0194] In one possible implementation, the fourth determination module is used to determine the second upper threshold and the second lower threshold based on the first historical data of the variable to be measured before the first moment, specifically: the fourth determination module is used to: obtain the first historical data of the variable to be measured before the first moment; according to a preset quantile rule, respectively obtain the high quantile data and the low quantile data included in the first historical data; respectively determine the high quantile data and the low quantile data as the second upper threshold and the second lower threshold.

[0195] In one possible implementation, the fourth determination module is used to determine the second upper limit threshold and the second lower limit threshold based on the first historical data of the variable to be measured before the first moment, specifically: the fourth determination module is used to: obtain the first historical data of the variable to be measured before the first moment including the historical maximum data and the historical minimum data; determine the second upper limit threshold based on the historical maximum data; determine the second lower limit threshold based on the historical minimum data.

[0196] In one possible implementation, the anomaly detection module is used to perform anomaly detection on the real data of the variable to be measured at the first moment based on a first detection rule, specifically: the anomaly detection module is used to: determine the size relationship between the real data and the detection upper limit threshold and the detection lower limit threshold; if the real data is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, then the result of the anomaly detection is normal; or, if the real data is less than the detection lower limit threshold, or, greater than the detection upper limit threshold, then the result of the anomaly detection is abnormal.

[0197] In a possible implementation, the device further includes: a training module, configured to train and generate the autoregressive prediction model based on second historical data of the variable to be measured before the first moment.

[0198] See also Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 may include a processor 401 and a memory 402; the memory 402 may be coupled to the processor 401. It is worth noting that the Figure 4 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0199] In a possible implementation, the functions of the data anomaly detection and correction device 300 may be integrated into the processor 401. The processor 401 may be configured to perform the following operations:

[0200] Based on the autoregressive prediction model, the prediction data of the variable to be measured at the first moment is obtained;

[0201] Based on a first detection rule, performing anomaly detection on real data of the variable to be measured at the first moment;

[0202] If the result of the abnormality detection is abnormal, the real data is corrected to the predicted data.

[0203] In another possible implementation, the data anomaly detection and correction device 300 may be configured separately from the processor 401 . For example, the data anomaly detection and correction device 300 may be configured as a chip connected to the processor 401 , and data anomaly detection and correction may be implemented through the control of the processor 401 .

[0204] In addition, in some optional implementations, the electronic device 400 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 400 does not necessarily include Figure 4 In addition, the electronic device 400 may also include Figure 4 For components not shown, reference may be made to the prior art.

[0205] In some optional implementations, the processor 401 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 400.

[0206] The memory 402 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The memory 402 may store the above-mentioned information related to the data anomaly detection and correction device 300, and may also store a program for executing the related information. The processor 401 may execute the program stored in the memory 402 to implement information storage or processing.

[0207] The input unit can provide input to the processor 401. The input unit is, for example, a key or a touch input device. The power supply can be used to provide power to the electronic device 400. The display can be used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0208] The memory 402 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 402 may also be some other type of device. The memory 402 includes a buffer memory (sometimes referred to as a buffer). The memory 402 may include an application / function storage unit for storing application programs and function programs or processes for executing the operation of the electronic device 400 by the processor 401.

[0209] The memory 402 may also include a data storage unit for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit of the memory 402 may include various drivers for the communication function of the computer device and / or for executing other functions of the computer device (such as a messaging application, a contact book application, etc.).

[0210] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 401 to provide input signals and receive output signals, which may be the same as in a conventional mobile communication terminal.

[0211] Based on different communication technologies, multiple communication modules may be provided in the same computer device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to the processor 401, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0212] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data anomaly detection and correction method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the data anomaly detection and correction method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0213] Based on the autoregressive prediction model, the prediction data of the variable to be measured at the first moment is obtained;

[0214] Based on a first detection rule, performing anomaly detection on real data of the variable to be measured at the first moment;

[0215] If the result of the abnormality detection is abnormal, the real data is corrected to the predicted data.

[0216] In summary, the data anomaly detection and correction method, data anomaly detection and correction device, electronic device, and computer-readable storage medium provided in the embodiments of the present application all have the following advantages:

[0217] By the above-mentioned data anomaly detection and correction method, data anomaly detection and correction device, electronic device, and computer-readable storage medium provided by the embodiments of the present application, the data of the measured variable at the first moment can be predicted based on the autoregressive prediction model to obtain the predicted data of the measured variable at the first moment. In addition, based on the first detection rule, the real data of the measured variable at the first moment can be detected for anomaly. When it is detected that the real data of the measured variable at the first moment is abnormal data, the real data can be corrected to the predicted data in time. Data processing can be performed in time according to the predicted data later, and the efficiency of subsequent data processing is higher and the accuracy is also higher.

[0218] In addition, the above method and device provided by the present application can be generally used in various application scenarios of data anomaly detection and correction, and have better versatility.

[0219] The above method and device provided by the present application can automatically generate detection rules without manual setting based on experience, thus avoiding errors caused by misoperation and avoiding excessive requirements on prior knowledge of operators. In addition, in the above method and device provided by the present application, the detection rules can be dynamically updated based on historical data, which can make up for the inflexibility of manually setting rules, and can adapt to the latest dynamics of the latest data, and have better applicability.

[0220] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0221] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0222] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0223] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0225] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. The orientation or position relationship indicated by the terms "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. It should be noted that the embodiments and features in the embodiments in this application can be combined with each other unless there is a conflict. The present application is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or permutation of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of the present application can be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application.

[0227] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "install", "connect" and "connect" should be understood in a broad sense. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0228] The present application has been described above in conjunction with preferred embodiments, but these embodiments are only exemplary and serve only as an illustration. On this basis, various replacements and improvements may be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A method for detecting and correcting data anomalies, characterized in that: The method comprises: Based on the autoregressive prediction model, the prediction data of the variable to be measured at the first moment is obtained; Based on a first detection rule, performing anomaly detection on real data of the variable to be measured at the first moment; If the result of the abnormality detection is abnormal, the real data is corrected to the predicted data.

2. The method according to claim 1, characterized in that The variable to be measured is an energy data variable, and the energy data variable includes electricity consumption, water consumption or gas consumption; The method further comprises: The value of the energy data variable of the first load device in the target building at the first moment is obtained through a sensing device; the sensing device is an electric meter, the first load device includes a lighting device and / or an elevator device, and the energy data variable is the power consumption; or the sensing device is a water meter, the first load device includes a water-using device, and the energy data variable is the water consumption; or the sensing device is a gas meter or a gas meter, the first load device includes a gas-using device, and the energy data variable is the gas consumption; Determine the value of the energy data variable at the first moment as the real data.

3. The method according to claim 1, characterized in that The variable to be measured is operating power; The method further comprises: Obtaining a value of the operating power of a second load device in the power grid management platform at the first moment; the second load device includes a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device; Determine that the value of the operating power at the first moment is the real data.

4. The method according to claim 1, characterized in that The variable to be measured is an operation data variable, and the operation data variable includes an opening frequency, an instantaneous temperature or a pressure; The method further comprises: Obtaining the value of the operating data variable of the Internet of Things device at the first moment; Determine the value of the operating data variable at the first moment as the real data.

5. The method according to any one of claims 1 to 4, characterized in that Before performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule, the method further includes: Determine an upper detection threshold and a lower detection threshold; The first detection rule is determined based on the detection upper limit threshold and the detection lower limit threshold.

6. The method according to claim 5, characterized in that The step of obtaining the prediction data of the variable to be measured at the first moment based on the autoregressive prediction model includes: Based on the autoregressive prediction model, the prediction data, the prediction upper limit value and the prediction lower limit value of the variable to be measured at the first moment are obtained; the prediction data is greater than or equal to the prediction lower limit value, and less than or equal to the prediction upper limit value; The step of determining the upper detection limit threshold and the lower detection limit threshold comprises: The prediction upper limit value and the prediction lower limit value are respectively determined as the detection upper limit threshold and the detection lower limit threshold.

7. The method according to claim 5, characterized in that The step of determining the upper detection limit threshold and the lower detection limit threshold comprises: Based on the predicted data, determining a first upper threshold and a first lower threshold; The first upper threshold and the first lower threshold are determined as the detection upper threshold and the detection lower threshold, respectively.

8. The method according to claim 5, characterized in that The step of determining the upper detection limit threshold and the lower detection limit threshold comprises: Based on the predicted data, determining a first upper threshold and a first lower threshold; Determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment; Determine a third upper threshold and a third lower threshold, wherein the third upper threshold is a value obtained by merging the first upper threshold and the second upper threshold, and the third lower threshold is a value obtained by merging the first lower threshold and the second lower threshold; The third upper threshold and the third lower threshold are determined as the detection upper threshold and the detection lower threshold, respectively.

9. The method according to claim 8, characterized in that The determining of the second upper threshold and the second lower threshold based on the first historical data of the variable to be measured before the first moment includes: Acquire first historical data of the variable to be measured before the first moment; According to a preset quantile rule, respectively obtain high quantile data and low quantile data included in the first historical data; The high quantile data and the low quantile data are respectively determined as the second upper threshold and the second lower threshold.

10. The method according to claim 8, characterized in that The determining of the second upper threshold and the second lower threshold based on the first historical data of the variable to be measured before the first moment includes: Acquire first historical data of the variable to be measured before the first moment, including historical maximum data and historical minimum data; Determine the second upper limit threshold based on the historical maximum data; The second lower threshold is determined based on the historical minimum data.

11. The method according to claim 5, characterized in that The performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule includes: Determine the magnitude relationship between the real data and the upper detection threshold and the lower detection threshold; If the real data is greater than or equal to the detection lower limit threshold and less than or equal to the detection upper limit threshold, the result of the abnormality detection is normal; or, If the real data is smaller than the detection lower limit threshold, or greater than the detection upper limit threshold, the result of the abnormality detection is abnormal.

12. The method according to claim 1, characterized in that The method further comprises: The autoregressive prediction model is trained and generated based on second historical data of the variable to be measured before the first moment.

13. A data anomaly detection and correction device, characterized in that: The device comprises: A first acquisition module is used to acquire the prediction data of the variable to be measured at the first moment based on the autoregressive prediction model; an anomaly detection module, configured to perform anomaly detection on real data of the variable to be measured at the first moment based on a first detection rule; A correction module is used to correct the real data to the predicted data if the result of the abnormality detection is abnormal.

14. The device according to claim 13, characterized in that The variable to be measured is an energy data variable, and the energy data variable includes electricity consumption, water consumption or gas consumption; The device also includes: A second acquisition module is used to acquire the value of the energy data variable of the first load device in the target building at the first moment through a sensing device; the sensing device is an electric meter, the first load device includes a lighting device and / or an elevator device, and the energy data variable is the power consumption; or the sensing device is a water meter, the first load device includes a water consumption device, and the energy data variable is the water consumption; or the sensing device is a gas meter or a gas meter, the first load device includes a gas consumption device, and the energy data variable is the gas consumption; The first determining module is used to determine the value of the energy data variable at the first moment as the real data.

15. The device according to claim 13, characterized in that The variable to be measured is operating power; The device also includes: A third acquisition module is used to obtain the value of the operating power of the second load device in the power grid management platform at the first moment; the second load device includes a distributed power supply device, an energy storage device, a load, a monitoring device or a protection device; The second determining module is used to determine that the value of the operating power at the first moment is the real data.

16. The device according to claim 13, characterized in that The variable to be measured is an operation data variable, and the operation data variable includes an opening frequency, an instantaneous temperature or a pressure; The device also includes: A fourth acquisition module, used to obtain the value of the operation data variable of the Internet of Things device at the first moment; The third determining module is used to determine that the value of the operating data variable at the first moment is the real data.

17. The device according to any one of claims 13 to 16, characterized in that The device also includes: A fourth determination module, used to determine a detection upper limit threshold and a detection lower limit threshold before performing anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule; A fifth determination module is used to determine the first detection rule based on the detection upper limit threshold and the detection lower limit threshold.

18. The device according to claim 17, characterized in that The first acquisition module is used to acquire the prediction data of the variable to be measured at the first moment based on the autoregressive prediction model, specifically: The first acquisition module is used to acquire the prediction data, the prediction upper limit value and the prediction lower limit value of the variable to be measured at the first moment based on the autoregressive prediction model; the prediction data is greater than or equal to the prediction lower limit value and less than or equal to the prediction upper limit value; The fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: The fourth determination module is used to determine the prediction upper limit value and the prediction lower limit value as the detection upper limit threshold and the detection lower limit threshold respectively.

19. The device according to claim 17, characterized in that The fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: The fourth determination module is used for: Based on the predicted data, determining a first upper threshold and a first lower threshold; The first upper threshold and the first lower threshold are determined as the detection upper threshold and the detection lower threshold, respectively.

20. The device according to claim 17, characterized in that The fourth determination module is used to determine the detection upper limit threshold and the detection lower limit threshold, specifically: The fourth determination module is used for: Based on the predicted data, determining a first upper threshold and a first lower threshold; Determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment; Determine a third upper threshold and a third lower threshold, wherein the third upper threshold is a value obtained by merging the first upper threshold and the second upper threshold, and the third lower threshold is a value obtained by merging the first lower threshold and the second lower threshold; The third upper threshold and the third lower threshold are determined as the detection upper threshold and the detection lower threshold, respectively.

21. The device according to claim 20, characterized in that The fourth determination module is used to determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment, specifically: The fourth determination module is used for: Acquire first historical data of the variable to be measured before the first moment; According to a preset quantile rule, respectively obtain high quantile data and low quantile data included in the first historical data; The high quantile data and the low quantile data are respectively determined as the second upper threshold and the second lower threshold.

22. The device according to claim 20, characterized in that The fourth determination module is used to determine a second upper threshold and a second lower threshold based on first historical data of the variable to be measured before the first moment, specifically: The fourth determination module is used for: Acquire first historical data of the variable to be measured before the first moment, including historical maximum data and historical minimum data; Determine the second upper limit threshold based on the historical maximum data; The second lower threshold is determined based on the historical minimum data.

23. The device according to claim 17, characterized in that The anomaly detection module is used to perform anomaly detection on the real data of the variable to be measured at the first moment based on the first detection rule, specifically: The anomaly detection module is used to: Determine the magnitude relationship between the real data and the upper detection threshold and the lower detection threshold; If the real data is greater than or equal to the detection lower limit threshold, and less than or equal to the detection upper limit threshold, the result of the abnormality detection is normal; or, If the real data is smaller than the detection lower limit threshold, or greater than the detection upper limit threshold, the result of the abnormality detection is abnormal.

24. The device according to claim 13, characterized in that The device also includes: A training module is used to train and generate the autoregressive prediction model based on second historical data of the variable to be measured before the first moment.

25. An electronic device, characterized in that: The electronic device comprises: A memory for storing a computer program product; A processor is used to execute a computer program product stored in the memory, and when the computer program product is executed, the method described in any one of claims 1 to 12 is implemented.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1 to 12.