Data cleaning methods, apparatus, computer equipment and storage media

Abstract

This application relates to the field of power grid monitoring technology, and in particular to a data cleaning method, apparatus, computer equipment, and storage medium. The method includes: acquiring initial temperature data and initial current data of a target cable line at each detection time; identifying abnormal temperature data in each initial temperature data set; determining a temperature data observation group including the abnormal temperature data from each initial temperature data set based on the abnormal detection time corresponding to the abnormal temperature data; determining a target current data observation group associated with the temperature data observation group from each initial current data set; determining a second trend corresponding to the temperature data observation group based on a first trend corresponding to the target current data observation group; and compensating for the abnormal temperature data in the temperature data observation group based on the second trend. This application can improve the accuracy of the data.

Description

Technical Field

[0001] This application relates to the field of power grid monitoring technology, and in particular to a data cleaning method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Exceeding the temperature limits of power transmission cable lines can cause damage to the power system or even major accidents. Therefore, monitoring the temperature of cable lines has always been a key research focus in the power system.

[0003] Currently, line temperature monitoring technology relies on the steady-state thermal balance equation of conductors under specific meteorological conditions. This involves directly measuring the conductor temperature by embedding temperature-sensing optical fibers inside the cable, or calculating the cable temperature field to obtain the correspondence between the cable surface temperature and the conductor temperature, and then combining the measured surface temperature value to derive the cable core temperature.

[0004] However, existing line temperature monitoring technology is greatly affected by factors such as meteorological environment, conductor electrical performance, mechanical performance and thermal phenomena, which may lead to data anomalies and reduce the accuracy of the obtained temperature data. Therefore, it is urgent to improve it. Summary of the Invention

[0005] Therefore, it is necessary to provide a data cleaning method, apparatus, computer equipment, and storage medium that can improve data accuracy in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a data cleaning method, which includes:

[0007] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0008] Identify abnormal temperature data in each initial temperature data set;

[0009] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0010] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0011] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0012] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group.

[0013] In one embodiment, determining a target current data observation group associated with the temperature data observation group from the initial current data includes:

[0014] Based on the anomaly detection time, at least one candidate current data observation group is determined from each initial current data;

[0015] Based on the similarity between each candidate current data observation group and the temperature data observation group, the target current data observation group associated with the temperature data observation group is determined from each candidate current data observation group.

[0016] In one embodiment, the data cleaning method further includes:

[0017] For each candidate current data observation group, determine the effective detection time corresponding to the initial temperature data in the temperature data observation group, excluding abnormal temperature data.

[0018] Determine the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and use the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0019] In one embodiment, determining the similarity between the initial current data corresponding to the valid detection time in the candidate current data observation group and the initial temperature data corresponding to the valid detection time in the temperature data observation group includes:

[0020] Based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, a comprehensive observation matrix is ​​determined.

[0021] Determine the covariance matrix based on the comprehensive observation matrix;

[0022] The extreme values ​​are determined from the comprehensive observation matrix, which includes the maximum temperature, minimum temperature, maximum current, and minimum current.

[0023] Based on the covariance matrix and extreme value data, the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined.

[0024] In one embodiment, identifying abnormal temperature data in each initial temperature data set includes:

[0025] For any given detection time, if the initial temperature data corresponding to that detection time is abnormal, but the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is considered as abnormal temperature data.

[0026] In one embodiment, compensation is performed on abnormal temperature data in the temperature data observation group based on a second trend, including:

[0027] Based on the second trend of change, determine the compensation value corresponding to the abnormal temperature data in the temperature data observation group;

[0028] If the error between the compensation value and the abnormal temperature data meets the preset conditions, then the abnormal temperature data in the temperature data observation group will be compensated according to the compensation value.

[0029] Secondly, this application also provides a data cleaning apparatus, which includes:

[0030] The acquisition module is used to acquire the initial temperature data and initial current data of the target cable line at each detection time.

[0031] The filtering module is used to identify abnormal temperature data in each initial temperature data set.

[0032] The temperature combination module is used to determine the temperature data observation group, which includes the abnormal temperature data, from each initial temperature data based on the abnormal detection time corresponding to the abnormal temperature data.

[0033] The current combination module is used to determine the target current data observation group associated with the temperature data observation group from each initial current data;

[0034] The calculation module is used to determine the second trend of the temperature data observation group based on the first trend of the target current data observation group.

[0035] The compensation module is used to compensate for abnormal temperature data in the temperature data observation group based on the second trend of change.

[0036] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0038] Identify abnormal temperature data in each initial temperature data set;

[0039] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0040] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0041] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0042] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group. Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0043] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0044] Identify abnormal temperature data in each initial temperature data set;

[0045] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0046] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0047] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0048] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group. Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0049] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0050] Identify abnormal temperature data in each initial temperature data set;

[0051] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0052] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0053] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0054] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group. The aforementioned data cleaning method, apparatus, computer equipment, and storage medium, when determining that abnormal temperature data exists in the initial temperature data, form a temperature data observation group including the abnormal temperature data based on the abnormal detection time corresponding to the abnormal temperature data, and determine a target current data observation group associated with the temperature data observation group. Based on the principle that changes in temperature data lag behind changes in current data, and based on the first trend of change in the target current data observation group, the second trend of change in the temperature data observation group can be determined. Based on the second trend of change, the compensation value corresponding to the abnormal detection time can be calculated, and compensation is applied to the abnormal temperature data, thus achieving the cleaning of the abnormal temperature data. Attached Figure Description

[0055] Figure 1 This is a diagram illustrating the application environment of a data cleaning method in one embodiment;

[0056] Figure 2 This is a flowchart illustrating a data cleaning method in one embodiment;

[0057] Figure 3 This is a schematic diagram illustrating the lag relationship between the current change trend and the temperature change trend in one embodiment;

[0058] Figure 4 This is a flowchart illustrating the process of determining a target current data observation group in one embodiment;

[0059] Figure 5 This is a flowchart illustrating the process of determining the similarity between candidate current data observation groups and temperature data observation groups in one embodiment.

[0060] Figure 6 This is a flowchart illustrating the process of determining the similarity between candidate current data observation groups and temperature data observation groups in another embodiment.

[0061] Figure 7 This is a schematic diagram of the process for compensating for abnormal temperature data in one embodiment;

[0062] Figure 8 This is a flowchart illustrating the data cleaning method in another embodiment;

[0063] Figure 9 This is a structural block diagram of a data cleaning apparatus in one embodiment;

[0064] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] The data cleaning method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, data acquisition devices 102 (e.g., temperature sensors and current sensors) communicate with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed in the cloud or on other network servers. For example, server 104 acquires initial temperature and initial current data of the target cable line at each detection time; determines abnormal temperature data in each initial temperature data set; based on the abnormal detection time corresponding to the abnormal temperature data, determines a temperature data observation group including the abnormal temperature data from each initial temperature data set; determines a target current data observation group associated with the temperature data observation group from each initial current data set; determines a second change trend corresponding to the temperature data observation group based on a first change trend corresponding to the target current data observation group; and compensates for abnormal temperature data in the temperature data observation group based on the second change trend. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0067] In one embodiment, such as Figure 2 As shown, a data cleaning method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0068] S201, acquire the initial temperature data and initial current data of the target cable line at each detection time.

[0069] Specifically, the temperature sensor acquires initial temperature data at each detection time with a preset sampling interval, and the current sensor acquires initial current data at each detection time with the same preset sampling interval. In particular, the temperature sensor may encounter encoding errors during data transmission, encoding some initial temperature data as incorrect data (e.g., data misalignment). Furthermore, encoding errors may also lead to abnormal timestamps in the initial temperature data.

[0070] In this embodiment, when the timestamps of some initial temperature data are abnormal, each initial temperature data is first preprocessed, that is, the order of each initial temperature data is determined according to the reading time sequence (sending time sequence or receiving time sequence) of each initial temperature data. In this case, the detection time corresponding to the sorted initial temperature data is calibrated, but there may still be erroneous data in some initial temperature data.

[0071] It is understandable that since the temperature data change is caused by the current data change, this embodiment uses the same preset sampling interval to obtain the initial current data at each detection time, in order to use the complete initial current data to compensate for the abnormal temperature data.

[0072] S202, identify abnormal temperature data in each initial temperature data set.

[0073] Understandably, if the target cable line is affected by extreme weather conditions or experiences severe operating conditions, or if both the temperature and current sensors malfunction, the initial temperature and current data at the same detection moment will both show abnormalities. In this case, it can be determined that the target cable line is faulty, and server 104 should send an alarm signal to the remote decision-making platform to obtain relevant control strategies to resolve the fault, without the need for cleaning up erroneous data (abnormal data).

[0074] However, if the temperature sensor (or initial temperature data) is abnormal, but the current sensor (initial current data) is not abnormal, the initial temperature data can be predicted and compensated based on the complete (correct) initial current data.

[0075] Therefore, in this embodiment, when determining whether any initial temperature data is abnormal, it is necessary to determine that the initial temperature data is abnormal, and at the same time, the initial current data corresponding to the initial temperature data is not abnormal. Only under these circumstances can the complete initial current data be used to compensate for the abnormal temperature data.

[0076] S203, Based on the abnormal detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data.

[0077] The verification module in server 104 detects data anomalies in the initial temperature data at a preset frequency, and the verification module in server 104 determines the detection time corresponding to the abnormal temperature data as the abnormal detection time.

[0078] It is understandable that, such as Figure 3 As shown, since changes in temperature data are caused by changes in current data, and the changes in temperature data lag behind changes in current data, the temperature data forms a corresponding temperature change trend within a certain time period, which should be similar to the current change trend within the previous time period. Therefore, in this embodiment, initial temperature data within a certain time period is extracted from each initial temperature data to form a temperature data observation group containing abnormal temperature data; furthermore, using the temperature change trend of this temperature data observation group, the current data group corresponding to a similar current change trend is calculated.

[0079] S204, from each initial current data, determine the target current data observation group associated with the temperature data observation group.

[0080] Among them, there is a correlation between the target current data observation group and the temperature data observation group. This correlation can be based on the similarity between data changes (i.e., the temperature change trend and the current change trend) satisfying the preset correlation conditions.

[0081] Specifically, when selecting the target current data observation group, based on the principle that the change in temperature data lags behind the change in current data, the time period corresponding to the current data observation group should be equal to or ahead of the time period corresponding to the temperature data observation group. In order to ensure the accuracy of the correlation, the number of initial current data in the current data observation group should be the same as the number of initial temperature data in the temperature data observation group.

[0082] For example, the order of detection times is represented by the size of the number (the larger the number, the later the time). If the detection time period corresponding to the temperature data observation group contains 100 detection times, the numbers of each detection time are {201, 202, 203, 204...300}. Correspondingly, the detection time period corresponding to the target current data observation group also contains 100 detection times, and the numbers of each detection time can be {201, 202, 203, 204...300} (representing the same time period as the temperature data observation group) or {101, 102, 103, 104...200} (representing the time period preceding the temperature data observation group).

[0083] S205, based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group.

[0084] Specifically, when determining the first trend of change corresponding to the target current data observation group, the current data at each detection time in the target current data observation group are connected. Based on these connections, the slope corresponding to each initial current data (detection time) in the target current data observation group is determined. Based on the slope of each initial current data (detection time), the first trend of change corresponding to the target current data observation group is determined.

[0085] Furthermore, the first trend of change, namely the slope corresponding to each initial current data (detection time), is assigned to the initial temperature data corresponding to the corresponding detection time in the temperature data observation group. In this case, the abnormal detection time is also assigned a corresponding slope. At this time, the slope of each initial temperature data (detection time) in the temperature data observation group is determined as the second trend of change corresponding to the temperature data observation group.

[0086] S206, based on the second trend of change, compensate for abnormal temperature data in the temperature data observation group.

[0087] Specifically, based on the second trend, for abnormal temperature data in the temperature data observation group, given the normal temperature data before and after the abnormal temperature data, as well as the slope of the abnormal temperature data, the corresponding compensation value can be calculated. Furthermore, based on this compensation value, the abnormal temperature data is compensated, thus achieving the cleaning of the abnormal temperature data.

[0088] In the above data cleaning method, when it is determined that there is abnormal temperature data in the initial temperature data, a temperature data observation group including the abnormal temperature data is formed based on the abnormal detection time corresponding to the abnormal temperature data, and a target current data observation group associated with the temperature data observation group is determined. Based on the principle that the change of temperature data lags behind the change of current data, the second change trend of the temperature data observation group can be determined based on the first change trend of the target current data observation group. Based on the second change trend, the compensation value corresponding to the abnormal detection time can be calculated to compensate for the abnormal temperature data, thereby achieving the cleaning of the abnormal temperature data.

[0089] In one embodiment, this embodiment provides an optional method for determining abnormal temperature data in each initial temperature data, that is, a method for refining S201. The specific implementation process may include: for any detection time, if the initial temperature data corresponding to that detection time is abnormal, and the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is taken as abnormal temperature data.

[0090] The verification module in server 104 detects the initial temperature data at each detection time. If it detects that the timestamp of the initial temperature data is abnormal, it determines that the initial temperature data corresponding to the detection time is abnormal. Furthermore, server 104 determines whether the initial current data corresponding to the detection time is abnormal, in order to verify whether the cause of the abnormal temperature data is a fault in the entire target cable line (including temperature sensor and current sensor) or simply a coding error of the temperature sensor.

[0091] In this embodiment, the initial temperature data corresponding to the abnormal detection time is determined to be abnormal temperature data only if the initial current data at the abnormal detection time is not abnormal, so as to ensure the effectiveness of the subsequent data cleaning process.

[0092] Furthermore, when comparing the temperature data observation group with the target current data observation group, the temperature data observation group can contain at least one abnormal temperature data. That is, after determining the abnormal detection time, the verification module in server 104 can record the number of abnormal detection times. When the number of abnormal detection times reaches the compensation condition, the subsequent compensation process is executed. In this case, multiple abnormal temperature data observation groups can be included in the same temperature data observation group to achieve synchronous cleaning.

[0093] To determine the accurate target current data observation group, such as Figure 4 As shown, this embodiment provides an optional method for determining the target current data observation group associated with the temperature data observation group from each initial current data, that is, a method for refining S204. The specific implementation process may include:

[0094] S401, Based on the anomaly detection time, determine at least one candidate current data observation group from each initial current data.

[0095] In cases where the lag time between the current change trend and the temperature change trend is unknown, it is necessary to select multiple candidate current data observation groups from the initial current data and compare each of the multiple candidate current data observation groups with the temperature data observation group. In addition, there may be overlap between the candidate current data observation groups or there may be no overlap, as in the example of the two candidate current data observation groups in S204 above.

[0096] S402, based on the similarity between each candidate current data observation group and the temperature data observation group, determine the target current data observation group associated with the temperature data observation group from each candidate current data observation group.

[0097] When calculating the similarity between each candidate current data observation group and the temperature data observation group, such as Figure 5 As shown, in one embodiment, the data cleaning method further includes:

[0098] S501, for each candidate current data observation group, determine the effective detection time corresponding to the initial temperature data in the temperature data observation group, excluding abnormal temperature data.

[0099] When determining the similarity between the candidate current data observation group and the temperature data observation group, it is first necessary to ensure that the number of data points is the same. Furthermore, since anomalous temperature data in the temperature data observation group is unusable (for example, if there are 5 anomalous temperature data points, then the remaining 95 are usable data), the corresponding number of usable data points in the candidate current data observation group should also be 95.

[0100] In this embodiment, the effective detection time corresponding to the initial temperature data in the temperature data observation group, excluding abnormal temperature data, is determined. As in the example above, the temperature data observation group has 95 effective detection times.

[0101] S502, determine the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and use the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0102] When determining the initial current data corresponding to the valid detection time in the candidate current data observation group, it can be determined based on the time difference Δt between the candidate current data observation group and the temperature data observation group. The time difference Δt can be represented by the difference in the numbers corresponding to the detection times.

[0103] As in the example in S204, Δt = 0, 100; then, correspondingly, assuming that in the temperature data observation group, one of the effective detection times corresponds to time t1, then the effective detection time corresponding to the detection time t1' in the candidate current data observation group is: t1' = t1 - Δt.

[0104] Furthermore, in one embodiment, when determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, such as... Figure 6 As shown, the specific process may include the following:

[0105] S601. Based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, determine the comprehensive observation matrix.

[0106] For ease of description, the effective detection time in the temperature data observation group is defined as the first effective detection time, and the detection time in the candidate current data observation group corresponding to the first effective detection time is defined as the second effective detection time; each first detection time corresponds to each second detection time, that is, the first detection time lags behind the second detection time by a time Δt.

[0107] Specifically, for any set of first and second effective detection times that have a corresponding relationship, the initial current data corresponding to the first effective detection time and the initial temperature data corresponding to the second effective detection time form an observation vector. The initial temperature data corresponding to each set of first effective detection times and the initial current data corresponding to each set of second effective detection times form multiple observation vectors X1, X2, X3...X 95. A comprehensive observation matrix [X] is formed based on each observation vector.

[0108] For example, the total number of initial temperature data in the temperature data observation group is N, the number of abnormal temperature data is z, and the number of initial temperature data corresponding to the first valid detection time is Nz. The observation vectors X1 and X2 are shown below:

[0109]

[0110] The comprehensive observation matrix [X] is shown below.

[0111]

[0112] Taking any one of the observation vectors X1 as an example, where x1 is the initial temperature data corresponding to the first effective detection time, and y1 is the initial current data corresponding to the second effective detection time.

[0113] S602, Determine the covariance matrix based on the comprehensive observation matrix.

[0114] Specifically, the process of calculating the covariance matrix is ​​as follows:

[0115] First, based on the mean vector M of each observation vector, for example, N = 100:

[0116]

[0117] Secondly, let If k = 1, 2, 3...(100-z), then the new matrix B is:

[0118]

[0119] Then, the covariance matrix S can be obtained from matrix B as follows:

[0120]

[0121] In this embodiment, the covariance matrix S is a 2-row, 2-column matrix. The purpose of forming the comprehensive observation matrix is ​​to calculate the covariance corresponding to the observation matrix. In probability theory and statistics, covariance is used to measure the overall error between two variables. The covariance matrix simply represents the covariance relationship of all variables in matrix form.

[0122] S603, determine the extreme values ​​from the comprehensive observation matrix.

[0123] The extreme values ​​include the maximum and minimum temperatures, as well as the maximum and minimum currents. Specifically, the maximum temperature x... max The maximum value among all initial temperature data in the comprehensive observation matrix, and the minimum temperature value x. minThe minimum value among all initial temperature data in the comprehensive observation matrix; the maximum current value y. max The maximum value and minimum value of current y are the values ​​of all initial current data in the comprehensive observation matrix. min It is the minimum value among all initial current data in the comprehensive observation matrix.

[0124] S604. Based on the covariance matrix and the target data, determine the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group.

[0125] Specifically, based on the maximum temperature x max and the maximum current y max This forms the maximum value vector X. max ,Right now Based on the maximum temperature x min and the maximum current y min This forms the maximum value vector X. min ,Right now

[0126] The process of calculating the similarity Δd between the candidate current data observation group and the temperature data observation group is as follows:

[0127]

[0128]

[0129] Where S is the covariance matrix in S602 above, and M is the mean vector M.

[0130] Furthermore, the similarity Δd is calculated, where Δd = d max (T,I)-d min (T, I).

[0131] Specifically, the smaller Δd is, the higher the similarity between the candidate current data observation group and the temperature data observation group. When determining the similarity between the candidate current data observation group and the temperature data observation group, only the similarity between the data is considered, and the effect of time difference is not taken into account.

[0132] like Figure 7 As shown, this embodiment provides an optional method for compensating for abnormal temperature data in the temperature data observation group based on a second trend, that is, a method for refining S206. The specific implementation process may include:

[0133] S701, Based on the second trend of change, determine the compensation value corresponding to the abnormal temperature data in the temperature data observation group.

[0134] Specifically, when determining the compensation value for abnormal temperature data corresponding to the temperature data observation group based on the second trend of change, the compensation value for any abnormal temperature data can be calculated based on the initial temperature data corresponding to the effective detection time before the abnormal temperature data, the initial temperature data corresponding to the effective detection time after the abnormal temperature data, and the slope corresponding to the abnormal temperature data.

[0135] S702, if the error between the compensation value and the abnormal temperature data meets the preset conditions, then the abnormal temperature data in the temperature data observation group is compensated according to the compensation value.

[0136] If the difference between the compensation value and the abnormal temperature data (coded error data) is less than 10%, the abnormal temperature data in the temperature data observation group will be compensated according to the compensation value; otherwise, the abnormal temperature data will be retained.

[0137] For example, based on the above embodiments, this embodiment provides an optional example of a data cleaning method. For instance... Figure 8 As shown, the specific implementation process includes:

[0138] S801, acquire the initial temperature data and initial current data of the target cable line at each detection time.

[0139] S802, for any detection time, if the initial temperature data corresponding to that detection time is abnormal, but the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is taken as abnormal temperature data.

[0140] S803, based on the abnormal detection time corresponding to the abnormal temperature data, determines the temperature data observation group including the abnormal temperature data from each initial temperature data.

[0141] S804, based on the anomaly detection time, determine at least one candidate current data observation group from each initial current data.

[0142] S805, based on the similarity between each candidate current data observation group and the temperature data observation group, determine the target current data observation group associated with the temperature data observation group from each candidate current data observation group.

[0143] Specifically, for each candidate current data observation group, the effective detection time corresponding to the initial temperature data (excluding abnormal temperature data) in the temperature data observation group is determined; the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined, and the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

[0144] Furthermore, determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group includes: determining a comprehensive observation matrix based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group; determining a covariance matrix based on the comprehensive observation matrix; determining the extreme values ​​from the comprehensive observation matrix, wherein the extreme values ​​include the maximum temperature, minimum temperature, maximum current, and minimum current; and determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group based on the covariance matrix and the extreme values.

[0145] S806, based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group.

[0146] S807, Based on the second trend of change, determine the compensation value corresponding to the abnormal temperature data in the temperature data observation group.

[0147] S808 If the error between the compensation value and the abnormal temperature data meets the preset conditions, then the abnormal temperature data in the temperature data observation group is compensated according to the compensation value.

[0148] The specific processes of S801-S808 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0149] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a data cleaning apparatus for implementing the data cleaning method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data cleaning apparatus embodiments provided below can be found in the limitations of the data cleaning method described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 9 As shown, a data cleaning device is provided, including: an acquisition module 11, a filtering module 12, a temperature combination module 13, a current combination module 14, a calculation module 15, and a compensation module 16, wherein:

[0152] The acquisition module 11 is used to acquire the initial temperature data and initial current data of the target cable line at each detection time.

[0153] Filtering module 12 is used to identify abnormal temperature data in each initial temperature data set;

[0154] Temperature combination module 13 is used to determine a temperature data observation group including abnormal temperature data from each initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data.

[0155] The current combination module 14 is used to determine the target current data observation group associated with the temperature data observation group from each initial current data;

[0156] Calculation module 15 is used to determine the second trend of temperature data observation group based on the first trend of the target current data observation group.

[0157] The compensation module 16 is used to compensate for abnormal temperature data in the temperature data observation group according to the second trend.

[0158] In one embodiment, the screening module 12 is further configured to: determine at least one candidate current data observation group from each initial current data according to the anomaly detection time;

[0159] Based on the similarity between each candidate current data observation group and the temperature data observation group, the target current data observation group associated with the temperature data observation group is determined from each candidate current data observation group.

[0160] In one embodiment, the data cleaning apparatus further includes a similarity calculation module 15, which includes:

[0161] The effective screening submodule is used to determine the effective detection time corresponding to the initial temperature data in the temperature data observation group, excluding abnormal temperature data, for each candidate current data observation group.

[0162] The similarity calculation submodule is used to determine the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and to use the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0163] In one embodiment, the similarity calculation submodule is further configured to: determine a comprehensive observation matrix based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group;

[0164] Determine the covariance matrix based on the comprehensive observation matrix;

[0165] The extreme values ​​are determined from the comprehensive observation matrix, which includes the maximum temperature, minimum temperature, maximum current, and minimum current.

[0166] Based on the covariance matrix and extreme value data, the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined.

[0167] In one embodiment, the filtering module 12 is further configured to: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, then the initial temperature data corresponding to the detection time is regarded as abnormal temperature data.

[0168] In one embodiment, the compensation module 16 is further configured to: determine the compensation value corresponding to the abnormal temperature data in the temperature data observation group according to the second trend of change;

[0169] If the error between the compensation value and the abnormal temperature data meets the preset conditions, then the abnormal temperature data in the temperature data observation group will be compensated according to the compensation value.

[0170] Each module in the aforementioned data cleaning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores XX data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data cleaning method.

[0172] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0174] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0175] Identify abnormal temperature data in each initial temperature data set;

[0176] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0177] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0178] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0179] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group.

[0180] In one embodiment, when the processor executes the logic of a computer program to determine a target current data observation group associated with a temperature data observation group from each initial current data, the specific steps are as follows: determining at least one candidate current data observation group from each initial current data according to the anomaly detection time; determining the target current data observation group associated with the temperature data observation group from each candidate current data observation group according to the similarity between each candidate current data observation group and the temperature data observation group.

[0181] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each candidate current data observation group, determining the effective detection time corresponding to the initial temperature data (excluding abnormal temperature data) in the temperature data observation group; determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and using the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0182] In one embodiment, when the processor executes a computer program to determine the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, the processor specifically implements the following steps: determining a comprehensive observation matrix based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group; determining a covariance matrix based on the comprehensive observation matrix; determining extreme value data from the comprehensive observation matrix, wherein the extreme value data includes the maximum temperature, minimum temperature, maximum current, and minimum current; and determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group based on the covariance matrix and the extreme value data.

[0183] In one embodiment, when the processor executes the logic of the computer program to determine the abnormal temperature data in each initial temperature data, it specifically implements the following steps: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, then the initial temperature data corresponding to the detection time is taken as the abnormal temperature data.

[0184] In one embodiment, when the processor executes the logic of the computer program to compensate for abnormal temperature data in the temperature data observation group according to the second trend, the following steps are specifically implemented: determining the compensation value corresponding to the abnormal temperature data in the temperature data observation group according to the second trend; if the error between the compensation value and the abnormal temperature data meets the preset condition, then compensating for the abnormal temperature data in the temperature data observation group according to the compensation value.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0186] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0187] Identify abnormal temperature data in each initial temperature data set;

[0188] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0189] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0190] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0191] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group.

[0192] In one embodiment, when the logic of the computer program determining the target current data observation group associated with the temperature data observation group from each initial current data is executed by the processor, the following steps are specifically implemented: determining at least one candidate current data observation group from each initial current data according to the anomaly detection time; determining the target current data observation group associated with the temperature data observation group from each candidate current data observation group according to the similarity between each candidate current data observation group and the temperature data observation group.

[0193] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each candidate current data observation group, determining the effective detection time corresponding to the initial temperature data (excluding abnormal temperature data) in the temperature data observation group; determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and using the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0194] In one embodiment, when the logic for determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is executed by the processor, the following steps are specifically implemented: A comprehensive observation matrix is ​​determined based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group; a covariance matrix is ​​determined based on the comprehensive observation matrix; extreme value data is determined from the comprehensive observation matrix, wherein the extreme value data includes the maximum temperature, minimum temperature, maximum current, and minimum current; and the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined based on the covariance matrix and the extreme value data.

[0195] In one embodiment, when the logic for determining abnormal temperature data in each initial temperature data is executed by the processor, the following steps are specifically implemented: for any detection time, if the initial temperature data corresponding to that detection time is abnormal, and the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is taken as abnormal temperature data.

[0196] In one embodiment, when the logic of the computer program compensating for abnormal temperature data in the temperature data observation group according to the second trend is executed by the processor, the following steps are specifically implemented: determining the compensation value corresponding to the abnormal temperature data in the temperature data observation group according to the second trend; if the error between the compensation value and the abnormal temperature data meets the preset condition, then compensating for the abnormal temperature data in the temperature data observation group according to the compensation value.

[0197] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0198] Acquire the initial temperature and initial current data of the target cable line at each detection time;

[0199] Identify abnormal temperature data in each initial temperature data set;

[0200] Based on the anomaly detection time corresponding to the abnormal temperature data, determine the temperature data observation group including the abnormal temperature data from each initial temperature data;

[0201] From the initial current data, determine the target current data observation group that is associated with the temperature data observation group;

[0202] Based on the first trend of change corresponding to the target current data observation group, determine the second trend of change corresponding to the temperature data observation group;

[0203] Based on the second trend of change, compensation is applied to abnormal temperature data in the temperature data observation group.

[0204] In one embodiment, when the logic of the computer program determining the target current data observation group associated with the temperature data observation group from each initial current data is executed by the processor, the following steps are specifically implemented: determining at least one candidate current data observation group from each initial current data according to the anomaly detection time; determining the target current data observation group associated with the temperature data observation group from each candidate current data observation group according to the similarity between each candidate current data observation group and the temperature data observation group.

[0205] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each candidate current data observation group, determining the effective detection time corresponding to the initial temperature data (excluding abnormal temperature data) in the temperature data observation group; determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, and using the similarity as the similarity between the candidate current data observation group and the temperature data observation group.

[0206] In one embodiment, when the logic for determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is executed by the processor, the following steps are specifically implemented: A comprehensive observation matrix is ​​determined based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group; a covariance matrix is ​​determined based on the comprehensive observation matrix; extreme value data is determined from the comprehensive observation matrix, wherein the extreme value data includes the maximum temperature, minimum temperature, maximum current, and minimum current; and the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined based on the covariance matrix and the extreme value data.

[0207] In one embodiment, when the logic for determining abnormal temperature data in each initial temperature data is executed by the processor, the following steps are specifically implemented: for any detection time, if the initial temperature data corresponding to that detection time is abnormal, and the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is taken as abnormal temperature data.

[0208] In one embodiment, when the logic of the computer program compensating for abnormal temperature data in the temperature data observation group according to the second trend is executed by the processor, the following steps are specifically implemented: determining the compensation value corresponding to the abnormal temperature data in the temperature data observation group according to the second trend; if the error between the compensation value and the abnormal temperature data meets the preset condition, then compensating for the abnormal temperature data in the temperature data observation group according to the compensation value.

[0209] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0210] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0211] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0212] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data cleaning method, characterized in that, The method includes: Acquire the initial temperature and initial current data of the target cable line at each detection time; For any detection time, if the initial temperature data corresponding to that detection time is abnormal, but the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is taken as abnormal temperature data. Based on the abnormal detection time corresponding to the abnormal temperature data, a temperature data observation group including the abnormal temperature data is determined from each initial temperature data. Based on the anomaly detection time, at least one candidate current data observation group is determined from each initial current data; Based on the similarity between each candidate current data observation group and the temperature data observation group, a target current data observation group associated with the temperature data observation group is determined from each candidate current data observation group. Based on the first trend of change corresponding to the target current data observation group, a second trend of change corresponding to the temperature data observation group is determined; wherein, the first trend of change is determined based on the slope of each initial current data in the target current data observation group in the line connecting each initial current data; the second trend of change is determined by the slope of each initial temperature data in the temperature data observation group, and the slope of each initial temperature data is the slope of the initial current data at the corresponding detection time in the target current data observation group; Based on the second trend of change, compensation is made for abnormal temperature data in the temperature data observation group.

2. The method according to claim 1, characterized in that, The method further includes: For each candidate current data observation group, determine the effective detection time corresponding to the initial temperature data in the temperature data observation group, excluding the abnormal temperature data. The similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined, and the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

3. The method according to claim 2, characterized in that, The determination of the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group includes: Based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, a comprehensive observation matrix is ​​determined. Based on the comprehensive observation matrix, determine the covariance matrix; The extreme values ​​are determined from the comprehensive observation matrix, wherein the extreme values ​​include the maximum temperature, the minimum temperature, the maximum current, and the minimum current; Based on the covariance matrix and the extreme value data, the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group is determined.

4. The method according to claim 3, characterized in that, The step of determining the comprehensive observation matrix based on the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group includes: The effective detection time in the temperature data observation group is determined as the first effective detection time, and the detection time in the candidate current data observation group corresponding to the first effective detection time is determined as the second effective detection time; For any pair of first and second effective detection times that have a corresponding relationship, the initial current data corresponding to the first effective detection time and the initial temperature data corresponding to the second effective detection time form an observation vector; The initial temperature data corresponding to the first effective detection time and the initial current data corresponding to the second effective detection time of each group are used to form multiple observation vectors, and a comprehensive observation matrix is ​​formed based on each observation vector.

5. The method according to claim 1, characterized in that, The compensation for abnormal temperature data in the temperature data observation group based on the second trend includes: The compensation value corresponding to the abnormal temperature data in the temperature data observation group is determined based on the second change trend. If the error between the compensation value and the abnormal temperature data meets the preset conditions, then the abnormal temperature data in the temperature data observation group is compensated according to the compensation value.

6. The method according to claim 5, characterized in that, The step of determining the compensation value corresponding to the abnormal temperature data in the temperature data observation group based on the second trend includes: Based on the second trend of change, the normal temperature data before the abnormal temperature data in the temperature data observation group, the normal temperature data after the abnormal temperature data, and the slope corresponding to the abnormal temperature data are determined. Based on the normal temperature data before the abnormal temperature data, the normal temperature data after the abnormal temperature data, and the slope corresponding to the abnormal temperature data, the compensation value corresponding to the abnormal temperature data is determined.

7. A data cleaning apparatus, characterized in that, The device includes: The acquisition module is used to acquire the initial temperature data and initial current data of the target cable line at each detection time. The filtering module is used to, for any detection time, if the initial temperature data corresponding to that detection time is abnormal, but the initial current data corresponding to that detection time is normal, then the initial temperature data corresponding to that detection time is regarded as abnormal temperature data. The temperature combination module is used to determine a temperature data observation group including the abnormal temperature data from each initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data. The current combination module is used to determine at least one candidate current data observation group from each initial current data according to the anomaly detection time; and to determine a target current data observation group associated with the temperature data observation group from each candidate current data observation group according to the similarity between each candidate current data observation group and the temperature data observation group. The calculation module is used to determine a second trend corresponding to the temperature data observation group based on a first trend corresponding to the target current data observation group; wherein, the first trend is determined based on the slope of each initial current data in the target current data observation group in the line connecting each initial current data; the second trend is determined by the slope corresponding to each initial temperature data in the temperature data observation group, and the slope corresponding to each initial temperature data is the slope corresponding to the initial current data at the corresponding detection time in the target current data observation group; The compensation module is used to compensate for abnormal temperature data in the temperature data observation group according to the second trend.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images