Abnormal data correction method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to the field of big data, and discloses an abnormal data correction method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of obtaining target business data of a target business data source and multiple pieces of first historical business data; fusing the plurality of pieces of first historical business data to obtain first fused business data; based on the difference between the first fusion service data and the target service data, performing anomaly detection on the target service data to obtain a data detection result; and under the condition that the data detection result represents that the target business data is abnormal, acquiring data source association weights between the target business data source and the preset business data sources, and correcting the target business data according to the data source association weights to obtain corrected business data. By adopting the method, the accuracy of abnormal data correction can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to an abnormal data correction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of business and the economy, people have increasingly more channels to conduct business, resulting in a growing amount of business data. When business channels malfunction, some business data may become abnormal. In order to ensure the normal processing of business data, it is necessary to correct the abnormal business data.

[0003] The traditional method for correcting abnormal business data is to replace the abnormal business data with business data from a backup data source.

[0004] However, when using the above-mentioned abnormal data correction methods, there may be situations where the business data source of the abnormal business data does not have a backup data source, making it impossible to correct the abnormal business data. Alternatively, the backup data source may also fail, causing the business data in the backup data source to also become abnormal. Overall, this results in low accuracy of abnormal data correction. Summary of the Invention

[0005] Therefore, it is necessary to provide an anomaly data correction method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of anomaly data correction in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for correcting anomalous data, including:

[0007] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0008] The multiple first historical business data are merged to obtain the first merged business data;

[0009] Based on the differences between the first fused service data and the target service data, anomaly detection is performed on the target service data to obtain data detection results.

[0010] If the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0011] In one embodiment, obtaining the data source association weights between the target business data source and each preset business data source includes:

[0012] Obtain the second historical business data from each preset business data source;

[0013] Based on the second historical business data, causal verification is performed between the target business data source and each preset business data source to obtain the causal verification results of each preset business data source for the target business data source.

[0014] Based on the causal verification results of each preset business data source for the target business data source, data source association weights are constructed between the target business data source and each preset business data source.

[0015] In one embodiment, the step of performing causality checks on the target business data source and each of the preset business data sources based on the second historical business data, to obtain the causality check results of each preset business data source relative to the target business data source, includes:

[0016] For each of the preset business data sources, the first fused business data and the second historical business data are fused to obtain the second fused business data;

[0017] Determine the distribution characteristic data between the first fused business data and the second fused business data, and determine the distribution characteristic data as the causal verification results of each preset business data source for the target business data source.

[0018] In one embodiment, the step of constructing data source association weights between the target business data source and each preset business data source based on the causal verification results of each preset business data source for the target business data source includes:

[0019] Based on the causal verification results of each preset business data source for the target business data source, related business data sources are selected from each preset business data source;

[0020] Construct the data source association weight between the target business data source and the associated business data source, and determine the data source association weight between the target business data source and the associated business data source as the data source association weight between the target business data source and each preset business data source respectively.

[0021] In one embodiment, the step of correcting the target business data according to the data source association weight to obtain corrected business data includes:

[0022] Based on the data source association weight, the second historical business data of the associated business data source are merged to obtain the third merged business data;

[0023] Extract the change feature data of the target business data;

[0024] Based on the third fused service data and the change characteristic data, the target service data is corrected to obtain corrected service data.

[0025] In one embodiment, the step of fusing the plurality of first historical business data to obtain first fused business data is performed by a federated learning model; after correcting the target business data according to the data source association weights to obtain corrected business data, the method further includes:

[0026] For each edge node, the node model deployed on the edge node is updated based on the corrected business data to obtain updated model parameters;

[0027] The federated learning model is updated based on the updated model parameters of each edge node.

[0028] In one embodiment, after correcting the target business data according to the data source association weight to obtain corrected business data, the method further includes:

[0029] The target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source are fused together to obtain blockchain information;

[0030] The blockchain information is stored in the blockchain.

[0031] Secondly, this application also provides an anomalous data correction device, comprising:

[0032] The acquisition module is used to acquire the target business data from the target business data source and multiple first historical business data.

[0033] The fusion module is used to fuse the multiple first historical business data to obtain first fused business data;

[0034] The detection module is used to perform anomaly detection on the target business data based on the difference between the first fused business data and the target business data, and obtain data detection results;

[0035] The correction module is used to obtain the data source association weights between the target business data source and each preset business data source when the data detection result indicates that the target business data is abnormal, and to correct the target business data according to the data source association weights to obtain corrected business data.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0038] The multiple first historical business data are merged to obtain the first merged business data;

[0039] Based on the differences between the first fused service data and the target service data, anomaly detection is performed on the target service data to obtain data detection results.

[0040] If the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained. Based on the data source association weights, the target business data is corrected to obtain corrected business data.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0043] The multiple first historical business data are merged to obtain the first merged business data;

[0044] Based on the differences between the first fused service data and the target service data, anomaly detection is performed on the target service data to obtain data detection results.

[0045] If the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0048] The multiple first historical business data are merged to obtain the first merged business data;

[0049] Based on the differences between the first fused service data and the target service data, anomaly detection is performed on the target service data to obtain data detection results.

[0050] If the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0051] The aforementioned abnormal data correction method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire target business data from a target business data source and multiple first historical business data; fuse the multiple first historical business data to obtain first fused business data; perform anomaly detection on the target business data based on the differences between the first fused business data and the target business data to obtain a data detection result; when the data detection result indicates that the target business data is abnormal, acquire the data source association weights between the target business data source and each preset business data source, and correct the target business data according to the data source association weights to obtain corrected business data.

[0052] Thus, based on the first historical business data, and considering the correlation between the first historical business data at the second time and the target business data at the first time, the difference between the first fused business data obtained by fusing the first historical business data and the target business data is used as the basis for anomaly detection of the target business data. This allows the detected data to accurately capture business data that changes abnormally over time. When the data detection results indicate that the target business data is abnormal, considering the possible correlation between each preset business data source and the target business data source, the target business data is corrected based on the data source correlation weight between the target business data source and each preset business data source. This ensures that each preset business data source participates in the correction decision-making process for the target business data, improving the accuracy of anomaly correction. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a diagram illustrating the application environment of an abnormal data correction method in one embodiment.

[0055] Figure 2 This is a flowchart illustrating an abnormal data correction method in one embodiment;

[0056] Figure 3 This is a flowchart illustrating the steps for obtaining the data source association weights between the target business data source and each preset business data source in one embodiment.

[0057] Figure 4 This is a flowchart illustrating the steps of correcting the target business data based on the association weight of the data source in one embodiment.

[0058] Figure 5 This is a structural block diagram of an abnormal data correction device in one embodiment;

[0059] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. It should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary and are intended only to illustrate the feasibility of implementing the technical solutions of this application, but do not imply that the applicant has already used or necessarily used such solutions.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations. The acquisition, storage, use and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0062] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various types of data, but these data are not limited by these terms. These terms are only used to distinguish between the first type of data and the second type of data. The term "comprising" and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "each" used in this application refers to one or more. The term "at least one" used in this application refers to one of the embodiments, or any combination of multiple embodiments. The term "one of the embodiments" used in this application refers to one of the embodiments.

[0063] The abnormal data correction method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 obtains target business data from the target business data source and multiple first historical business data sets; it merges the multiple first historical business data sets to obtain first merged business data; based on the differences between the first merged business data and the target business data, it performs anomaly detection on the target business data to obtain data detection results; if the data detection results indicate that the target business data is abnormal, it obtains the data source association weights between the target business data source and each preset business data source, and corrects the target business data according to the data source association weights to obtain corrected business data. Specifically, terminal 102 can be a device used to log into the business system or a device used to handle business; therefore, server 104 sends corrected business data to terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In one exemplary embodiment, such as Figure 2 As shown, an abnormal data correction method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0065] Step 202: Obtain the target business data from the target business data source and multiple first historical business data.

[0066] In step 202, the target business data source is a data source awaiting data anomaly detection and correction. The target business data source can be set by the user as needed or specified periodically. For example, a corresponding time period for anomaly detection and correction can be specified for each business data source, and the business data source that meets the time period is determined as the target business data source. The target business data is generated in the first time, and multiple first historical business data are generated in the second time. The first and second times can be time periods or points in time, with the second time earlier than the first time (specifically, any point in the second time is earlier than any point in the first time). The first time can be set by the user as needed or specified in chronological order. For example, if the target business data source includes data a1, a2, a3, and a4 that change over time, then a1 and a2 can be initially set as first historical business data, and a3 as target business data. After anomaly detection is performed on a3, a1, a2, and a3 are set as first historical business data, and a4 is set as target business data.

[0067] In step 202, the target business data can be a single data point or multiple data points. However, to facilitate subsequent data detection and correction, the target business data is usually a single data point. The first historical business data can also be a single data point or multiple data points. In order to ensure that as much first historical business data as possible can participate in the decision-making basis of the first fused business data,

[0068] Step 204: Merge multiple first historical business data sets to obtain first merged business data.

[0069] For example, step 204 includes: obtaining initial data corresponding to the target business data; obtaining the influence coefficient of each first historical business data on the target business data; merging each first historical business data according to the influence coefficient of each first historical business data on the target business data to obtain first merged impact business data; and merging the first merged impact business data and the initial data to obtain first merged business data at the first time.

[0070] The initial data are empirical values, and the influence coefficients of each first historical business data on the target business data are empirical values, which can also be set by the user as needed. The influence coefficients of the first historical business data on the target business data are used to characterize the degree of influence of the first historical business data on the target business data.

[0071] Furthermore, as an embodiment, based on the influence coefficient of each first historical business data on the target business data, the first historical business data are fused to obtain the first fused impact business data, including: multiplying and fusing each first historical business data with its influence coefficient on the target business data to obtain the impact business data corresponding to the first historical business data; and summing and fusing the impact business data corresponding to each first historical business data to obtain the first fused impact business data.

[0072] As one embodiment, the first fusion-affected business data and the initial data are fused to obtain the first fusion business data at the first time, including: summing and fusing the first fusion-affected business data, the initial data and the allowed residual to obtain the first fusion business data at the first time.

[0073] The allowable residual can be set by the user as needed, or it can be an empirical value.

[0074] Optionally, the first fusion-affected business data, initial data, and allowable residuals are summed and fused to obtain the first fusion business data at the first moment, which can be expressed by the following formula:

[0075]

[0076] in, For the first integrated business data, For initial data, For the i-th first historical business data, Let be the impact coefficient of the i-th historical business data on the target business data. To allow for residuals.

[0077] As another embodiment, the first fusion-affected business data and the initial data are fused to obtain the first fusion business data at the first time, including: taking the second historical business data from the preset business data source at the second time, and fusing the second historical business data according to the influence coefficient of each second historical business data on the target business data to obtain the second fusion-affected business data; and summing and fusing the first fusion-affected business data, the second fusion-affected business data, the initial data and the allowable residual to obtain the first fusion business data at the first time.

[0078] Optionally, the first fusion-affected business data, the second fusion-affected business data, the initial data, and the allowable residual are summed and fused to obtain the first fusion business data at the first time, which can be expressed by the following formula:

[0079]

[0080] in, For the j-th second historical business data, Let be the impact coefficient of the j-th second historical business data on the target business data.

[0081] As another embodiment, the first fusion-affected business data and the initial data are fused to obtain the first fusion business data at a first time, including: summing and fusing the first fusion-affected business data, the initial data and the allowable residual to obtain a first sum value; summing and fusing the first fusion-affected business data, the second fusion-affected business data, the initial data and the allowable residual to obtain a second sum value; and determining the first fusion business data at the first time based on the first sum value and the second sum value.

[0082] Optionally, the first converged service data for a first time period is determined based on the first sum and the second sum, including one of the following: determining the average value between the first sum and the second sum as the first converged service data for the first time period; determining the maximum value between the first sum and the second sum as the first converged service data for the first time period; or determining the minimum value between the first sum and the second sum as the first converged service data for the first time period.

[0083] Step 206: Based on the differences between the first fused business data and the target business data, perform anomaly detection on the target business data to obtain the data detection results.

[0084] In step 206, the difference between the first fused business data and the target business data is positively correlated with the degree of data anomaly represented by the data detection results; the difference between the first fused business data and the target business data includes the difference value between the first fused business data and the target business data.

[0085] As an example, step 206 includes: obtaining the preset difference value range corresponding to each degree of anomaly; selecting the target difference value range corresponding to the difference value between the first fused business data and the target business data from each preset difference value range; and determining the degree of anomaly corresponding to the target difference value range as the data detection result.

[0086] In another embodiment, step 206 includes: if the difference between the first fused service data and the target service data is greater than a preset difference threshold, then a data detection result indicating that the target service data is abnormal is generated; if the difference between the first fused service data and the target service data is not greater than the preset difference threshold, then a data detection result indicating that the target service data is not abnormal is generated.

[0087] The preset difference range can be an empirical value; the preset difference threshold can be an empirical value.

[0088] Step 208: If the data detection results indicate that the target business data is abnormal, obtain the data source association weights between the target business data source and each preset business data source, and correct the target business data according to the data source association weights to obtain corrected business data.

[0089] As an example, the above method further includes: if the degree of data anomaly represented by the data detection result is greater than a preset degree threshold, determining that the data detection result represents an anomaly in the target business data; if the degree of data anomaly represented by the data detection result is not greater than the preset degree threshold, determining that the data detection result represents no anomaly in the target business data.

[0090] The preset threshold can be an empirical value.

[0091] As an example, the above method further includes: if the data detection result indicates that the target business data is abnormal, updating the target business data and returning to the step of obtaining the first historical business data of the target business data source at a second time.

[0092] Furthermore, updating the target business data includes: identifying data in the target business data source that has not undergone anomaly detection as the updated target business data; or, identifying a business data source containing data that has not undergone anomaly detection as the updated business data source, and identifying data in the updated business data source that has not undergone anomaly detection as the updated target business data.

[0093] In this way, the updating of target business data can be automated, ensuring that all data in the target business data source that has not undergone anomaly detection can be detected; it can also ensure that all data in each business data source that has not undergone anomaly detection can be detected.

[0094] For example, the target business data is corrected according to the data source association weight to obtain corrected business data, including: correcting the target business data according to the data source association weight between the target business data source and each preset business data source and the second historical business data of each preset business data source to obtain corrected business data.

[0095] Understandably, business systems may be located in various places, and business data may be massive. To ensure the accuracy of the first integrated business data, the step of merging multiple first historical business data to obtain the first integrated business data is executed by the federated learning model.

[0096] As an example, after correcting the target business data according to the data source association weight and obtaining the corrected business data, the above method further includes: for each edge node, updating the node model deployed on the edge node based on the corrected business data to obtain updated model parameters; and updating the federated learning model according to the updated model parameters of each edge node.

[0097] In this way, after correcting the abnormal target business data, the node models deployed on each edge node are updated in a timely manner based on the corrected business data to obtain updated model parameters. Based on the updated model parameters, the federated learning model is updated, thereby improving the accuracy of the federated learning model output.

[0098] It is understandable that the correction process of the target business data may be accurate or inaccurate. Therefore, in order to ensure that the corrected business data can be traced and queried, as an example, the target business data is corrected according to the data source association weight. After obtaining the corrected business data, the above method also includes: fusing the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source to obtain blockchain information; and storing the blockchain information on the blockchain.

[0099] In this way, the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source used to record the correction process of the target business data are all used as blockchain information, which ensures that the information recording the correction process of the target business data is immutable, and also ensures the authenticity and security of the information recording the correction process of the target business data.

[0100] In the above-mentioned abnormal data correction method, based on the first historical business data, and considering the certain correlation between the first historical business data at the second time and the target business data at the first time, the difference between the first fused business data obtained by fusing the first historical business data and the target business data is used as the basis for anomaly detection of the target business data. This allows the detected data detection results to accurately capture business data that changes abnormally over time. When the data detection results indicate that the target business data is abnormal, considering the possible correlation between each preset business data source and the target business data source, the target business data is corrected based on the data source correlation weight between the target business data source and each preset business data source. This ensures that each preset business data source participates in the correction decision-making process for the target business data, improving the accuracy of abnormal data correction.

[0101] In one exemplary embodiment, such as Figure 3As shown, step 208, obtaining the data source association weights between the target business data source and each preset business data source, includes steps 302 to 306. Wherein:

[0102] Step 302: Obtain the second historical business data of each preset business data source.

[0103] In step 302, each preset business database is a business database other than the target business database; the second historical business data is generated at the third time, which can be a time period or a point in time. The third time is earlier than the first time (specifically, any point in the third time is earlier than any point in the first time). The third time can be the same as the second time or different from the second time, and there is no restriction here.

[0104] Step 304: Based on the second historical business data, perform causal verification between the target business data source and each preset business data source to obtain the causal verification results of each preset business data source relative to the target business data source.

[0105] For example, step 304 includes: for each preset business data source, fusing the first fused business data with the second historical business data to obtain the second fused business data; determining the distribution characteristic data between the first fused business data and the second fused business data, and determining the distribution characteristic data as the causal verification result of each preset business data source for the target business data source.

[0106] Optionally, the specific implementation of fusing the first fused business data with the second historical business data to obtain the second fused business data can refer to the above-mentioned specific implementation steps of summing and fusing the first fused business data, the second fused business data, the initial data and the allowable residual to obtain the first fused business data at the first time. At this time, the first fused business data is obtained by summing and fusing the first fused business data, the initial data and the allowable residual.

[0107] As one embodiment, determining the distribution characteristic data between the first converged service data and the second converged service data includes: generating an F statistic between the first converged service data and the second converged service data based on the allowable residuals corresponding to the first converged service data and the second converged service data, and determining the F statistic as the distribution characteristic data between the first converged service data and the second converged service data.

[0108] Thus, considering that there is a certain correlation between the preset business data source and the target business data source, there is a certain distribution relationship between the first and second fused business data obtained by the two fusion methods. Therefore, the F statistic between the first and second fused business data is used as the basis for judging the causal verification result, thereby realizing the indirect identification of the correlation between the preset business data source and the target business data source.

[0109] Step 306: Based on the causal verification results of each preset business data source for the target business data source, construct the data source association weights between the target business data source and each preset business data source.

[0110] For example, step 306 includes: filtering related business data sources from each preset business data source based on the causal verification results of each preset business data source for the target business data source; constructing the data source association weight between the target business data source and the related business data source; and determining the data source association weight between the target business data source and the related business data source as the data source association weight between the target business data source and each preset business data source.

[0111] As one embodiment, the process of filtering related business data sources from each preset business data source based on the causal verification results of each preset business data source relative to the target business data source includes: filtering related business data sources from each preset business data source whose causal verification results for the target business data source indicate a causal relationship.

[0112] Specifically, if the F statistic in the causal verification result of the preset business data source to the target business data source is greater than the preset statistic threshold, it can be determined that the causal verification result of the preset business data source to the target business data source indicates a causal relationship.

[0113] As one embodiment, constructing the data source association weight between the target business data source and the related business data source includes: obtaining the data source priority of the related business data source; determining the data correlation coefficient between the target business data source and the related business data source; and constructing the data source association weight between the target business data source and the related business data source based on the data source priority and the data correlation coefficient.

[0114] As one embodiment, the data source association weight between the target business data source and the related business data source is constructed based on the data source priority and the data correlation coefficient. This includes: obtaining a normalized coefficient, and then normalizing and summing the data source priority and the data correlation coefficient based on the normalized coefficient to obtain the data source association weight between the target business data source and the related business data source.

[0115] Optionally, based on the normalization coefficient, the data source priority and data correlation coefficient are normalized and summed to obtain the data source association weight between the target business data source and the related business data source, which can be expressed by the formula:

[0116]

[0117] in, The data source association weight between target business data source A and related business data source B. The normalization coefficient is... The correlation coefficient between target business data source A and related business data source B. The data source priority for associated business data source B.

[0118] Therefore, considering that the descriptive magnitudes of the data correlation coefficient and the data source priority may be different, the normalized coefficient is used to unify the descriptive magnitudes of the data correlation coefficient and the data source priority, thus ensuring the accuracy of the construction of the data source association weights.

[0119] In this embodiment, considering that there may be a causal relationship between each preset business data source and the target business data source, the causal verification process between the target and each preset business data source is implemented based on the second historical business data of each preset business data source. The data source association weights between the target business data source and each preset business data source are constructed based on the causal verification results of each preset business data source with respect to the target business data source. The causal relationship is directly used as the basis for constructing the data source association weights, so that the generated data source association weights take into account the possible causal relationship between each preset business data source and the target business data source, thereby improving the accuracy of the data source association weights.

[0120] In one exemplary embodiment, such as Figure 4 As shown, step 208, which involves correcting the target business data based on the data source association weight to obtain corrected business data, includes steps 402 to 406. Wherein:

[0121] Step 402: Based on the data source association weight, merge the second historical business data of the associated business data source to obtain the third merged business data.

[0122] For example, step 402 includes: multiplying and fusing the second historical business data of each associated business data source with the associated weight of the data source to obtain the associated business data of each associated business data source; and summing and fusing the associated business data of each associated business data source to obtain the third fused business data.

[0123] Step 404: Extract the change feature data of the target business data.

[0124] The change characteristic data in step 404 includes at least one of change trend data, seasonal change data, and random noise data.

[0125] Step 406: Based on the third fused business data and change characteristic data, the target business data is corrected to obtain the corrected business data.

[0126] For example, step 406 includes: fusing trend data, seasonal variation data and random noise data to obtain variation fused data; fusing variation fused data and third fused business data to obtain corrected business data.

[0127] Optionally, the step of extracting the change feature data of the target business data can be performed by a time series model. As an example, the change fusion data and the third fusion business data are fused to obtain the corrected business data, including: obtaining the model confidence level corresponding to the time series model; and fusing the change fusion data and the third fusion business data according to the model confidence level to obtain the corrected business data.

[0128] Optionally, based on the model confidence level, the changed fused data and the third-party fused business data are fused to obtain the corrected business data, which can be expressed by the formula:

[0129]

[0130] in, Correcting business data for target business data source A. For model confidence, To integrate data in response to changes, The data source association weight between the target business data source A and the related business data source k. This is the second historical business data of the associated business data source k.

[0131] Therefore, considering that the target business data may have trend characteristics, seasonal characteristics and random noise, the change fusion data obtained by fusing the trend data, seasonal data and random noise data, together with the third fused business data, is used as the basis for determining the corrected business data, thereby improving the accuracy of the corrected business data determination.

[0132] In this embodiment, by using the data source association weight as the basis for fusion of the second historical business data, the proportion of the second historical business data participating in the correction of the target business data corresponds to the data source association weight, thus ensuring the accuracy of the fusion of the third fused business data. Considering that the data changes of the target business data source have certain characteristics, the change characteristic data and the third fused business data are used together as the basis for the correction of the target business data, thereby improving the accuracy of abnormal data correction.

[0133] As a detailed embodiment, target business data from the target business data source and multiple first historical business data are obtained; the multiple first historical business data are merged to obtain first merged business data; based on the differences between the first merged business data and the target business data, anomaly detection is performed on the target business data to obtain data detection results.

[0134] Furthermore, when the data detection results indicate anomalies in the target business data, second historical business data from each preset business data source is acquired. For each preset business data source, the first fused business data and the second historical business data are fused to obtain second fused business data. The distribution characteristic data between the first fused business data and the second fused business data is determined, and this distribution characteristic data is used as the causal verification result of each preset business data source relative to the target business data source. Based on the causal verification result of each preset business data source relative to the target business data source, related business data sources are selected from each preset business data source. The data source association weights between the target business data source and related business data sources are constructed, and these data source association weights are used as the data source association weights between the target business data source and each preset business data source. Based on these data source association weights, the second historical business data from the related business data sources are fused to obtain third fused business data. Change characteristic data of the target business data is extracted. Based on the third fused business data and the change characteristic data, the target business data is corrected to obtain corrected business data.

[0135] Thus, by using the first historical business data as a basis, and considering the certain correlation between the first historical business data at the second time and the target business data at the first time, the difference between the first fused business data obtained by fusing the first historical business data and the target business data is used as the basis for anomaly detection of the target business data. This allows the detected data to accurately capture business data that changes abnormally over time. When the data detection results indicate that the target business data is abnormal, considering the possible correlation between each preset business data source and the target business data source, the target business data is corrected based on the data source correlation weight between the target business data source and each preset business data source. This ensures that each preset business data source participates in the correction decision-making process for the target business data, improving the accuracy of anomaly correction.

[0136] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0137] Based on the same inventive concept, this application also provides an anomaly data correction apparatus for implementing the aforementioned anomaly data correction method. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more anomaly data correction apparatus embodiments provided below can be found in the limitations of the anomaly data correction method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 5 As shown, an abnormal data correction device is provided, comprising: an acquisition module, a fusion module, a detection module, and a correction module, wherein:

[0139] The acquisition module is used to acquire the target business data from the target business data source, as well as multiple first historical business data.

[0140] The fusion module is used to merge multiple first historical business data to obtain the first fused business data.

[0141] The detection module is used to perform anomaly detection on the target business data based on the differences between the first fused business data and the target business data, and obtain the data detection results.

[0142] The correction module is used to obtain the data source association weights between the target business data source and each preset business data source when the data detection results indicate that the target business data is abnormal, and to correct the target business data according to the data source association weights to obtain corrected business data.

[0143] In one embodiment, the correction module is further configured to: obtain second historical business data of each preset business data source at a second time; perform causal verification between the target business data source and each preset business data source based on the second historical business data, and obtain the causal verification results of each preset business data source with respect to the target business data source; and construct the data source association weights between the target business data source and each preset business data source based on the causal verification results of each preset business data source with respect to the target business data source.

[0144] In one embodiment, the correction module is further configured to: for each preset business data source, merge the first fused business data with the second historical business data to obtain the second fused business data; determine the distribution characteristic data between the first fused business data and the second fused business data, and determine the distribution characteristic data as the causal verification result of each preset business data source relative to the target business data source.

[0145] In one embodiment, the correction module is further configured to: filter related business data sources from each preset business data source according to the causal verification results of each preset business data source for the target business data source; construct the data source association weight between the target business data source and the related business data source; and determine the data source association weight between the target business data source and the related business data source as the data source association weight between the target business data source and each preset business data source.

[0146] In one embodiment, the correction module is further configured to: merge the second historical business data of the associated business data source according to the data source association weight to obtain the third merged business data; extract the change feature data of the target business data; and correct the target business data according to the third merged business data and the change feature data to obtain the corrected business data.

[0147] In one embodiment, the step of fusing multiple first historical business data to obtain first fused business data is performed by the federated learning model; the apparatus further includes an update module, configured to: for each edge node, update the node model deployed on the edge node based on the corrected business data to obtain updated model parameters; and update the federated learning model according to the updated model parameters of each edge node.

[0148] In one embodiment, the target business data is corrected according to the data source association weight. After obtaining the corrected business data, the device further includes a storage module for: fusing the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source to obtain blockchain information; and storing the blockchain information on the blockchain.

[0149] Each module in the aforementioned abnormal data correction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0150] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an abnormal data correction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0151] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0153] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0154] Multiple sets of first-historical business data are merged to obtain the first merged business data;

[0155] Based on the differences between the first fused business data and the target business data, anomaly detection is performed on the target business data to obtain data detection results.

[0156] When the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0157] In one embodiment, when the processor executes the computer program, it further implements the following steps: obtaining second historical business data of each preset business data source; performing causal verification between the target business data source and each preset business data source based on the second historical business data, and obtaining the causal verification results of each preset business data source for the target business data source; and constructing data source association weights between the target business data source and each preset business data source based on the causal verification results of each preset business data source for the target business data source.

[0158] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each preset business data source, the first fused business data and the second historical business data are fused to obtain the second fused business data; the distribution characteristic data between the first fused business data and the second fused business data are determined, and the distribution characteristic data is determined as the causal verification result of each preset business data source relative to the target business data source.

[0159] In one embodiment, when the processor executes the computer program, it further performs the following steps: based on the causal verification results of each preset business data source for the target business data source, it filters related business data sources from each preset business data source; it constructs the data source association weights between the target business data source and the related business data sources, and determines the data source association weights between the target business data source and the related business data sources as the data source association weights between the target business data source and each preset business data source.

[0160] In one embodiment, when the processor executes the computer program, it further performs the following steps: merging the second historical business data of the associated business data source according to the data source association weight to obtain the third merged business data; extracting the change feature data of the target business data; and correcting the target business data according to the third merged business data and the change feature data to obtain corrected business data.

[0161] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each edge node, updates the node model deployed on the edge node based on the corrected business data to obtain updated model parameters; and updates the federated learning model according to the updated model parameters of each edge node.

[0162] In one embodiment, when the processor executes the computer program, it further performs the following steps: fusing the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source to obtain blockchain information; and storing the blockchain information on the blockchain.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0164] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0165] Multiple sets of first-historical business data are merged to obtain the first merged business data;

[0166] Based on the differences between the first fused business data and the target business data, anomaly detection is performed on the target business data to obtain data detection results.

[0167] When the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0168] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: obtaining second historical business data of each preset business data source; based on the second historical business data, performing causal verification between the target business data source and each preset business data source respectively, and obtaining the causal verification results of each preset business data source for the target business data source respectively; and constructing data source association weights between the target business data source and each preset business data source respectively based on the causal verification results of each preset business data source for the target business data source respectively.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each preset business data source, the first fused business data and the second historical business data are fused to obtain the second fused business data; the distribution characteristic data between the first fused business data and the second fused business data are determined, and the distribution characteristic data is determined as the causal verification result of each preset business data source relative to the target business data source.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: based on the causal verification results of each preset business data source for the target business data source, it filters related business data sources from each preset business data source; it constructs the data source association weights between the target business data source and the related business data sources, and determines the data source association weights between the target business data source and the related business data sources as the data source association weights between the target business data source and each preset business data source.

[0171] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: merging the second historical business data of the associated business data source according to the data source association weight to obtain the third merged business data; extracting the change feature data of the target business data; and correcting the target business data according to the third merged business data and the change feature data to obtain the corrected business data.

[0172] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each edge node, updating the node model deployed on the edge node based on the corrected business data to obtain updated model parameters; and updating the federated learning model according to the updated model parameters of each edge node.

[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: fusing the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source to obtain blockchain information; and storing the blockchain information on the blockchain.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0175] Obtain the target business data from the target business data source, as well as multiple first-hand historical business data.

[0176] Multiple sets of first-historical business data are merged to obtain the first merged business data;

[0177] Based on the differences between the first fused business data and the target business data, anomaly detection is performed on the target business data to obtain data detection results.

[0178] When the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

[0179] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: obtaining second historical business data of each preset business data source; based on the second historical business data, performing causal verification between the target business data source and each preset business data source respectively, and obtaining the causal verification results of each preset business data source for the target business data source respectively; and constructing data source association weights between the target business data source and each preset business data source respectively based on the causal verification results of each preset business data source for the target business data source respectively.

[0180] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each preset business data source, the first fused business data and the second historical business data are fused to obtain the second fused business data; the distribution characteristic data between the first fused business data and the second fused business data are determined, and the distribution characteristic data is determined as the causal verification result of each preset business data source relative to the target business data source.

[0181] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: based on the causal verification results of each preset business data source for the target business data source, it filters related business data sources from each preset business data source; it constructs the data source association weights between the target business data source and the related business data sources, and determines the data source association weights between the target business data source and the related business data sources as the data source association weights between the target business data source and each preset business data source.

[0182] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: merging the second historical business data of the associated business data source according to the data source association weight to obtain the third merged business data; extracting the change feature data of the target business data; and correcting the target business data according to the third merged business data and the change feature data to obtain the corrected business data.

[0183] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each edge node, updating the node model deployed on the edge node based on the corrected business data to obtain updated model parameters; and updating the federated learning model according to the updated model parameters of each edge node.

[0184] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: fusing the target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source to obtain blockchain information; and storing the blockchain information on the blockchain.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for correcting abnormal data, characterized in that, The method includes: Obtain the target business data from the target business data source, as well as multiple first-hand historical business data. The multiple first historical business data are merged to obtain the first merged business data; Based on the differences between the first fused service data and the target service data, anomaly detection is performed on the target service data to obtain data detection results. If the data detection results indicate that the target business data is abnormal, the data source association weights between the target business data source and each preset business data source are obtained, and the target business data is corrected according to the data source association weights to obtain corrected business data.

2. The method according to claim 1, characterized in that, The step of obtaining the data source association weights between the target business data source and each preset business data source includes: Obtain the second historical business data from each preset business data source; Based on the second historical business data, causal verification is performed between the target business data source and each preset business data source to obtain the causal verification results of each preset business data source for the target business data source. Based on the causal verification results of each preset business data source for the target business data source, data source association weights are constructed between the target business data source and each preset business data source.

3. The method according to claim 2, characterized in that, The step of performing causality checks on the target business data source and each of the preset business data sources based on the second historical business data, to obtain the causality check results of each preset business data source relative to the target business data source, includes: For each of the preset business data sources, the first fused business data and the second historical business data are fused to obtain the second fused business data; Determine the distribution characteristic data between the first fused business data and the second fused business data, and determine the distribution characteristic data as the causal verification results of each preset business data source for the target business data source.

4. The method according to claim 2, characterized in that, The step of constructing data source association weights between the target business data source and each preset business data source based on the causal verification results of each preset business data source for the target business data source includes: Based on the causal verification results of each preset business data source for the target business data source, related business data sources are selected from each preset business data source; Construct the data source association weight between the target business data source and the associated business data source, and determine the data source association weight between the target business data source and the associated business data source as the data source association weight between the target business data source and each preset business data source respectively.

5. The method according to claim 4, characterized in that, The step of correcting the target business data according to the association weight of the data source to obtain corrected business data includes: Based on the data source association weight, the second historical business data of the associated business data source are merged to obtain the third merged business data; Extract the change feature data of the target business data; Based on the third fused service data and the change characteristic data, the target service data is corrected to obtain corrected service data.

6. The method according to claim 1, characterized in that, The step of fusing the multiple first historical business data to obtain first fused business data is executed by a federated learning model; after correcting the target business data according to the data source association weights to obtain corrected business data, the method further includes: For each edge node, the node model deployed on the edge node is updated based on the corrected business data to obtain updated model parameters; The federated learning model is updated based on the updated model parameters of each edge node.

7. The method according to claim 4, characterized in that, After correcting the target business data according to the association weight of the data source to obtain corrected business data, the method further includes: The target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source are fused together to obtain blockchain information; The blockchain information is stored in the blockchain.

8. An abnormal data correction device, characterized in that, The device includes: The acquisition module is used to acquire the target business data from the target business data source and multiple first historical business data. The fusion module is used to fuse the multiple first historical business data to obtain first fused business data; The detection module is used to perform anomaly detection on the target business data based on the difference between the first fused business data and the target business data, and obtain data detection results; The correction module is used to obtain the data source association weights between the target business data source and each preset business data source when the data detection result indicates that the target business data is abnormal, and to correct the target business data according to the data source association weights to obtain corrected business data.

9. The apparatus according to claim 8, characterized in that, The correction module is also used for: Obtain the second historical business data from each preset business data source; Based on the second historical business data, causal verification is performed between the target business data source and each preset business data source to obtain the causal verification results of each preset business data source for the target business data source. Based on the causal verification results of each preset business data source for the target business data source, data source association weights are constructed between the target business data source and each preset business data source.

10. The apparatus according to claim 9, characterized in that, The correction module is also used for: For each of the preset business data sources, the first fused business data and the second historical business data are fused to obtain the second fused business data; Determine the distribution characteristic data between the first fused business data and the second fused business data, and determine the distribution characteristic data as the causal verification results of each preset business data source for the target business data source.

11. The apparatus according to claim 9, characterized in that, The correction module is also used for: Based on the causal verification results of each preset business data source for the target business data source, related business data sources are selected from each preset business data source; Construct the data source association weight between the target business data source and the associated business data source, and determine the data source association weight between the target business data source and the associated business data source as the data source association weight between the target business data source and each preset business data source respectively.

12. The apparatus according to claim 11, characterized in that, The correction module is also used for: Based on the data source association weight, the second historical business data of the associated business data source are merged to obtain the third merged business data; Extract the change feature data of the target business data; Based on the third fused service data and the change characteristic data, the target service data is corrected to obtain corrected service data.

13. The apparatus according to claim 8, characterized in that, The step of fusing the plurality of first historical business data to obtain first fused business data is executed by a federated learning model; the device further includes an update module, used for: For each edge node, the node model deployed on the edge node is updated based on the corrected business data to obtain updated model parameters; The federated learning model is updated based on the updated model parameters of each edge node.

14. The apparatus according to claim 11, characterized in that, After correcting the target business data according to the association weight of the data source to obtain corrected business data, the device further includes a storage module for: The target business data, the corrected business data, the data source association weight, and the data source identifier of the associated business data source are fused together to obtain blockchain information; The blockchain information is stored in the blockchain.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images