Data reconciliation method and device based on anomaly detection

Abstract

The embodiment of the invention discloses a data reconciliation method and device based on anomaly detection, belongs to the technical field of intelligent reconciliation, and can improve the efficiency and accuracy of data reconciliation. Comprising the steps of obtaining source data of right and interest data to be reconciliated, and calculating a reconciliation result corresponding to the right and interest data according to a preset reconciliation algorithm; inputting the reconciliation result into a pre-trained anomaly detection model, and determining an abnormal value of the reconciliation result according to an expected value of a path length of a first isolated forest in the anomaly detection model and reconciliation factors of the reconciliation result; and detecting an account checking result based on the abnormal value, when the account checking result is abnormal, detecting source data of the right and interest data by using a pre-trained abnormal detection model to obtain a detection result, and performing right and interest data account checking based on the detection result, the detection result comprising normal information or error information of the source data.

Description

Technical Field

[0001] The present application relates to the field of intelligent reconciliation technology, and in particular to a data reconciliation method and device based on anomaly detection. Background Art

[0002] With the increasing maturity of Internet technology, people's daily lives are inseparable from the transactions of virtual goods. If the equity data is in the billions, then enterprise-level reconciliation and settlement will need to manage diverse order transactions and payment methods to cope with large-scale and high-frequency transactions.

[0003] In the related technology, a full-link reconciliation method based on a distributed identity identification number (IDentity, ID) is usually adopted. Although a distributed ID is used throughout the entire link, when dealing with complex contracts and pricing models and different data sources, the corresponding reconciliation algorithm is too complicated, and the location of data reconciliation problems becomes difficult, which affects the reconciliation results. It is impossible to accurately judge whether the results of the data reconciliation are correct, resulting in low efficiency and accuracy of data reconciliation. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data reconciliation method, device, electronic device and storage medium based on anomaly detection, so as to solve the problem of low efficiency and accuracy of data reconciliation.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides a data reconciliation method based on anomaly detection, comprising: obtaining source data of equity data to be reconciled, and calculating a reconciliation result corresponding to the equity data according to a preset reconciliation algorithm; inputting the reconciliation result into a pre-trained anomaly detection model, and determining an anomaly value of the reconciliation result according to an expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and a reconciliation factor; based on the anomaly value, detecting the reconciliation result, and when the reconciliation result is abnormal, using the pre-trained anomaly detection model to detect the source data of the equity data to obtain a detection result, and reconciling the equity data based on the detection result, wherein the detection result includes normal information or error information of the source data.

[0006] In a second aspect, an embodiment of the present application provides a data reconciliation device based on anomaly detection, comprising: a metadata management module, used to obtain source data of equity data to be reconciled, and calculate the reconciliation result corresponding to the equity data according to a preset reconciliation algorithm; an anomaly detection module, used to input the reconciliation result into a pre-trained anomaly detection model, and determine the anomaly value of the reconciliation result according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; a detection module, used to detect the reconciliation result based on the anomaly value, and when the reconciliation result is abnormal, use the pre-trained anomaly detection model to detect the source data of the equity data to obtain a detection result, and reconcile the equity data based on the detection result, and the detection result includes normal information or error information of the source data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned data reconciliation method based on anomaly detection.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned data reconciliation method based on anomaly detection.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above-mentioned data reconciliation method based on anomaly detection.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the above-mentioned data reconciliation method based on anomaly detection.

[0011] The technical solution of the embodiment of the present application is adopted, by obtaining the source data of the equity data to be reconciled, and according to the preset reconciliation algorithm, the reconciliation result corresponding to the equity data is calculated; the reconciliation result is input into the pre-trained anomaly detection model, and the anomaly value of the reconciliation result is determined according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; based on the anomaly value, the reconciliation result is detected, and when the reconciliation result is abnormal, the source data of the equity data is detected by using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data. It can be seen that through the pre-trained anomaly detection model, the anomaly value of the reconciliation result is calculated by using the expected value of the path length of the first isolated forest of the reconciliation result and the reconciliation factor, and when the reconciliation result is detected to be abnormal, the source data of the equity data is detected again, and multiple detections can accurately locate the abnormal equity data, so that the reconciliation results of a large amount of equity data are more accurate, solving the problem of low efficiency and accuracy of data reconciliation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 It is a structural diagram of relevant modules of a data reconciliation method based on anomaly detection according to an embodiment of the present application; Figure 2 is a schematic flow chart of a data reconciliation method based on anomaly detection according to an embodiment of the present application; Figure 3 is a structural diagram of an abnormality detection module according to an embodiment of the present application; Figure 4 is a reconciliation flow chart of a data reconciliation method based on anomaly detection according to an embodiment of the present application; Figure 5 is a schematic flow chart of a data reconciliation method based on anomaly detection according to another embodiment of the present application; Figure 6 is a schematic block diagram of a data reconciliation device based on anomaly detection according to an embodiment of the present application; Figure 7 It is a schematic diagram of the hardware structure of a data reconciliation device based on anomaly detection according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present application provide a data reconciliation method and device based on anomaly detection to solve the problem of low efficiency and accuracy of data reconciliation.

[0015] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0016] The data reconciliation method based on anomaly detection provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0017] Figure 1 1 is a schematic diagram of the structure of the modules of a data reconciliation method based on anomaly detection according to an embodiment of the present application. Figure 1 As shown, the modules related to the method include a data entry module, a metadata management module, a data reconciliation module and an anomaly detection module. Specifically, the data entry module includes a business data source, i.e., data to be reconciled, such as file data, relational data and non-relational data, which is used to input the verified equity data into the data lake through data integration, and store it in the data lake in the form of source data; the metadata management module is used to process the equity data input into the data lake by the data entry module, and establish a unified data directory by generating metadata information describing the source data, which is used to find the source data corresponding to the equity data in the data lake, and can manage the data analyzed by other modules; the data reconciliation module is used to directly obtain the source data by querying the metadata information, calculate and analyze the source data corresponding to the equity data, and generate the corresponding reconciliation results by executing the reconciliation tasks in the reconciliation algorithm; the anomaly detection module is used to detect the input reconciliation results or source data through the anomaly detection model, and store the detection results in the data lake.

[0018] In the following, in conjunction with the accompanying drawings, a data reconciliation method based on anomaly detection provided by an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0019] Figure 2A schematic flow chart of a data reconciliation method based on anomaly detection provided by an embodiment of the present invention is shown, and the method comprises the following steps: S202, obtaining source data of the equity data to be reconciled, and calculating the reconciliation result corresponding to the equity data according to a preset reconciliation algorithm.

[0020] Equity data refers to the specific numerical value or quantitative indicator of a certain right or interest, for example, some data used to measure the equity of corporate owners.

[0021] The reconciliation algorithm includes algorithms written in a variety of programming languages ​​and database query languages. Specifically, according to the customized reconciliation configuration information, the reconciliation rules matching the equity data business scenario are matched, and the source data of the equity data to be reconciled is processed and analyzed using the data query and analysis engine, such as obtaining the source data of the equity data, and using the source data to perform the reconciliation tasks in the reconciliation algorithm. Based on the reconciliation algorithm, the reconciliation result is calculated. The reconciliation result includes normal data or abnormal data related to the reconciliation status.

[0022] S204, inputting the reconciliation result into the pre-trained anomaly detection model, and determining the anomaly value of the reconciliation result according to the expected value of the path length of the first isolation forest of the anomaly detection model and the reconciliation factor of the reconciliation result.

[0023] The anomaly detection model is an improved model based on the isolation forest algorithm, which is used to automatically identify anomalies in the reconciliation results, such as calculating the outlier values ​​of the reconciliation results based on the first isolation forest.

[0024] Specifically, the reconciliation factor includes at least one of the following factors: the order status of the reconciliation party, the reconciliation result status, the time factor, and the order quantity. The calculation formula of the abnormal value of the reconciliation result can be expressed as Formula 1: (1) in, Indicates abnormal values ​​of the reconciliation results. Represents the path length of the first isolated forest in the anomaly detection model for the reconciliation result. It represents the expected value of the path length of the first isolated forest in the anomaly detection model of the reconciliation result. Refers to the influencing factors, i.e. the reconciliation factors. Represents a multiplication operation. In the reconciliation result data, the specific calculation formula is as follows: (2) in, Refers to the order status of the account holder of the equity data, that is, the quantitative factor of the account holder. Refers to the reconciliation result status of equity data, that is, the quantitative factor of the reconciliation result status. It refers to the time factor, that is, the time quantification factor. Combined with the actual usage of users, for the situation where the amount of equity data orders is large at the beginning of the month, the time factor can be set to a smaller value to reduce the impact on abnormal values. Indicates the order quantity, i.e. the order quantity of the equity data of a certain reconciliation strategy. Represents a multiplication operation.

[0025] S206, based on the outliers, detect the reconciliation results. When there are anomalies in the reconciliation results, use the pre-trained anomaly detection model to detect the source data of the equity data, obtain the detection results, and reconcile the equity data based on the detection results.

[0026] The detection result includes normal information or error information of the source data.

[0027] According to the abnormal value obtained in S204, the reconciliation result is detected, including: by calculating the average value and standard deviation of the abnormal value of the historical data, the threshold is preset by adding or subtracting the multiple of the standard deviation of the average value. Generally, 2 times of the standard deviation is used as the preset threshold. It can also be adjusted accordingly according to the actual situation of user feedback. When the abnormal value is greater than the preset threshold, the reconciliation result is determined to be abnormal.

[0028] When an abnormality appears in the reconciliation result, in order to more rigorously determine the problematic data, the source data corresponding to the equity data with abnormal reconciliation results is input into the anomaly detection model, and the source data is detected using the anomaly detection model to obtain the detection result. The equity data is reconciled based on the detection result, that is, the location of the problem in the equity data is determined. Among them, the abnormal source data may be missing key data in the source data, the amount is negative, the key field value is abnormal, etc., which requires further coordination and processing by the data reconciliation party.

[0029] By adopting the technical solution of the embodiment of the present application, the source data of the equity data to be reconciled is obtained, and the reconciliation result corresponding to the equity data is calculated according to the preset reconciliation algorithm; the reconciliation result is input into the pre-trained anomaly detection model, and the anomaly value of the reconciliation result is determined according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; wherein the reconciliation factor includes at least one of the following factors: the order status of the reconciliation party, the reconciliation result status, the time factor, and the order quantity; based on the anomaly value, the reconciliation result is detected, and when the reconciliation result is abnormal, the source data of the equity data is detected using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data. It can be seen that through the pre-trained anomaly detection model, the expected value of the path length of the first isolated forest of the reconciliation result and the reconciliation factor are used to calculate the anomaly of the reconciliation result. When the reconciliation result is detected to be abnormal, the source data of the equity data is tested again. Multiple tests can accurately locate the abnormal equity data, making the reconciliation results of a large amount of equity data more accurate, and solving the problem of low efficiency and accuracy of data reconciliation.

[0030] In one embodiment, the following step A may also be performed: Step A: When there is no abnormality in the reconciliation result, the pre-trained anomaly detection model is used to output normal reconciliation information, and the equity data is reconciled based on the normal reconciliation information.

[0031] The reconciliation results may include the data of the reconciling party and the data of the beneficiary, as well as the reconciliation status after the reconciliation.

[0032] The reconciliation results are input into the pre-trained anomaly detection model. After the first isolated forest calculation, the normal reconciliation information is output. That is, all the data in the reconciliation results are tested. If there are no anomalies in the reconciliation results, the equity data is reconciled according to the normal reconciliation information.

[0033] In this embodiment, when there is no anomaly in the reconciliation result, after the reconciliation result is input into the pre-trained anomaly detection model, normal reconciliation information is output, and equity data reconciliation is performed based on the normal reconciliation information. There is no need to reconcile the source data again, and the equity data reconciliation can be performed directly based on the reconciliation result.

[0034] In one embodiment, the source data of the equity data to be reconciled is obtained (ie, S202), and the following step B may be performed: Step B: Input the equity data into the data lake to generate source data of the equity data.

[0035] The source data includes metadata information corresponding to the equity data.

[0036] A data lake is a storage system used to store large amounts of raw data. It can accommodate various types of data, store large-scale data, and support access and use of multiple data processing and analysis tools.

[0037] Integrate different types of raw data from data sources within the enterprise or across enterprises, input them into the data lake as equity data, generate source data in the data lake, and use metadata management tools to create metadata information for the source data in the data lake to accurately describe the specific location and naming of the source data of the equity data in the data lake.

[0038] In this embodiment, the equity data is input into the data lake, and large-scale massive equity data is effectively stored and managed. The equity data can be stored more conveniently and rationally without the need for cumbersome data integration. Since the source data of the equity data is stored in the data lake and metadata information is generated, the source data in the data lake can be accurately located and searched according to the metadata information, thereby improving the efficiency of searching for the target source data.

[0039] In one embodiment, the equity data is input into the data lake (i.e., step B), and the following steps C1-C2 may be performed: Step C1, using full extraction and incremental real-time extraction methods to collect relational data and / or non-relational data in the data to be reconciled.

[0040] For non-relational data, such as reconciliation data in the data to be reconciled that is settled on a daily, monthly, or quarterly basis, batch collection or full extraction is used. For example, the data to be reconciled can be extracted from the source system through customized implementation of extraction, transformation, and loading (ETL) operations.

[0041] For relational data, such as near-real-time relational data such as contracts, partners, products, and users, full extraction and incremental real-time extraction are used for collection. When acquiring relational data for the first time, full data extraction is used. When synchronizing the relational data subsequently, the log changes of the database are tracked first, and then the relational data is written and modified through real-time analysis. Among them, incremental real-time extraction refers to obtaining only the changed data from the data source, rather than extracting all the data from the entire data source like full extraction.

[0042] Step C2: Verify the relational data and / or non-relational data, and input the verified data into the data lake as equity data.

[0043] For file data, the Message-Digest Algorithm 5 (MD5) algorithm is used to verify the file data to ensure the integrity of the file data and prevent tampering. If the MD5 value is consistent, the verification passes and the verified file data is input into the data lake as equity data.

[0044] For relational data or non-relational data, first verify the data mode, quantity and specific data content to prevent the data from missing necessary information. If the verification passes, the verified data will be input into the data lake as equity data.

[0045] In this embodiment, the data to be reconciled is collected by full extraction and incremental real-time extraction, and the data to be reconciled is verified and processed. The data after verification is input into the data lake as equity data, which can keep the source data of the equity data in the data entering the lake intact and consistent with the data to be reconciled before entering the lake, thereby improving the integrity and consistency of the data entering the lake.

[0046] In one embodiment, the source data of the equity data is generated (i.e., step B), and the following steps D1-D4 may be specifically performed: Step D1: Use metadata management tools to automatically extract structured data from equity data in the data lake.

[0047] After the equity data is input into the data lake in step B, different methods are used to construct the information of the source data in the data lake for different types of data.

[0048] ‌‌Structured data refers to data with fixed formats and rules, usually presented in the form of tables, tree structures or relational models. Among them, metadata management tools are used to automatically extract structured data from equity data. Specifically, metadata management tools include streaming data processing platform DataHub metadata management tools. Structured data refers to structured relational data or non-relational data. The information of structured data is collected through metadata management tools, and this information can be used as metadata information of the corresponding structured data.

[0049] Step D2, using preset rules or statistical analysis methods to extract semi-structured data from the equity data in the data lake.

[0050] Semi-structured data refers to file data that has a certain structure, but the structure is not fixed, regular or implicit. It is preferred to use predefined rules or statistical analysis methods to parse and obtain the metadata information of the file. For example, the file data includes comma-separated values ​​(CSV) files, etc. The parsing rules defined in the CSV file are used to parse the data of the CSV file to obtain the structured data information of the file, which can be used as the metadata information of the corresponding semi-structured data.

[0051] Step D3: Determine the metadata information corresponding to the equity data in the data lake based on the structured data and semi-structured data.

[0052] Among them, metadata information is used to locate and search for source data corresponding to equity data in the data lake.

[0053] According to step D1 and step D2, structured data and semi-structured data are obtained, and the extracted structured data information and semi-structured data information are obtained, and the extracted information is used as metadata information of the corresponding equity data. A unified data directory is generated by labeling and classifying metadata information to facilitate low-latency metadata search, and can support full-text search, ranking and classification based on metadata attributes.

[0054] Specifically, metadata information includes: data storage size, storage location information, total data volume, data generation time, data last update time, data access times, data structure information, data content characteristics, data version information, and label identification, etc., which are used to quickly and accurately locate the source data of different types of equity data in the data lake.

[0055] Step D4, determining the source data of the equity data according to the metadata information.

[0056] In this embodiment, by acquiring the structured data and board structured data of the equity data in the data lake, the metadata information of the equity data in the data lake is determined, and based on the metadata information, the source data corresponding to the equity data is determined, so that the position and size of the source data of the equity data in the data lake and other information can be accurately located.

[0057] In one embodiment, according to a preset reconciliation algorithm, the reconciliation result corresponding to the equity data is calculated (ie, S202), and the following steps E1-E3 may be performed: Step E1, determine the reconciliation algorithm based on the business scenario corresponding to the equity data.

[0058] Different equity data corresponds to different business scenarios, and different reconciliation algorithms are used according to different business scenarios. Based on the ability to match different business scenarios with equity data, the specific matching process includes: data reconciliation party, reconciliation mode, reconciliation execution efficiency, reconciliation matching rules, and reconciliation settlement strategy. Among them, the data reconciliation party refers to the party responsible for sending and receiving reconciliation information during the transaction process, such as the reconciliation information of the equity data of the data reconciliation party and the reconciliation information of the equity data party corresponding to the target business, or when checking whether the balance of each account in the general ledger is equal to the sum of the balances of each account in the detailed ledger, all data are used as data reconciliation parties. The reconciliation mode can be one-to-one or one-to-many, such as one business corresponding to multiple data reconciliation parties. The reconciliation execution frequency can be performed by time, day, month, quarter, or year, or a specific execution time and execution frequency can be specified to clarify the execution time of the reconciliation task or the expected generation time of the reconciliation result. The matching rules for reconciliation include, but are not limited to, determining which data of the equity data party to match and reconcile, which field data information the reconciliation results should contain, how to store it, how long it is stored, and whether to provide a download method, etc. The reconciliation and settlement strategy involves the status information of the equity data order, which includes success, failure, pending activation, activation, and other situations. For the normal or abnormal reconciliation status, when there is a difference in the settlement, it is necessary to determine whether an adjustment is required, and whether the adjustment method is manual or automatic, and whether manual review is required.

[0059] According to the business scenarios defined by the above matching, the corresponding reconciliation and settlement algorithms are written. The reconciliation algorithms generate reconciliation tasks that match the business scenarios through the supported multiple programming languages ​​and database query languages. The specific matching process of matching or defining business scenarios is not limited and can be adjusted according to actual business conditions.

[0060] Step E2: According to the reconciliation algorithm, find the metadata information corresponding to the equity data in the data lake and determine the reconciliation task.

[0061] According to the reconciliation algorithm, by directly reading the source data from the data lake, or according to the metadata information corresponding to the equity data in the data lake, finding the source data corresponding to the equity data in the data lake, determining the required equity data, generating the corresponding reconciliation task, and using the cluster computing framework for big data processing to execute the reconciliation task, among which the distributed computing framework Spark execution engine is preferentially used.

[0062] The reconciliation task may include obtaining target data from the data reconciliation party and performing data matching and comparison.

[0063] Step E3, obtaining the reconciliation result by executing the reconciliation task.

[0064] The reconciliation result includes the success data or failure data of executing the reconciliation task.

[0065] Executing a reconciliation task includes executing the reconciliation task determined according to the reconciliation algorithm, forming a corresponding reconciliation execution plan during the execution of the reconciliation task, and using the reconciliation execution plan to view the specific situation of the execution process of the reconciliation task. For example, when the reconciliation task fails to execute, the reconciliation execution plan can be used to view the reason for the failure of the reconciliation task, determine the specific location of the problem in the execution of the reconciliation task, and can trigger notification information (including email, SMS, phone call, etc.) to prompt the user.

[0066] The execution result of the reconciliation task is determined as the reconciliation result, including the success data or failure data in the process of executing the reconciliation task in the reconciliation execution plan.

[0067] It should be noted that the reconciliation results are stored and the corresponding permissions are set, and the corresponding modification records are saved. For example, the reconciliation results are stored according to the configured reconciliation result storage method, and the reconciliation results include data in the process of executing the reconciliation task. If each reconciliation result has a corresponding audit rule, when modifying the data corresponding to the reconciliation result, it is necessary to comply with the corresponding permissions and record the modifier information and modification records.

[0068] In this embodiment, the reconciliation algorithm is determined by the business scenario, and the source data required in the reconciliation algorithm is searched from the data lake according to the metadata information of the equity data according to the reconciliation algorithm, and the reconciliation task is determined. By executing the reconciliation task, the successful data or failed data related to the execution of the reconciliation task is determined as the reconciliation result. The reconciliation task can be automatically executed and the reconciliation result can be determined according to the algorithm, and the reconciliation task corresponding to the business scenario of the customized equity data can be efficiently executed, thereby improving the efficiency and timeliness of the reconciliation of massive equity data.

[0069] In one embodiment, the source data of the equity data is detected using a pre-trained anomaly detection model to obtain a detection result (ie, S206), and the following steps F1-F2 may be performed: Step F1, according to the reconciliation result, find the metadata information corresponding to the reconciliation result from the data lake, determine the source data of the equity data according to the metadata information, input the source data into the second isolation forest of the anomaly detection model, and calculate the anomaly score.

[0070] When the reconciliation result is abnormal, the source data of the equity data corresponding to the reconciliation result in the data lake is determined by searching the metadata information corresponding to the reconciliation result according to the reconciliation result, and the source data is input into the second isolated forest of the anomaly detection model. The algorithm corresponding to the improved second isolated forest is used to calculate the anomaly score of the source data of the equity data corresponding to the abnormal reconciliation result. The anomalies of the source data include but are not limited to: abnormal mobile phone number value, abnormal settlement amount value, and abnormal values ​​of certain specific fields.

[0071] Step F2, determining the detection result according to the abnormality score.

[0072] Among them, the detection results include whether there are any abnormalities in the equity data.

[0073] The anomaly score is mapped between 0 and 1 in a normalized manner. When the anomaly score is close to 1, the detection result is that the equity data corresponding to the source data is abnormal. When the anomaly scores are close to 0.5, it can be considered that the detection result is that the equity data corresponding to the source data is normal.

[0074] In this embodiment, the source data of the equity data corresponding to the reconciliation result is confirmed from the data lake, the source data is input into the second isolated forest of the anomaly detection model, the anomaly score is calculated, and based on the anomaly score, whether the equity data is abnormal is detected. When the reconciliation result of the equity data is detected to be abnormal, it is possible to check whether the corresponding source data is abnormal, and potential problems in the source data can be discovered in a timely manner through the algorithm of the second isolated forest of the anomaly detection model.

[0075] In one embodiment, the source data is input into the second isolation forest of the anomaly detection model, and the anomaly score is calculated (ie, step F1), and specifically, the following steps G1-G2 may be performed: Step G1, input the source data into the second isolation forest of the anomaly detection model, construct a binary tree corresponding to the source data based on the source data, and determine the average path length of the binary tree corresponding to the source data.

[0076] The binary tree constructed based on the source data is a binary tree constructed based on the source data of all equity data in the second isolation forest. The average path length of the binary tree is calculated using the formulas shown in Formula 3 and Formula 4: (3) (4) in, It refers to the average path length of the binary tree constructed by the source data corresponding to n equity data. refers to a harmonic number, where represents Euler's constant, which is approximately 0.5772156649. represents the logarithm, Represents a multiplication operation.

[0077] Step G2, determining an anomaly score based on the average path length and the expected value of the path length of the source data in the second isolation forest.

[0078] Specifically, the average path length refers to the binary tree constructed based on the source data of all equity data in step G1, and the average path length of the binary tree is calculated. The expected value of the path length in the second isolated forest refers to the expected value of the path length of the binary tree of the source data in the second isolated forest. The conventional normalized anomaly score calculation method is used, as shown in Formula 5: (5) in, Indicates abnormal score; It represents the expected value of the path length of the source data in the second isolation forest; Refers to the average path length calculated in step G1.

[0079] In this embodiment, the source data is input into the second isolation forest of the anomaly detection model, and according to the algorithm corresponding to the second isolation forest, the average path length of the binary tree corresponding to the source data and the path length of the source data in the second isolation forest are determined, and then the anomaly score of the source data is determined. In this way, it is determined whether there is a problem with the source data based on the anomaly score.

[0080] In one embodiment, the method further includes training an anomaly detection model, which may specifically be performed in the following steps H1-H5: Step H1, obtain multiple sample equity data, sample source data corresponding to the sample equity data and sample reconciliation results, and input the sample reconciliation results into the anomaly detection model.

[0081] The sample source data and sample reconciliation results corresponding to the sample equity data are used as training sets, and multiple sample equity data are used as validation sets to train the anomaly detection model. The sample reconciliation results are input into the anomaly detection model.

[0082] Step H2, extracting sample feature data from the sample reconciliation result, constructing a binary tree based on the feature data, and determining the first isolation forest of the sample based on the binary tree constructed with the feature data.

[0083] First, a tree is constructed. Specifically, some data is randomly subsampled from the original data, i.e., the sample reconciliation results, as the training set of the first isolated tree. The sample reconciliation results are preprocessed, such as sample reconciliation result cleaning, normalization, and sample feature extraction. Sample feature data is randomly extracted from the sample reconciliation results, and the sample reconciliation results are binary split to construct a binary tree. The tree is stopped when each sample feature data in the data set of the sample reconciliation results becomes a leaf node or reaches the maximum depth of the tree set in advance. The sample feature data may include factors such as the order status and order quantity of the reconciliation party corresponding to the sample reconciliation results.

[0084] By repeating the above steps, multiple isolated trees are recursively constructed to form an isolation forest as the first isolation forest.

[0085] Step H3, based on the sample first isolation forest, determine the expected value of the path length of the sample reconciliation result, and determine the sample outlier of the sample reconciliation result according to the expected value of the path length of the sample reconciliation result and the sample reconciliation factor.

[0086] Among them, the reconciliation factors include at least one of the following factors: the order status of the sample reconciliation party, the sample reconciliation result status, the sample time factor, and the sample order quantity.

[0087] The path length of the sample reconciliation result refers to the path length (the number of branches passed) from the root node to its leaf node in the constructed random tree. Generally speaking, the shorter the path length of the sample reconciliation result, the greater the possibility that it is an outlier.

[0088] The reconciliation factors include: inputting the sample reconciliation results of the sample equity data into the anomaly detection model for detection, and are also related to the reconciliation settlement strategy, that is, the strategy applied in the reconciliation algorithm. The reconciliation settlement strategy includes the order status of the sample reconciliation party of the sample equity data, the status of the sample reconciliation results, and the number of sample orders. The relevant factors are determined as reconciliation factors. For example, in order to identify the situations of normal result status with a large number of orders, abnormal reconciliation status with a small number of orders, and a large number of orders, it is necessary to quantify the factors related to the reconciliation settlement strategy; among which, the order status of the sample equity data reconciliation party includes: success, failure, pending activation, activation, and others, which can be represented by a predefined value, such as success is 1, others is 2, pending activation is 3, activation is 4, and failure is 5. The specific value can be adjusted according to the actual distribution of the order status; the sample reconciliation result status includes: normal and abnormal, and a binary variable can be set, that is, 1 is normal and 0 is abnormal.

[0089] Based on the constructed first isolation forest, the expression for calculating the sample outlier value of the sample reconciliation result is as shown in Formula 6 and Formula 7: (6) in, , represents the abnormal value of the sample reconciliation result, Indicates the path length of the first isolated forest in the anomaly detection model for the sample reconciliation result. It represents the expected value of the path length of the first isolated forest in the anomaly detection model of the sample reconciliation result. Refers to the influencing factors, i.e. the reconciliation factors. Represents a multiplication operation. In the sample reconciliation result data, the specific calculation formula is as follows: (7) in, Refers to the order status of the account reconciliation party of the equity data, that is, the quantitative factor of the sample account reconciliation party. Refers to the reconciliation result status of equity data, i.e., the quantitative factor of the sample reconciliation result status. It refers to the sample time factor, that is, the time quantification factor. Combined with the actual usage of users, for the situation where the amount of equity data orders is large at the beginning of the month, the time factor can be set to a smaller value to reduce the impact on outliers. Indicates the number of sample orders, i.e. the number of orders for the equity data of a certain reconciliation strategy. Represents a multiplication operation.

[0090] Step H4, when the sample outlier value is greater than the preset threshold, the sample reconciliation result is abnormal, the sample source data corresponding to the abnormal sample reconciliation result is input into the sample second isolation forest, and the sample anomaly score is calculated.

[0091] The preset thresholds include: calculating the mean and standard deviation of the outliers in the historical data, and using the mean plus or minus multiples of the standard deviation to preset the threshold. Generally, 2 times the standard deviation is used as the preset threshold, and it can also be adjusted accordingly based on the actual situation of user feedback. If the sample outlier value is greater than the preset threshold, the corresponding sample reconciliation result will be determined as abnormal.

[0092] According to the sample outlier value calculated in step H3, the sample reconciliation result is determined. When the sample outlier value is greater than the preset threshold, the sample reconciliation result is abnormal. The sample source data corresponding to the abnormal sample reconciliation result is input into the sample second isolation forest. The calculation process of the outlier score of the sample source data in the second isolation forest is consistent with the calculation formula for calculating the outlier score in the above steps G1 and G2. According to the relevant contents of formulas 3 and 4, the sample outlier score of the sample source data is calculated.

[0093] Step H5, determine the sample detection result according to the sample outlier value and the sample outlier score, and adjust the parameters in the outlier detection model based on the sample detection result and the sample equity data to train the outlier detection model.

[0094] According to the sample abnormal value and the sample abnormal score, the sample detection result is determined, including: when the sample abnormal value is greater than the preset threshold, the sample reconciliation result is abnormal, when the detection of the sample reconciliation result is abnormal, the sample source data of the sample equity data corresponding to the sample reconciliation result is detected, and the sample abnormal score is calculated. According to the sample abnormal score, it is determined whether the data reconciliation of the equity data is abnormal. When the detection of the sample reconciliation result is normal, the sample detection result of the equity data reconciliation is normal. The sample abnormal detection result is output through the abnormal detection model, and the parameters of the abnormal detection model are adjusted according to the sample detection result and the sample equity data, so that the output sample detection result is consistent with the sample equity data. If the sample detection result is that the sample source data is abnormal, then the sample equity data is also abnormal, that is, the amount received by the user in the sample source data is inconsistent with the amount to be received, then the amount received by the user in the sample equity data is inconsistent with the amount to be received. Among them, the parameters of the adjustment of the abnormal detection model may include time factors, sample order quantity or parameters in the abnormal detection model formula, etc., which are not specifically limited.

[0095] In this embodiment, the sample reconciliation result is detected by training the first isolated forest. When the sample reconciliation result is abnormal, the sample source data of the sample reconciliation result is detected by training the second isolated forest. The sample detection result is determined according to the calculated abnormal score of the sample source data. The parameters of the abnormal detection model are adjusted by comparing multiple sample detection results with sample equity data, and the abnormal detection model including the first isolated forest and the second isolated forest is trained. The abnormal detection model formed by the improved isolated forest algorithm can automatically identify abnormalities in the equity data reconciliation task and discover problems in time, thereby reducing the cost of manual inspection operations and improving the accuracy of equity data reconciliation and settlement.

[0096] In one embodiment, the training of the anomaly detection model may further include the following step I: In step I, user feedback information on sample detection results is input into the trained anomaly detection model, and the anomaly detection model is determined by adjusting the parameters of the anomaly detection model offline.

[0097] User feedback information includes: relevant information for feedback on sample detection results based on actual conditions. For example, manually spot check some abnormal source data and reconciliation result data to verify the feedback results, add the feedback results to the offline training data set for offline training, and similarly, add historical reconciliation result data or historical source data to the offline training data set. During offline training, by continuously comparing the anomalies detected by the anomaly detection model, the actual situation of user feedback, and the actual situation of historical data reconciliation, the algorithm parameters of the isolation forest, the custom anomaly score and reconciliation factor, and the preset threshold of the anomaly score are adjusted accordingly, and then the performance of the anomaly detection model is evaluated, and the anomaly detection model is continuously updated and improved based on the evaluation results, and the user feedback information is stored in the data lake, thereby improving the accuracy and timeliness of the anomaly detection model.

[0098] It should be noted that the structural diagram of the anomaly detection module related to equity data reconciliation is as follows: Figure 3 As shown, it includes real-time detection of reconciliation results and offline training of anomaly detection models. Specifically, real-time detection of reconciliation results includes: obtaining the reconciliation results of equity data and the corresponding source data, extracting the features of the reconciliation results and the source data, using the first isolation forest and the second isolation forest in the anomaly detection model during real-time detection to detect the extracted features, identify abnormal points, and obtain detection results. Offline training of anomaly detection models includes: inputting user feedback on the detection results into an offline training data set, extracting the features of the data in the training data set, inputting the offline anomaly detection model, training the offline anomaly detection model by adjusting parameters, and evaluating and tuning the offline anomaly detection model, and updating the anomaly detection model used in real-time inspection according to the tuned offline anomaly detection model.

[0099] In this embodiment, by adding user feedback information to the sample training set, the anomaly detection model is trained offline to obtain an offline anomaly detection model. The anomaly detection model during real-time detection is continuously optimized and adjusted according to actual conditions, so that the trained anomaly detection model can be more accurate.

[0100] In one embodiment, the reconciliation flow chart of the data reconciliation method based on anomaly detection is as follows: Figure 4As shown, the data reconciliation of the equity data begins. Through the data collection of the equity data, the collected equity data is checked through the data entry module. The checking of the equity data includes: checking whether the equity data is complete. If the equity data is incomplete, the equity data is deleted and the subsequent reconciliation process is terminated. If the equity data is complete data, it is input into the data lake. According to the equity data input into the data lake, the corresponding metadata information is generated through the metadata management module in the data lake, and the source data corresponding to the equity data is stored in the data lake. According to the metadata information managed by the metadata management module, the corresponding reconciliation task is executed in the data lake through the reconciliation algorithm preset in the data reconciliation module to obtain the reconciliation result. If the reconciliation result is normal, the reconciliation result can be directly output, but it is also necessary to use the first isolated forest of the anomaly detection model in the anomaly detection module to detect the reconciliation result to determine whether the reconciliation result needs to be rerun, that is, whether it is abnormal. If it is abnormal, the source data corresponding to the equity data is detected through the second isolated forest of the anomaly detection model in the anomaly detection module according to the information in the data reconciliation module, and the detection result is output, then the data reconciliation result is output and the data reconciliation process is terminated.

[0101] It should be noted that when the detection result is that the source data is abnormal, new source data can be generated by modifying the relevant information of the source data, and the reconciliation tasks in the reconciliation algorithm can be automatically re-executed based on the source data to generate a reconciliation result, and then the corresponding reconciliation result is tested. If the corresponding reconciliation result is abnormal, the new source data is tested again to obtain the inspection result.

[0102] In this embodiment, the reconciliation process of the data reconciliation method based on anomaly detection can clearly determine the execution steps in the data reconciliation process, and further improve the accuracy and efficiency of data reconciliation by detecting the reconciliation results and the source data.

[0103] Figure 5 is a schematic flow chart of a data reconciliation method based on anomaly detection according to another embodiment of the present application, such as Figure 5 As shown, the method comprises the following steps: S501, obtain the equity data to be reconciled and input it into the data lake.

[0104] S502, extracting structured data and semi-structured data of equity data in the data lake, and generating metadata information of the equity data.

[0105] S503: Determine the source data of the equity data in the data lake according to the metadata information.

[0106] S504: Determine the reconciliation algorithm according to the business scenario corresponding to the equity data, and obtain the source data in the data lake according to the metadata information.

[0107] S505, executing the reconciliation task in the reconciliation algorithm according to the reconciliation algorithm and the searched source data, and obtaining the reconciliation result of the equity data.

[0108] Specifically, the distribution of abnormal reconciliation data and normal reconciliation data included in the reconciliation results of the equity data is viewed and displayed through the visualization tools in the pre-trained anomaly detection model. When the reconciliation anomaly of the equity data corresponding to the reconciliation result is identified, first check whether the corresponding reconciliation task is completed normally. If an abnormal interruption occurs, re-run the reconciliation task incrementally or in full according to the abnormal interruption to keep the reconciliation task running normally.

[0109] S506, inputting the reconciliation result into the pre-trained anomaly detection model, and determining the anomaly value of the reconciliation result according to the expected value of the path length of the first isolation forest of the anomaly detection model and the reconciliation factor of the reconciliation result.

[0110] S507, detecting the reconciliation result based on the abnormal value.

[0111] S508, when there is no abnormality in the reconciliation result, the abnormality detection model outputs the reconciliation result.

[0112] S509: When there is an anomaly in the reconciliation result, the anomaly score is determined according to the expected value of the path length of the second isolation forest of the source data in the pre-trained anomaly detection model and the average path length of the binary tree corresponding to the source data.

[0113] S510, obtaining a detection result according to the abnormality score, and performing data reconciliation of the equity data based on the detection result.

[0114] The specific process from S501 to S510 has been described in detail in the above embodiment and will not be repeated here.

[0115] The technical solution of the embodiment of the present application is adopted, by obtaining the source data of the equity data to be reconciled, and according to the preset reconciliation algorithm, the reconciliation result corresponding to the equity data is calculated; the reconciliation result is input into the pre-trained anomaly detection model, and the anomaly value of the reconciliation result is determined according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; based on the anomaly value, the reconciliation result is detected, and when the reconciliation result is abnormal, the source data of the equity data is detected by using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data. It can be seen that through the pre-trained anomaly detection model, the anomaly value of the reconciliation result is calculated by using the expected value of the path length of the first isolated forest of the reconciliation result and the reconciliation factor, and when the reconciliation result is detected to be abnormal, the source data of the equity data is detected again, and multiple detections can accurately locate the abnormal equity data, so that the reconciliation results of a large amount of equity data are more accurate, solving the problem of low efficiency and accuracy of data reconciliation.

[0116] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0117] The above is a data reconciliation method based on anomaly detection provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a data reconciliation device based on anomaly detection.

[0118] Figure 6 is a schematic diagram of the structure of a data reconciliation device based on anomaly detection according to an embodiment of the present invention. Figure 6 As shown, the data reconciliation device based on anomaly detection includes: a metadata management module 61, an anomaly detection module 62, and a detection module 63: The metadata management module 61 is used to obtain the source data of the equity data to be reconciled and calculate the reconciliation result corresponding to the equity data according to a preset reconciliation algorithm; An anomaly detection module 62, used to input the reconciliation result into a pre-trained anomaly detection model, and determine the anomaly value of the reconciliation result according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; The detection module 63 is used to detect the reconciliation result based on the abnormal value. When the reconciliation result is abnormal, the source data of the equity data is detected using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result. The detection result includes normal information or error information of the source data.

[0119] In one embodiment, the apparatus further comprises: The reconciliation module is used to output normal reconciliation information using a pre-trained anomaly detection model when there is no anomaly in the reconciliation result, and to reconcile the equity data based on the normal reconciliation information.

[0120] In one embodiment, the device further includes a lake entry module, which is specifically used to input the equity data into the data lake to generate source data of the equity data, where the source data includes metadata information corresponding to the equity data.

[0121] In one embodiment, the data lake entry module is also used to collect relational data and / or non-relational data in the data to be reconciled by using full extraction and incremental real-time extraction; verify the relational data and / or non-relational data, and input the verified data into the data lake as equity data.

[0122] In one embodiment, the device also includes a metadata management module, which is specifically used to automatically extract structured data from the equity data in the data lake using a metadata management tool; extract semi-structured data from the equity data in the data lake using preset rules or statistical analysis methods; determine metadata information corresponding to the equity data in the data lake based on the structured data and the semi-structured data, the metadata information is used to locate and search for source data corresponding to the equity data in the data lake; determine the source data of the equity data based on the metadata information.

[0123] In one embodiment, the device also includes a data reconciliation module, which is used to determine a reconciliation algorithm based on the business scenario corresponding to the equity data; according to the reconciliation algorithm, search for metadata information corresponding to the equity data in the data lake, and determine the reconciliation task; and obtain a reconciliation result by executing the reconciliation task, wherein the reconciliation result includes success data or failure data of executing the reconciliation task.

[0124] In one embodiment, the detection module 63 is also used to search for metadata information corresponding to the reconciliation result from the data lake based on the reconciliation result, determine the source data of the equity data based on the metadata information, input the source data into the second isolation forest of the anomaly detection model, and calculate the anomaly score; determine the detection result based on the anomaly score, and the detection result includes whether the equity data is abnormal.

[0125] In one embodiment, the detection module 63 is also used to input the source data into the second isolation forest of the anomaly detection model, construct a binary tree corresponding to the source data based on the source data, and determine the average path length of the binary tree corresponding to the source data; based on the average path length and the expected value of the path length of the source data in the second isolation forest, determine the anomaly score.

[0126] In one embodiment, the training module specifically includes: An acquisition unit, used to acquire a plurality of sample equity data, sample source data corresponding to the sample equity data, and sample reconciliation results, and input the sample reconciliation results into an anomaly detection model; An extraction unit, used to extract sample feature data from the sample reconciliation result, construct a binary tree based on the feature data, and determine the first isolation forest of the sample according to the binary tree constructed with the feature data; A determination unit is used to determine an expected value of the path length of the sample reconciliation result based on the sample first isolation forest, and determine a sample outlier of the sample reconciliation result according to the expected value of the path length of the sample reconciliation result and a sample reconciliation factor; wherein the sample reconciliation factor includes at least one of the following factors: an order status of the sample reconciliation party, a sample reconciliation result status, a sample time factor, and a sample order quantity; A calculation unit, used for, when the sample abnormal value is greater than a preset threshold, the sample reconciliation result is abnormal, inputting the sample source data corresponding to the abnormal sample reconciliation result into the sample second isolation forest, and calculating the sample abnormality score; The training unit is used to determine the sample detection result according to the sample outlier value and the sample outlier score, and adjust the parameters in the anomaly detection model based on the sample detection result and the sample equity data to train the anomaly detection model.

[0127] In one embodiment, the training module is further configured to input user feedback information on sample detection results into the trained anomaly detection model, and determine the anomaly detection model by adjusting parameters of the anomaly detection model offline.

[0128] The technical solution of the embodiment of the present application is adopted, by obtaining the source data of the equity data to be reconciled, and according to the preset reconciliation algorithm, the reconciliation result corresponding to the equity data is calculated; the reconciliation result is input into the pre-trained anomaly detection model, and the anomaly value of the reconciliation result is determined according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; based on the anomaly value, the reconciliation result is detected, and when the reconciliation result is abnormal, the source data of the equity data is detected by using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data. It can be seen that through the pre-trained anomaly detection model, the anomaly value of the reconciliation result is calculated by using the expected value of the path length of the first isolated forest of the reconciliation result and the reconciliation factor, and when the reconciliation result is detected to be abnormal, the source data of the equity data is detected again, and multiple detections can accurately locate the abnormal equity data, so that the reconciliation results of a large amount of equity data are more accurate, solving the problem of low efficiency and accuracy of data reconciliation.

[0129] Those skilled in the art should understand that Figure 6 The data reconciliation device based on anomaly detection can be used to implement the data reconciliation method based on anomaly detection described above, and the detailed description thereof should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0130] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned data reconciliation method based on anomaly detection. Figure 7 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call a computer program stored in the memory 730 and executable on the processor 710 to perform the following steps: Obtain the source data of the equity data to be reconciled, and calculate the reconciliation result corresponding to the equity data according to the preset reconciliation algorithm; Input the reconciliation result into the pre-trained anomaly detection model, and determine the anomaly value of the reconciliation result according to the expected value of the path length of the first isolated forest in the anomaly detection model and the reconciliation factor; Based on outliers, the reconciliation results are detected. When there are anomalies in the reconciliation results, the pre-trained anomaly detection model is used to detect the source data of the equity data to obtain the detection results. The equity data is reconciled based on the detection results. The detection results include normal information or error information of the source data.

[0131] The technical solution of the embodiment of the present application is adopted, by obtaining the source data of the equity data to be reconciled, and according to the preset reconciliation algorithm, the reconciliation result corresponding to the equity data is calculated; the reconciliation result is input into the pre-trained anomaly detection model, and the anomaly value of the reconciliation result is determined according to the expected value of the path length of the first isolated forest of the reconciliation result in the anomaly detection model and the reconciliation factor; based on the anomaly value, the reconciliation result is detected, and when the reconciliation result is abnormal, the source data of the equity data is detected by using the pre-trained anomaly detection model to obtain the detection result, and the equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data. It can be seen that through the pre-trained anomaly detection model, the anomaly value of the reconciliation result is calculated by using the expected value of the path length of the first isolated forest of the reconciliation result and the reconciliation factor, and when the reconciliation result is detected to be abnormal, the source data of the equity data is detected again, and multiple detections can accurately locate the abnormal equity data, so that the reconciliation results of a large amount of equity data are more accurate, solving the problem of low efficiency and accuracy of data reconciliation.

[0132] The specific execution steps can refer to the various steps of the above-mentioned data reconciliation method embodiment based on anomaly detection, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0133] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0134] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0135] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0136] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0137] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data reconciliation method embodiment based on anomaly detection is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0138] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data reconciliation method embodiment based on anomaly detection, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0141] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0142] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data reconciliation method based on anomaly detection, characterized in that: The method comprises: Obtain source data of the equity data to be reconciled, and calculate the reconciliation result corresponding to the equity data according to a preset reconciliation algorithm; Inputting the reconciliation result into a pre-trained anomaly detection model, and determining an abnormal value of the reconciliation result according to an expected value of the path length of the first isolation forest of the reconciliation result in the anomaly detection model and a reconciliation factor; Based on the abnormal value, the reconciliation result is detected. When the reconciliation result is abnormal, the pre-trained abnormality detection model is used to detect the source data of the equity data to obtain a detection result. The equity data is reconciled based on the detection result, and the detection result includes normal information or error information of the source data.

2. The method according to claim 1, characterized in that The method comprises: When there is no abnormality in the reconciliation result, the pre-trained abnormality detection model is used to output normal reconciliation information, and the equity data reconciliation is performed based on the normal reconciliation information.

3. The method according to claim 1, characterized in that The step of obtaining source data of the equity data to be reconciled includes: The equity data is input into a data lake to generate source data of the equity data, where the source data includes metadata information corresponding to the equity data.

4. The method according to claim 3, characterized in that The inputting the equity data into the data lake includes: Collect relational data and / or non-relational data from the data to be reconciled by using full extraction and incremental real-time extraction; Verification processing is performed on the relational data and / or the non-relational data, and the verified data is input into the data lake as the equity data.

5. The method according to claim 4, characterized in that The source data for generating the equity data includes: Automatically extracting structured data from the equity data in the data lake using a metadata management tool; Extracting semi-structured data from the equity data in the data lake using a preset rule or statistical analysis method; Determine, according to the structured data and the semi-structured data, the metadata information corresponding to the equity data in the data lake, wherein the metadata information is used to locate and search for the source data corresponding to the equity data in the data lake; The source data of the equity data is determined according to the metadata information.

6. The method according to claim 3, characterized in that The calculating the reconciliation result corresponding to the equity data according to the preset reconciliation algorithm includes: Determining the reconciliation algorithm based on the business scenario corresponding to the equity data; According to the reconciliation algorithm, searching for the metadata information corresponding to the equity data in the data lake, and determining the reconciliation task; The reconciliation result is obtained by executing the reconciliation task, wherein the reconciliation result includes success data or failure data of executing the reconciliation task.

7. The method according to claim 3, characterized in that The using the pre-trained anomaly detection model to detect the source data of the equity data to obtain a detection result includes: According to the reconciliation result, searching the metadata information corresponding to the reconciliation result from the data lake, determining the source data of the equity data according to the metadata information, inputting the source data into the second isolation forest of the anomaly detection model, and calculating the anomaly score; The detection result is determined according to the abnormality score, and the detection result includes whether there is an abnormality in the equity data.

8. The method according to claim 7, characterized in that The step of inputting the source data into the second isolation forest of the anomaly detection model and calculating anomaly scores comprises: Inputting the source data into the second isolation forest of the anomaly detection model, constructing a binary tree corresponding to the source data based on the source data, and determining an average path length of the binary tree corresponding to the source data; The anomaly score is determined based on the average path length and an expected value of the path length of the source data in the second isolation forest.

9. The method according to claim 1, characterized in that: The training of the anomaly detection model includes: Acquire a plurality of sample equity data, sample source data corresponding to the sample equity data, and sample reconciliation results, and input the sample reconciliation results into the anomaly detection model; Extracting sample feature data from the sample reconciliation result, constructing a binary tree based on the feature data, and determining a sample first isolation forest based on the binary tree constructed with the feature data; Based on the sample first isolation forest, determine the expected value of the path length of the sample reconciliation result, and determine the sample outlier of the sample reconciliation result according to the expected value of the path length of the sample reconciliation result and the sample reconciliation factor; wherein the sample reconciliation factor includes at least one of the following factors: the order status of the sample reconciliation party, the sample reconciliation result status, the sample time factor, and the sample order quantity; When the sample abnormal value is greater than a preset threshold, the sample reconciliation result is abnormal, the sample source data corresponding to the abnormal sample reconciliation result is input into the sample second isolation forest, and the sample abnormality score is calculated; A sample detection result is determined according to the sample outlier value and the sample outlier score, and parameters in the outlier detection model are adjusted based on the sample detection result and the sample equity data to train the outlier detection model.

10. The method according to claim 9, characterized in that The training of the anomaly detection model further includes: The user feedback information on the sample detection result is input into the trained anomaly detection model, and the anomaly detection model is determined by adjusting the parameters of the anomaly detection model offline.

Images