Data recovery system, method, device, storage medium and computer program product

Abstract

The invention provides a data recovery system, method and device, a storage medium and a computer program product. The data recovery system comprises: a data acquisition module, which is used for acquiring first original data from a first data source and a second data source; the data storage module is used for storing the first original data; the data checking and repairing module is used for extracting to-be-checked data and feature data from the first original data, generating a label according to a relation between the feature data and a preset threshold value, training and verifying a preset model to obtain a trained checking model, predicting collected second original data according to the checking model, and outputting the predicted second original data; and when the predicted value output by the check model is an abnormal label, repairing the second original data according to a predefined repairing rule to obtain repaired data. By the adoption of the system, the problem that data of different data sources are inconsistent can be solved, manual intervention is reduced, and the accuracy and efficiency of data checking and repairing are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and more specifically, to a data repair system, method, device, storage medium and computer program product. Background Art

[0002] In modern consumer credit business, financial companies often make joint loans with other financial institutions to expand their business scope and improve their loan service capabilities. The debt of such joint loans usually belongs to the financial company, but the accounting and management are carried out by other financial institutions (i.e., "other accounting"). Due to the differences in accounting rules and standards between self-accounting (the processing of loan data by the internal system of the financial company) and other accounting (the processing of the same data by external financial institutions), data inconsistencies often occur. These data inconsistencies may be caused by the different calculation rules between self-accounting and other accounting, such as interest rate calculation method, repayment plan, fee processing, etc. Or they may be due to rounding errors, data loss or recording errors during data transmission and processing. Traditional manual intervention methods are inefficient, costly and prone to errors when faced with such data differences. In order to improve data processing efficiency, reduce manual intervention, and ensure data accuracy, an automated data verification and repair solution is urgently needed. Summary of the invention

[0003] In order to solve the problems existing in the above-mentioned prior art or at least part of the problems, the embodiments of the present invention provide a data repair system, method, device, storage medium and computer program product, which use machine learning algorithms to automatically predict and repair differences in data from different data sources, thereby solving the problem of data inconsistency between different data sources, reducing manual intervention, and improving the accuracy and efficiency of data verification and repair.

[0004] According to a first aspect of the present invention, an embodiment of the present invention provides a data repair system, which includes: a data acquisition module, used to collect first original data from a first data source and a second data source; a data storage module, used to store the first original data; a data verification and repair module, used to perform the following operations: extracting data to be verified from the first original data according to first information, and extracting feature data from the data to be verified; generating labels according to the relationship between the feature data and a preset threshold, the labels including abnormal labels and normal labels; training and verifying a preset model according to the feature data and the labels to obtain a trained verification model; predicting the collected second original data according to the verification model, and when the prediction value output by the verification model is an abnormal label, repairing the second original data according to a predefined repair rule to obtain repaired data.

[0005] According to the above-mentioned embodiments of the present invention, by extracting the characteristic data of the data to be checked from different data sources and generating labels, and then using a machine learning algorithm to automatically predict and repair the differences in the data from different data sources, the problem of data inconsistency between different data sources can be solved, manual intervention can be reduced, and the accuracy and efficiency of data checking and repair can be improved.

[0006] In some embodiments of the present invention, the data repair system also includes: an exception handling module for detecting and processing exception information generated during the data repair process; and a report generation module for generating a data verification and repair report based on the second original data, repair data and exception information.

[0007] According to the above-mentioned implementation mode of the present invention, by generating a report including data comparison before and after repair and abnormal conditions during the repair process, it is convenient for relevant management personnel to observe the online data status at any time.

[0008] In some embodiments of the present invention, the first data source is a self-accounting system, and the second data source is an external accounting system; the first original data and the second original data are transaction data, and the transaction data include: loan application information, interest rate information, loan note information, repayment plan information, repayment flow information, and accrual information; the first information is the loan note number.

[0009] According to the above-mentioned embodiments of the present invention, the data difference between the self-calculation and the other-calculation can be automatically repaired, which reduces the manual intervention and improves the repair efficiency.

[0010] In some embodiments of the present invention, generating a label based on the relationship between the feature data and a preset threshold includes: calculating the absolute value of the difference between the first feature data in the self-accounting system and the second feature data in the other-accounting system as a difference value; when the difference value is greater than the preset threshold, generating an abnormal label; when the difference value is less than or equal to the preset threshold, generating a normal label.

[0011] In some implementations of the present invention, storing the first original data includes: using a relational database MySQL to store the first original data as a table structure.

[0012] In some embodiments of the present invention, storing the first original data also includes: allocating the first original data to a corresponding database according to the product number; storing the first original data in a corresponding storage table in the database according to the promissory note number, the storage table including: a loan table, a repayment table, a repayment plan table, a transaction table, and an accrual table; associating the storage tables according to the promissory note number.

[0013] According to the above-mentioned embodiment of the present invention, the original data is stored in the database table according to the product number and the loan note number, so that the accounting data can be quickly queried and accurately matched.

[0014] According to the second aspect of the present invention, an embodiment of the present invention provides a data repair method, which includes: collecting first original data from a first data source and a second data source; storing the first original data; extracting data to be checked from the first original data according to first information, and extracting feature data from the data to be checked; generating a label according to a relationship between the feature data and a preset threshold, the label including an abnormal label and a normal label; training and verifying a preset model according to the feature data and the label to obtain a trained verification model; predicting the collected second original data according to the verification model, and when the predicted value output by the verification model is an abnormal label, repairing the second original data according to a predefined repair rule to obtain repaired data.

[0015] According to the above-mentioned embodiments of the present invention, by extracting the characteristic data of the data to be checked from different data sources and generating labels, and then using a machine learning algorithm to automatically predict and repair the differences in the data from different data sources, the problem of data inconsistency between different data sources can be solved, manual intervention can be reduced, and the accuracy and efficiency of data checking and repair can be improved.

[0016] In some embodiments of the present invention, the data repair method further includes: detecting and processing abnormal information generated during the data repair process; and generating a data verification and repair report based on the second original data, the repair data and the abnormal information.

[0017] According to the above-mentioned implementation mode of the present invention, by generating a report including data comparison before and after repair and abnormal conditions during the repair process, it is convenient for relevant management personnel to observe the online data status at any time.

[0018] In some embodiments of the present invention, the first data source is a self-accounting system, and the second data source is an external accounting system; the first original data and the second original data are transaction data, and the transaction data include: loan application information, interest rate information, loan note information, repayment plan information, repayment flow information, and accrual information; the first information is the loan note number.

[0019] According to the above-mentioned embodiments of the present invention, the data difference between the self-calculation and the other-calculation can be automatically repaired, which reduces the manual intervention and improves the repair efficiency.

[0020] In some embodiments of the present invention, generating a label based on the relationship between the feature data and a preset threshold includes: calculating the absolute value of the difference between the first feature data in the self-accounting system and the second feature data in the other-accounting system as a difference value; when the difference value is greater than the preset threshold, generating an abnormal label; when the difference value is less than or equal to the preset threshold, generating a normal label.

[0021] In some implementations of the present invention, storing the first original data includes: using a relational database MySQL to store the first original data as a table structure.

[0022] In some embodiments of the present invention, storing the first original data also includes: allocating the first original data to a corresponding database according to the product number; storing the first original data in a corresponding storage table in the database according to the promissory note number, the storage table including: a loan table, a repayment table, a repayment plan table, a transaction table, and an accrual table; associating the storage tables according to the promissory note number.

[0023] According to the above-mentioned embodiment of the present invention, the original data is stored in the database table according to the product number and the loan note number, so that the accounting data can be quickly queried and accurately matched.

[0024] According to the third aspect of the present invention, an embodiment of the present invention provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the computer performs the following operations: the operations include the steps included in the data repair method described in any one of the above embodiments.

[0025] According to the fourth aspect of the present invention, an embodiment of the present invention provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the processor, can implement the data repair method described in any one of the above embodiments.

[0026] According to a fifth aspect of the present invention, an embodiment of the present invention provides a computer program product including a computer program, and when the computer program is executed by a processor, the data repair method as described in any one of the above embodiments is implemented.

[0027] From the above, it can be seen that the data repair system, method, device, storage medium and computer program product provided in the embodiments of the present invention can solve the problem of data inconsistency between different data sources by extracting feature data of the data to be checked from different data sources and generating labels, and then using machine learning algorithms to automatically predict and repair the data in different data sources, thereby reducing manual intervention and improving the accuracy and efficiency of data verification and repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of the architecture of a data repair system according to Embodiment 1 of the present invention;

[0029] Figure 2 is a flow chart of a data repair method according to Embodiment 2 of the present invention;

[0030] Figure 3 4 is a flow chart of a data repair method according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0031] The various aspects of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, well-known modules, units and their connections, links, communications or operations are not shown or described in detail. In addition, the described features, architectures or functions can be combined in any way in one or more embodiments. It should be understood by those skilled in the art that the various embodiments described below are only used for illustration and are not intended to limit the scope of protection of the present invention. It can also be easily understood that the modules or units or processing methods in each embodiment described herein and shown in the accompanying drawings can be combined and designed in various different configurations.

[0032] The following is a brief description of the terms used in this article.

[0033] Self-accounting: The loan claims belong to the current financial institution (our company), and the accounting (loan management party) also belongs to our company.

[0034] Other accounting: The loan claims belong to the current financial institution (our company), but the accounting (loan management party) is another financial institution, and the accounts are recorded through batch accounting files provided by the other party.

[0035] API: Application Programming Interface, application programming interface.

[0036] SQL: Structured Query Language, a standard language specifically used to manage relational databases.

[0037] CVS file: A data format that is a plain text file with commas as separators between fields and newline characters as separators between records. CSV stands for "Comma-Separated Values".

[0038] Accuracy: An indicator used to evaluate the performance of a classification model. Accuracy is the ratio of the number of samples correctly predicted by the classification model to the total number of samples.

[0039] Recall: One of the important indicators for evaluating the performance of classification models. Recall refers to the proportion of samples that are correctly predicted to be positive among all samples that are actually positive.

[0040] [Example 1]

[0041] Figure 1 Schematic diagram of the architecture of a data repair system according to Embodiment 1 of the present invention.

[0042] like Figure 1 As shown, the data repair system includes: a data acquisition module 110 , a data storage module 120 , a data checking and repairing module 130 , an exception handling module 140 , and a report generating module 150 .

[0043] The data collection module 110 is used to collect first original data from a first data source and a second data source. In some embodiments, the first data source is a self-accounting system, and the second data source is a foreign accounting system; the first original data and the second original data described later are transaction data, and the transaction data includes but is not limited to one or more of the following: loan application information, interest rate information, loan note information, repayment plan information, repayment flow information, and provision information.

[0044] In a further implementation, a flexible configuration method is used to configure data sources by product, for example, by configuring the corresponding relationship between product number and database (accounting library), and by changing the product number to switch different databases for query; by API interface call, dynamic multi-data source switching SQL query, parsing CVS files, etc., the original transaction data is collected daily from the systems of various channels (external financial institutions) and the internal systems of financial companies. Among them, the data obtained by external financial institutions or synchronized by external financial institutions to the current financial institution (self-accounting system), and the data parsed by the self-accounting system and stored in the library are collectively referred to as other accounting data, and the data collected internally by the current financial institution (self-accounting system) is referred to as self-accounting data.

[0045] The data storage module 120 is used to store the first original data.

[0046] In some implementations, storing the collected raw data includes: using a relational database MySQL for data storage, and the storage format is a table structure, thereby ensuring the reliability and scalability of data storage.

[0047] In a further implementation, the collected raw data is divided into databases according to product numbers, and stored in separate tables according to loan numbers; various types of data are classified, and one-to-many associations are made through loan numbers and other loan information, that is, the loan table, repayment table, repayment plan table, transaction table, provision table, etc. are associated according to the loan number. Among them, the product number is the unique number of different products, which is used to distinguish a certain product, such as the Xingfuhua product number is 1001, and the Maxiao product number is 8009. Therefore, the raw data is distributed and stored in the database table according to the product number and loan number, which can realize the rapid query and accurate matching of accounting data.

[0048] In an optional implementation, the implementation method of database sharding includes: using dynamic data source technology, such as surround monitoring annotations, to implement switching of operations of DynamicDataSourceContextHolder (thread-level dynamic data source context holder, used to store the id identifier of the dynamic data source) that calls dynamicDatasource (dynamic data source) to different databases; the implementation method of table sharding includes: taking a modulus based on the IOU ID, such as IOU number%16 (total number of sub-tables), to obtain the sub-table serial number.

[0049] The data verification and repair module 130 is used to perform the following operations: extract the data to be verified from the first original data according to the first information, and extract feature data from the data to be verified; generate a label according to the relationship between the feature data and a preset threshold, and the label includes an abnormal label and a normal label; train and verify the preset model according to the feature data and the label to obtain a trained verification model; predict the collected second original data according to the verification model, and when the predicted value output by the verification model is an abnormal label, repair the second original data according to a predefined repair rule to obtain repaired data.

[0050] Among them, the characteristic data includes but is not limited to the loan information of the self-accounting system and other accounting systems, loan interest rate, loan term, repayment plan, accrual amount and other information; the predefined repair rules include the difference items obtained by comparing the self-accounting system data with the other accounting system data, including the repayment plan amount, loan interest rate, settlement date, overdue days and other data.

[0051] In some embodiments, when the first data source and the second data source are respectively a self-accounting system and an external accounting system, the data to be verified is extracted from the first original data according to the loan note number.

[0052] In a further embodiment, generating a label based on the relationship between the feature data and a preset threshold includes: calculating the absolute value of the difference between the first feature data in the self-accounting system and the second feature data in the other accounting system as a difference value; when the difference value is greater than the preset threshold, generating an abnormal label; when the difference value is less than or equal to the preset threshold, generating a normal label.

[0053] In an exemplary embodiment, the data checking and repairing module performs the following steps to check and repair the data differences between the self-accounting system and the other accounting system:

[0054] Step 1: Feature extraction

[0055] Extract raw data from the database according to the loan note number, extract features from the raw data, and calculate difference data / difference information according to the difference detection formula. The difference information includes but is not limited to: loan amount difference, repayment plan difference, interest rate difference, fee amount difference, overdue days difference, etc. The difference data constitutes a feature vector, which will be used as the input of the machine learning model.

[0056] Optionally, the difference detection formula is as follows: ΔV=|V self -V other |, when ΔV>T (preset threshold), there is a difference, and the difference data is analyzed and recorded.

[0057] Step 2: Generate labels

[0058] Generate a data label indicating whether there is a difference in the data based on the preset threshold rule. Optionally, if the difference in loan amount, interest or repayment plan exceeds the preset threshold, it is marked as abnormal (labeled as 1), otherwise it is normal (labeled as 0).

[0059] Step 3: Model training

[0060] Select a suitable machine learning algorithm (such as the random forest algorithm) for model training. The machine learning algorithm learns the relationship between features and targets through training data and generates a model for prediction. Among them, the random forest algorithm improves the accuracy of the model through the combination of multiple decision trees.

[0061] Optionally, perform model training by following these steps:

[0062] (1) Input data: Use the extracted features and labels as input data.

[0063] (2) Loss function: Select mean square error (MSE) or cross entropy loss function to measure the accuracy of model prediction.

[0064] For example, using mean square error as the optimization target of the loss function, the mean square error formula is as follows:

[0065]

[0066] Where n is the number of samples, y i is the true value, is the predicted value.

[0067] (3) Training process: By optimizing the loss function, the model parameters are adjusted so that it can accurately predict the differences in the data. In the code, the loss function of the model training will be automatically optimized through the algorithm, such as using methods such as gradient descent to minimize the value of the loss function.

[0068] Step 4: Model Validation

[0069] (1) Use K-Fold Cross Validation to evaluate model performance. Through multiple training and validation, ensure the generalization ability of the model and avoid overfitting.

[0070] In this embodiment, the code example of using K-fold cross validation to evaluate model performance is as follows:

[0071]

[0072]

[0073] (2) Calculate precision and recall

[0074] We evaluate the performance of the model in detecting differences in the data by calculating precision and recall. A high precision indicates that the model can accurately identify the differences, and a high recall indicates that the model can capture most of the differences.

[0075] The accuracy is calculated according to the following formula:

[0076]

[0077] The recall rate is calculated according to the following formula:

[0078]

[0079] Among them, TP, TN, FP and FN are the number of true positive examples, true negative examples, false positive examples and false negative examples respectively.

[0080] In this embodiment, the code example for calculating the two indicators of precision and recall is as follows:

[0081] return precision.measure(testLabels,predictions);

[0082] }

[0083] public double calculateRecall(RandomForest model,double[][]testData,int[]testLabels){

[0084] int[]predictions=model.predict(testData);

[0085] Recall recall=new Recall();

[0086] return recall.measure(testLabels,predictions);

[0087] }

[0088] }

[0089] Step 5: Save the model. Save the trained model to disk for use by the data verification and repair module.

[0090] Step 6: Data repair

[0091] (1) Model loading: Load a trained machine learning model from disk.

[0092] (2) Data repair: Apply the trained model to new accounting data, automatically detect differences and perform repairs. When the predicted value of the model output is 1 (indicating a difference), the predefined repair rules are used to adjust the data so that the repair value = model prediction (feature data). The repair value refers to the value after the abnormal data is corrected according to certain rules after the model detects the abnormality; the expected value refers to a reasonable value used for reference in the data repair process, which is usually a value derived from the business logic based on the data of another party and the deduced data from the data of another party; the repair rule is implemented based on the output of the machine learning model, such as adjusting a specific feature value to make it closer to the expected value. Optionally, if the model indicates that the value of a feature is abnormal, the corresponding repair rule includes adjusting the feature value to a reasonable range.

[0093] (3) Result update: Update the repaired data to the database. If the self-accounting system detects data that is inconsistent with the other accounting system using the repair model, the self-accounting system data needs to be repaired and the differences recorded. Finally, the difference data and the repaired self-accounting system data are saved back to the database.

[0094] The data verification and repair module executes the above processing steps and uses machine learning algorithms such as random forest to handle complex nonlinear data repair problems, making data repair more accurate. It ensures the accuracy and stability of the model through training and verification processes, thereby improving the quality of repaired data. It automatically repairs data differences between self-accounting and other-accounting through machine learning models, reducing manual intervention and improving repair efficiency. It predicts missing values ​​or outliers based on historical data, which is more accurate than traditional methods. The repaired data can be updated to the database in real time to support real-time data analysis and decision-making.

[0095] The exception processing module 140 is used to detect and process the exception information generated during the data repair process.

[0096] In some embodiments, the exception handling module monitors the abnormal conditions in the data repair process, such as model prediction failure, data update failure, etc., and saves the information of the repair abnormality to the repair log table, which includes fields such as the loan number, difference amount, difference reason, and repair status. Further, the abnormal conditions are handled, such as retrying the repair operation, alarming and notifying manual intervention, etc. Moreover, when manual intervention is handled, the intelligent system records and learns the strategy for automatic repair when the same problem occurs later.

[0097] The report generation module 150 is used to generate a data verification and repair report according to the second original data, the repair data and the abnormal information.

[0098] In some embodiments, the report generation module generates reports for daily checked and repaired data and provides multiple report formats, such as PDF, Excel, etc. The report content includes data comparison before and after repair, abnormal conditions during the repair process, etc. Further, the report data is sent to a designated mailbox and group for relevant managers to observe the online data at any time.

[0099] By generating reports including data comparison before and after repair and abnormal situations during the repair process, relevant managers can observe online data at any time.

[0100] The data repair system of Example 1 of the present invention can solve the problem of data inconsistency between different data sources by extracting feature data of the data to be checked from different data sources and generating labels, and then using a machine learning algorithm to automatically predict and repair the data in different data sources, thereby reducing manual intervention and improving the accuracy and efficiency of data checking and repair.

[0101] [Example 2]

[0102] Figure 2 4 is a flow chart of a data repair method according to Embodiment 2 of the present invention.

[0103] like Figure 2 As shown, in Embodiment 2 of the present invention, the data repair method may include: step S21, step S22, step S23, step S24, step S25 and step S26, and the above steps are described in detail below.

[0104] In step S21, first original data is collected from a first data source and a second data source. In some embodiments, the first data source is a self-accounting system, and the second data source is a foreign accounting system; the first original data is transaction data, and the transaction data includes: loan application information, interest rate information, loan note information, repayment plan information, repayment flow information, and provision information.

[0105] In step S22, the first original data is stored.

[0106] In some implementations, storing the first original data includes: using a relational database MySQL to store the first original data as a table structure, thereby ensuring the reliability and scalability of data storage.

[0107] In a further embodiment, storing the first original data includes: allocating the first original data to a corresponding database according to the product number; storing the first original data in a corresponding storage table in the database according to the loan note number, the storage table including: a loan table, a repayment table, a repayment plan table, a transaction table, and a provision table; associating the storage tables according to the loan note number. Thus, the original data is distributed and stored in the database table according to the product number and the loan note number, which can realize fast query and accurate matching of accounting data.

[0108] In step S23, the data to be verified is extracted from the first original data according to the first information, and the feature data is extracted from the data to be verified. In some embodiments, when the first data source and the second data source are a self-accounting system and an external accounting system, respectively, the data to be verified is extracted from the first original data according to the loan note number.

[0109] In step S24, a label is generated according to the relationship between the feature data and a preset threshold, and the label includes an abnormal label and a normal label.

[0110] In some embodiments, generating a label based on the relationship between the feature data and a preset threshold includes: calculating the absolute value of the difference between the first feature data in the self-accounting system and the second feature data in the other accounting system as a difference value; when the difference value is greater than the preset threshold, generating an abnormal label; when the difference value is less than or equal to the preset threshold, generating a normal label.

[0111] In step S25, the preset model is trained and verified according to the feature data and labels to obtain a trained verification model.

[0112] In step S26, the collected second original data is predicted according to the verification model, and when the prediction value output by the verification model is an abnormal label, the second original data is repaired according to a predefined repair rule to obtain repaired data.

[0113] The above-mentioned data repair method of Example 2 of the present invention is adopted to solve the problem of data inconsistency between different data sources, reduce manual intervention, and improve the accuracy and efficiency of data verification and repair by extracting feature data of the data to be verified in different data sources and generating labels, and then using machine learning algorithms to automatically predict and repair the data in different data sources.

[0114] In a further embodiment, the data repair method further includes: detecting and processing abnormal information generated during the data repair process; generating a data verification and repair report based on the second original data, the repair data and the abnormal information. By generating a report including data comparison before and after the repair and abnormal conditions during the repair process, it is convenient for relevant management personnel to observe the online data situation at any time.

[0115] [Example 3]

[0116] Figure 3 4 is a flow chart of a data repair method according to Embodiment 3 of the present invention.

[0117] like Figure 3 As shown, in embodiment 3 of the present invention, the data repair method includes the following steps:

[0118] (1) The data collection module collects initial data from various channel systems through the API interface and from the internal system of the financial company through the SQL query method.

[0119] (2) The data storage module stores the initial data collected by the data collection module.

[0120] (3) The data verification and repair module extracts the initial data from the data storage module, and performs feature extraction, label generation, model training and verification based on the initial data to obtain a trained verification model, and then automatically detects the difference between the newly collected initial data and performs data repair based on the verification model, and updates the repaired data to the data for storage. The specific implementation method is the same as the aforementioned embodiment 1, so it will not be repeated here.

[0121] (4) The exception handling module detects and processes the exception information generated by the data verification and repair module during the data repair process.

[0122] (5) The report generation module generates report content including data comparison before and after repair and abnormal conditions during the repair process.

[0123] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation modes of the present invention or some parts of the implementation modes.

[0124] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium on which computer-readable instructions or programs are stored. When the computer-readable instructions or programs are executed by a processor, the computer performs the following operations: the operations include the steps included in the data repair method described in any of the above embodiments, which are not repeated here. The storage medium may include, for example, an optical disk, a hard disk, a floppy disk, a flash memory, a magnetic tape, etc.

[0125] In addition, an embodiment of the present invention further provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions or programs, wherein the one or more computer-readable instructions or programs can implement the data repair method described in any of the above embodiments when executed by the processor. The computer device can be, for example, a server, a desktop computer, a notebook computer, a tablet computer, etc.

[0126] The present invention also provides a computer program product including a computer program, wherein the computer program includes program code for executing the data repair method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data repair method provided by the embodiment of the present disclosure.

[0127] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention. Therefore, the protection scope of the present invention shall be subject to the claims.

Claims

1. A data repair system, characterized in that: The data repair system comprises: A data collection module, used for collecting first original data from a first data source and a second data source; A data storage module, used for storing the first original data; The data verification and repair module is used to perform the following operations: extracting the data to be checked from the first original data according to the first information, and extracting feature data from the data to be checked; Generate a label according to the relationship between the feature data and a preset threshold, wherein the label includes an abnormal label and a normal label; Training and verifying the preset model according to the feature data and labels to obtain a trained verification model; The collected second original data is predicted according to the verification model, and when the prediction value output by the verification model is an abnormal label, the second original data is repaired according to a predefined repair rule to obtain repaired data.

2. A data repair method, characterized in that: The data repair method comprises: Collecting first raw data from a first data source and a second data source; Storing the first original data; extracting the data to be checked from the first original data according to the first information, and extracting feature data from the data to be checked; Generate a label according to the relationship between the feature data and a preset threshold, wherein the label includes an abnormal label and a normal label; Training and verifying the preset model according to the feature data and labels to obtain a trained verification model; The collected second original data is predicted according to the verification model, and when the prediction value output by the verification model is an abnormal label, the second original data is repaired according to a predefined repair rule to obtain repaired data.

3. The data repair method according to claim 2, characterized in that: The data repair method further comprises: Detect and process abnormal information generated during data repair; A data verification and repair report is generated based on the second original data, the repair data and the abnormal information.

4. The data repair method according to claim 2, characterized in that: The first data source is a self-accounting system, and the second data source is an external accounting system; The first original data and the second original data are transaction data, and the transaction data includes: loan application information, interest rate information, loan note information, repayment plan information, repayment flow information, and provision information; The first information is the loan note number.

5. The data repair method according to claim 4, characterized in that: Generating a label according to the relationship between the feature data and a preset threshold comprises: Calculating an absolute value of a difference between the first characteristic data in the self-accounting system and the second characteristic data in the other-accounting system as a difference value; When the difference value is greater than a preset threshold, an abnormal label is generated; When the difference value is less than or equal to a preset threshold, a normal label is generated.

6. The data repair method according to claim 4, characterized in that: Storing the first original data includes: using the relational database MySQL to store the first original data as a table structure.

7. The data repair method according to claim 6, characterized in that: Storing the first original data further includes: Allocating the first original data to a corresponding database according to the product number; Storing the first original data in a corresponding storage table in the database according to the loan note number, wherein the storage table includes: a loan table, a repayment table, a repayment plan table, a transaction table, and a provision table; The storage table is associated according to the loan note number.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that: The computer-readable instructions are executed by a processor to implement the data repair method according to any one of claims 2 to 7.

9. A computer device comprising a memory and a processor, The memory stores computer-readable instructions, characterized in that: The processor executes the computer-readable instructions to implement the data repair method according to any one of claims 2 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data repair method according to any one of claims 2 to 7 is implemented.

Images