Multi-file data processing method and related device

Abstract

The invention discloses a multi-file data processing method and a related device, and the method comprises the steps: obtaining a file list and n file blocks from m original files, merging the n file blocks according to the file list, and obtaining the m original files. And respectively processing each piece of sub-data in the target original file according to a bank data quality rule to obtain standard data, and respectively taking the m original files as the target original files to obtain standard data respectively corresponding to the m original files. And obtaining a data demand table for satisfying the plurality of data demands, obtaining a target data demand, and processing the standard data according to the data demand table and the target data demand to obtain a data report satisfying the target data demand. Therefore, the efficiency and accuracy of processing the original file to obtain the standard data are improved, the intuition and analyzability of the standard data are ensured to be improved, and the efficiency and accuracy in the process of applying the standard data are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-file data processing method and related devices. Background Art

[0002] With the rapid development of information technology, data processing has become an indispensable part of all walks of life, especially in the era of big data, which requires data processing of massive amounts of information. For example, in the financial industry, in order to improve the quality of regulatory data, banks need to verify the regulatory data of multiple files and perform other data processing to obtain standard data that can be used to make demand tables.

[0003] For multi-file data, bank employees usually verify and check each file one by one, which is prone to errors, resulting in low accuracy and efficiency of data processing. Summary of the invention

[0004] In response to the above problems, the present application provides a multi-file data processing method and related devices to solve the problem of low data processing accuracy and efficiency.

[0005] Based on this, this application discloses the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for processing multi-file data, the method comprising:

[0007] Obtain a file list and n file blocks from m original files, wherein the original file includes at least one file block, and the file list is used to represent the attribution relationship between the n file blocks and the m original files, wherein n is a positive integer, and m is a positive integer less than or equal to n;

[0008] Merging the n file blocks according to the file list to obtain the m original files;

[0009] Processing each sub-data in the target original file according to the bank data quality rule to obtain standard data, wherein the target original file is one of the m original files, and the bank data quality rule is used to identify abnormal fields in the sub-data and modify the abnormal fields, and the abnormal fields are not included in each sub-data included in the standard data;

[0010] The m original files are respectively used as the target original files, and the standard data corresponding to the m original files are obtained;

[0011] Obtaining a data requirement table for satisfying a plurality of data requirements, the data requirement table comprising a plurality of data processing rules, the data processing rules being used to process data into data satisfying the data requirements corresponding to the data processing rules, different data processing rules corresponding to different data requirements;

[0012] Obtain target data requirements;

[0013] The standard data is processed according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

[0014] In a second aspect, an embodiment of the present application provides a multi-file data processing device, the device comprising: an acquisition unit, a merging unit and a processing unit;

[0015] The acquisition unit is used to acquire a file list and n file blocks from m original files, wherein the original files include at least one file block, and the file list is used to represent the attribution relationship between the n file blocks and the m original files, wherein n is a positive integer, and m is a positive integer less than or equal to n;

[0016] The merging unit is used to merge the n file blocks according to the file list to obtain the m original files;

[0017] The processing unit is used to process each sub-data in the target original file according to the bank data quality rule to obtain standard data, the target original file is one of the m original files, the bank data quality rule is used to identify abnormal fields in the sub-data and modify the abnormal fields, and the abnormal fields are not included in each sub-data included in the standard data;

[0018] The processing unit is further used to use the m original files as the target original files respectively, and obtain the standard data corresponding to the m original files respectively;

[0019] The acquisition unit is further used to acquire a data requirement table for satisfying multiple data requirements, wherein the data requirement table includes multiple data processing rules, and the data processing rules are used to process data into data that satisfies the data requirements corresponding to the data processing rules, and different data processing rules correspond to different data requirements;

[0020] The acquisition unit is also used to acquire target data requirements;

[0021] The processing unit is further used to process the standard data according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

[0022] In a third aspect, an embodiment of the present application provides a computer device, wherein the computer device includes a processor and a memory:

[0023] The memory is used to store a computer program and transmit the computer program to the processor;

[0024] The processor is configured to execute the method according to the first aspect of claim 1 according to the computer program.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the first aspect above.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when executed on a computer device, enables the computer device to execute the method described in the first aspect above.

[0027] It can be seen from the above technical solutions that the present application has at least the following beneficial effects:

[0028] Obtain a file list and n file blocks from m original files, merge the n file blocks according to the file list to obtain m original files, and merge the file blocks belonging to the same original file into the same original file to facilitate subsequent data processing. Process each sub-data in the target original file according to the bank data quality rules to obtain standard data, use the m original files as target original files, and obtain the standard data corresponding to the m original files, so as to identify the abnormal fields in each sub-data and modify the abnormal fields according to the bank data quality rules, and obtain standard data without abnormal fields, thereby improving the efficiency and accuracy of data processing. Obtain a data requirement table for meeting multiple data requirements, obtain the target data requirements, process the standard data according to the data requirement table and the target data requirements, and obtain a data report that meets the target data requirements, thereby improving the intuitiveness and analyzability of the standard data, and improving the efficiency and accuracy of the standard data application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1A flowchart of a multi-file data processing method provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the structure of a multi-file data processing device provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0034] In the financial industry, the data generated by banks in their business often has the characteristics of large data volume and many files. In order to improve the quality of regulatory data, banks need to verify the regulatory data of multiple files and perform other data processing to ensure the accuracy and integrity of the data. This data processing work not only involves the management of a large number of files, but also includes a series of complex processing steps such as data cleaning, anomaly detection, and data standardization. The traditional multi-file data processing method mainly relies on bank employees to manually check and verify each file one by one. This method has the following significant disadvantages:

[0035] (1) Inefficiency: Manual processing is slow and cannot cope with the demand for rapid processing of large-scale data.

[0036] (2) High error rate: Manual operations are prone to errors, especially when processing large amounts of data, and it is easy to miss or misjudge abnormal data.

[0037] (3) Poor consistency: The operating standards of different employees may be inconsistent, resulting in inconsistent data processing results.

[0038] (4) High cost: It requires a large amount of human resources, which increases the bank’s operating costs.

[0039] Based on this, the embodiment of the present application provides a multi-file data processing method and related devices, obtains a file list and n file blocks from m original files, merges the n file blocks according to the file list, obtains m original files, and merges the file blocks belonging to the same original file into the same original file, so as to facilitate subsequent data processing. Each sub-data in the target original file is processed according to the bank data quality rules to obtain standard data, and the m original files are respectively used as the target original files to obtain the standard data corresponding to the m original files, so as to identify the abnormal fields in each sub-data through the bank data quality rules and modify the abnormal fields to obtain standard data without abnormal fields, thereby improving the efficiency and accuracy of data processing. Obtain a data requirement table for meeting multiple data requirements, obtain the target data requirements, process the standard data according to the data requirement table and the target data requirements, and obtain a data report that meets the target data requirements, thereby improving the intuitiveness and analyzability of the standard data, and improving the efficiency and accuracy in the process of applying the standard data.

[0040] The multi-file data processing method provided in this application can be applied to computer devices with data processing capabilities, such as terminal devices and servers. Specifically, the terminal device can be a desktop computer, a laptop computer, a mobile phone, a tablet computer, etc.; the server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0041] See also Figure 1 , which is a flow chart of the multi-file data processing method provided by the embodiment of the present application. For the convenience of description, the following embodiment is introduced by taking the execution subject of the multi-file data processing method as a server as an example. Figure 1 As shown, the multi-file data processing method includes S101-S107.

[0042] S101: Obtain a file list and n file blocks from m original files.

[0043] The original file includes at least one file block, and the file list is used to represent the attribution relationship between n file blocks and m original files, where n is a positive integer and m is a positive integer less than or equal to n.

[0044] The objects of data processing are usually multiple original files, and the received data are limited by the data transmission method to be multiple file blocks. These file blocks are obtained by dividing or splitting the original files. Multiple file blocks or one file block belongs to one original file, and the ownership relationship between these file blocks and files is recorded in the form of a file list.

[0045] For example, 200 file blocks are obtained, each of which has a size of 2 Gigabytes (GB), and the 200 file blocks come from 60 original files respectively. The file list includes the ownership relationship between the file blocks and the original files.

[0046] S102: Merge n file blocks according to the file list to obtain m original files.

[0047] According to the attribution relationship of the file list, the file blocks belonging to the same file list are merged in turn, thereby obtaining m original files. The file list may also include basic file information such as file type, number of files, and file name, which is used to verify the integrity of the file block or the original file.

[0048] S103: Process each sub-data in the target original file according to the bank data quality rules to obtain standard data.

[0049] Among them, the target original file is an original file among m original files, the sub-data is a data record in the original file, a sub-data can be a field or include multiple fields, and a sub-data can include user name, user contact information, user deposit balance and other information in banking business.

[0050] Abnormal fields are data items that do not comply with bank data quality rules. Abnormal fields may be format errors, values ​​outside the default data range, or inconsistent with other data. Bank data quality rules can be used to identify abnormal fields in sub-data and modify them to obtain standard data that does not include abnormal fields.

[0051] The embodiment of the present application does not limit the type of bank data quality rules. Two types of bank data quality rules are described below. Bank data quality rules include data association rules and default value rules. The data association rules are used to process sub-data with association relationships separately, and the default value rules are used to process sub-data with a data default range in the original file.

[0052] First, the data processing method of the data association rule type is explained, see A1-A4:

[0053] A1: Determine the first sub-data in the target original file and the second sub-data in the target original file according to the data association rule.

[0054] The first sub-data and the second sub-data have an association relationship, and the processing order of the first sub-data precedes the processing order of the second sub-data. The data association rule is a bank data quality rule, including the association relationship of the sub-data, the processing order of the sub-data, and the processing rules corresponding to each sub-data.

[0055] A2: Perform a first process on the first sub-data according to the data association rule.

[0056] A3: If the first processing is successful, the second processing is performed on the second sub-data according to the data association rule to obtain standard data.

[0057] A4: If the first processing fails or the second processing fails, a second alarm message is issued.

[0058] For example, the target original file file1.csv includes two sub-data from different fields, the first sub-data ("Beijing") and the second sub-data (100001), where the field corresponding to the first sub-data is City and the field corresponding to the second sub-data is PostalCode. The data association rule stipulates that the PostalCode field can only be processed when the City field is valid (i.e., the first processing is successful).

[0059] The first sub-data "Beijing" is processed in the first way to determine whether it is empty and whether it belongs to the database corresponding to the City field. If it is not empty and belongs to the database, the first processing is successful. The second sub-data "100001" is processed in the second way to determine whether it is empty and whether it belongs to the postal code database of Beijing. If it is not empty and belongs to the postal code database, the second processing is successful and standard data is obtained. If it is not empty but the postal code has extra special characters, the extra special characters can be deleted for correction to obtain standard data.

[0060] If the first sub-data or the second sub-data is empty, the first processing fails or the second processing fails, and a second alarm message is sent. The second alarm message includes the data association rule, and the first sub-data and the second sub-data corresponding to the data association rule.

[0061] Data association rules ensure that sub-data with association relationships can be processed in the correct order, improving the accuracy and reliability of data processing. At the same time, through a timely alarm mechanism, problems in data processing can be quickly discovered and solved, further improving the quality of standard data.

[0062] The following is an explanation of how to process data with default value rules as the type, see B1-B2:

[0063] B1: According to the default value rule, determine the data default range corresponding to each sub-data.

[0064] The default range of data is the allowable value range of the field type corresponding to the sub-data. The default range of sub-data of different field types is different. For example, the target original file file1.csv includes three sub-data, corresponding to the fields Age, Salary and YearsOfExperience. Depending on the field type, the default ranges of the data corresponding to these three sub-data can be 18-120, 0-10000000 and 0-60.

[0065] B2: According to the default data range corresponding to each sub-data, each sub-data is processed respectively to obtain standard data.

[0066] Among them, each sub-data of the standard data belongs to the corresponding data default range respectively.

[0067] For example, the default range of the data corresponding to the Age field is 18-70, and there is a sub-data belonging to the Age field with a value of 17, which can be corrected to 18 to complete the processing and obtain standard data.

[0068] As an implementation manner, a warning message may be issued for the sub-data that does not belong to the data default range, and the warning message includes the data default range and the sub-data.

[0069] The default value rules ensure that each sub-data complies with its corresponding data default range, which improves the accuracy and consistency of the data and reduces human errors.

[0070] S104: taking the m original files as target original files respectively, and obtaining standard data corresponding to the m original files respectively.

[0071] S105: Obtain a data requirement table for satisfying multiple data requirements.

[0072] The data requirement table is a rule table preset based on expert experience, including multiple data processing rules. The data processing rules are used to process data into data that meets the data requirements corresponding to the data processing rules. Different data processing rules correspond to different data requirements.

[0073] For example, take the three data processing rules in the data requirement table as an example:

[0074] Rule 1: Format_Check, ensure that all date fields are in the format of YYYY-MM-DD, and the corresponding requirement is Reporting.

[0075] Rule 2: Range_Check, ensures that the salary field is between 0 and 1,000,000, and the corresponding requirement is Compliance.

[0076] Rule 3: Data_Conversion, converts all currency values ​​into US dollars, and the corresponding requirement is International_Report.

[0077] S106: Obtain target data requirements.

[0078] The target data requirement is one of multiple data requirements corresponding to multiple data processing rules.

[0079] S107: Processing the standard data according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

[0080] According to the target data requirements, the data processing rules corresponding to the target data requirements can be determined from the data requirements table, so that the standard data can be processed according to the data processing rules to obtain a data report that meets the target data requirements.

[0081] Take the three data processing rules in the above example:

[0082] If the target data requirement is to generate reports (Reporting), the Format_Check rule is applied.

[0083] If the target data requirement is compliance check, the Range_Check rule is applied.

[0084] If the target data requirement is an international report (International_Report), the Data_Conversion rule is applied.

[0085] It can be seen from the above technical solution that a file list and n file blocks from m original files are obtained, and the n file blocks are merged according to the file list to obtain m original files, so that the file blocks belonging to the same original file are merged into the same original file, which is convenient for subsequent data processing. Each sub-data in the target original file is processed separately according to the bank data quality rules to obtain standard data, and the m original files are respectively used as target original files to obtain the standard data corresponding to the m original files, so that the abnormal fields in each sub-data are identified and modified according to the bank data quality rules, and standard data without abnormal fields is obtained, thereby improving the efficiency and accuracy of data processing. A data requirement table for meeting multiple data requirements is obtained, and the target data requirements are obtained. The standard data is processed according to the data requirement table and the target data requirements to obtain a data report that meets the target data requirements, thereby improving the intuitiveness and analyzability of the standard data, and improving the efficiency and accuracy in the process of applying the standard data.

[0086] The sources of bank data quality rules are varied, which may come from inside or outside the branch. The present application embodiment provides a processing method for multi-source bank data quality rules, see C1-C3:

[0087] C1: Verify the branch data quality rules according to the head office data quality rules and obtain the verification results.

[0088] The head office data quality rule is a bank data quality rule formulated by the head office, the China Banking Regulatory Commission, the Financial Regulatory Bureau or other external departments of the branch, and is applicable to the regulatory data of the entire banking system. As an implementation method, the head office data quality rule can be updated synchronously. The branch data quality rule is a bank data quality rule that is internal to the branch or already exists in the branch.

[0089] The data quality rules of the branch bank are verified according to the data quality rules of the head office, and a verification result is obtained to indicate whether the data quality rules of the branch bank conflict with the data quality rules of the head office.

[0090] C2: If the verification result indicates no conflict, each sub-data in the target original file is processed according to the head office data quality rules and the branch bank data quality rules to obtain standard data.

[0091] C3: If the verification result indicates a conflict, a first warning message is issued.

[0092] The first warning information includes the head office data quality rule corresponding to the verification result and the branch bank data quality rule corresponding to the verification result.

[0093] The above method ensures the consistency of branch data quality rules from inside and outside the branch with the head office data quality rules, avoids data processing errors caused by rule conflicts, and through a timely alarm mechanism, can quickly discover and resolve rule conflicts, further improving the accuracy and reliability of data processing.

[0094] In the process of multi-file data processing, the original files are usually processed by the bank data quality rules of the analysis library and the governance library. However, when the standard data is obtained after processing and the report is generated, abnormal data is sometimes found. In view of this situation, the embodiment of the present application provides a data backtracking method, see D1-D3:

[0095] D1: Store m original files into the standard database, and establish corresponding relationships between each original file and the corresponding standard data.

[0096] The m original files are stored in the standard database, so as to back up the original files, and after the original files are processed by the bank data quality rules to obtain the standard data, the corresponding relationship between the original files and the standard data is established. Specifically, the corresponding relationship between the sub-data in the standard data corresponding to each sub-data in the original file is established respectively.

[0097] As an implementation method, the raw data in the raw file can be visually analyzed. Specifically, the raw file in the standard database can be loaded, and after data cleaning, various statistical indicators in the raw file can be calculated to draw a data distribution diagram or data distribution table about each statistical indicator.

[0098] D2: In response to receiving the backtracking request, determine the original data corresponding to the abnormal standard data in the backtracking request according to the corresponding relationship.

[0099] The original data is the data in the original file, and the backtracking request is a request initiated by the user or system to find the original data corresponding to the abnormal data in the standard data. The original data can be located and determined through the correspondence between the abnormal data and its corresponding original data, and the data can be backtracked.

[0100] D3: Displays the original data.

[0101] This ensures the traceability of the original data, and enables the rapid retrieval of the corresponding original data when anomalies are found in standard data or data reports, helping users or systems to conduct further analysis and corrections.

[0102] The present application embodiment also provides a method for verifying and reporting a data report, see E1-E4:

[0103] E1: Obtain a report requirement table for satisfying multiple report requirements.

[0104] The report requirement table includes multiple report processing rules. The report processing rules are used to process the report into a report that meets the report requirements corresponding to the report processing rules. Different report processing rules correspond to different report requirements. Report requirements are requirements for the format, size, name and other elements of the data report.

[0105] E2: Obtain target report requirements.

[0106] The target report requirement is one of the multiple report requirements corresponding to the multiple report processing rules.

[0107] E3: Process the data report according to the report requirement table and the target report requirements to obtain a final data report that meets the target report requirements.

[0108] According to the target report requirement, the report processing rules corresponding to the target report requirement can be determined from the report requirement table, so that the data report is processed according to the report processing rules to obtain the final data report that meets the target data requirement.

[0109] E4: Submit final data report.

[0110] The final data report is the data report processed by the report processing rules.

[0111] In this way, the data report can be verified and the final data report can be generated to meet the specific report requirements of users or business systems, and the automation level of data report verification and reporting can be improved.

[0112] The data requirement may be a requirement recorded in natural language. The embodiment of the present application provides a method for determining data processing rules according to the data requirement, see F1-F2:

[0113] F1: Get the rule model and multiple data requirements.

[0114] The rule model is a pre-training model (PTM) used to generate data processing rules corresponding to data requirements. It can be trained through data requirement samples and data processing rule labels.

[0115] F2: Input multiple data requirements into the rule model respectively, and generate data processing rules corresponding to each data requirement through the rule model.

[0116] For example, the rule model can be used to generate the corresponding data processing rule (database type) based on the data requirement of "generate a user salary report summarized by region", "SELECT Region, SUM (Salary) AS Total_Salary FROM Users GROUP BY Region ORDER BY Region;".

[0117] Therefore, converting the data requirements recorded in natural language into specific data processing rules not only improves the automation of data processing, but also reduces the workload of manually writing rules, ensuring the accuracy and consistency of data processing. The use of rule models enables the system to flexibly respond to various data requirements and improves the efficiency and reliability of data processing.

[0118] See also Figure 2 , Figure 2 A multi-file data processing device provided in an embodiment of the present application, the device 200 includes: an acquisition unit 201, a merging unit 202 and a processing unit 203;

[0119] The acquisition unit is used to acquire a file list and n file blocks from m original files, wherein the original files include at least one file block, and the file list is used to represent the attribution relationship between the n file blocks and the m original files, wherein n is a positive integer, and m is a positive integer less than or equal to n;

[0120] The merging unit is used to merge the n file blocks according to the file list to obtain the m original files;

[0121] The processing unit is used to process each sub-data in the target original file according to the bank data quality rule to obtain standard data, the target original file is one of the m original files, the bank data quality rule is used to identify abnormal fields in the sub-data and modify the abnormal fields, and the abnormal fields are not included in each sub-data included in the standard data;

[0122] The processing unit is further used to use the m original files as the target original files respectively, and obtain the standard data corresponding to the m original files respectively;

[0123] The acquisition unit is further used to acquire a data requirement table for satisfying multiple data requirements, wherein the data requirement table includes multiple data processing rules, and the data processing rules are used to process data into data that satisfies the data requirements corresponding to the data processing rules, and different data processing rules correspond to different data requirements;

[0124] The acquisition unit is also used to acquire target data requirements;

[0125] The processing unit is further used to process the standard data according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

[0126] It can be seen from the above technical solution that the acquisition unit acquires the file list and n file blocks from m original files, and the merging unit merges the n file blocks according to the file list to obtain m original files, thereby merging the file blocks belonging to the same original file into the same original file, which is convenient for subsequent data processing. The processing unit processes each sub-data in the target original file according to the bank data quality rules to obtain standard data, and takes the m original files as the target original files respectively to obtain the standard data corresponding to the m original files respectively, thereby identifying the abnormal fields in each sub-data through the bank data quality rules and modifying the abnormal fields to obtain standard data without abnormal fields, thereby improving the efficiency and accuracy of data processing. The acquisition unit acquires a data requirement table for meeting multiple data requirements and acquires the target data requirements. The processing unit processes the standard data according to the data requirement table and the target data requirements to obtain a data report that meets the target data requirements, thereby improving the intuitiveness and analyzability of the standard data, and improving the efficiency and accuracy in the process of applying the standard data.

[0127] As a possible implementation manner, the processing unit is specifically used for:

[0128] Verifying the branch bank data quality rule according to the head office data quality rule to obtain a verification result, wherein the verification result is used to indicate whether the branch bank data quality rule conflicts with the head office data quality rule;

[0129] If the verification result indicates no conflict, each sub-data in the target original file is processed according to the head office data quality rule and the branch bank data quality rule to obtain standard data;

[0130] If the verification result indicates a conflict, a first warning message is issued, wherein the first warning message includes the head office data quality rule corresponding to the verification result and the branch bank data quality rule corresponding to the verification result.

[0131] As a possible implementation manner, the device further includes a tracing unit:

[0132] The m original files are stored in the standard database, and corresponding relationships between the original files and the corresponding standard data are established respectively;

[0133] In response to receiving a backtracking request, determining, according to the corresponding relationship, original data corresponding to the abnormal standard data in the backtracking request;

[0134] The raw data is displayed.

[0135] As a possible implementation manner, the device further includes a reporting unit:

[0136] Acquire a report requirement table for satisfying a plurality of report requirements, wherein the report requirement table includes a plurality of report processing rules, wherein the report processing rules are used to process a report into a report satisfying the report requirements corresponding to the report processing rules, and different report processing rules correspond to different report requirements;

[0137] Obtain target report requirements;

[0138] Processing the data report according to the report requirement table and the target report requirement to obtain a final data report that meets the target report requirement;

[0139] Submit the final data report.

[0140] As a possible implementation, the bank data quality rules include data association rules and default value rules. The data association rules are used to process sub-data with association relationships respectively, and the default value rules are used to process sub-data with data default ranges in the original file.

[0141] As a possible implementation manner, the processing unit is specifically used for:

[0142] Determine, according to the data association rule, the first sub-data in the target original file and the second sub-data in the target original file, the first sub-data and the second sub-data have an association relationship, and the processing order of the first sub-data is prior to the processing order of the second sub-data;

[0143] performing a first process on the first sub-data according to the data association rule;

[0144] If the first processing is successful, performing a second processing on the second sub-data according to the data association rule to obtain the standard data;

[0145] If the first processing fails or the second processing fails, a second alarm message is issued, wherein the second alarm message includes the data association rule, and the first sub-data and the second sub-data corresponding to the data association rule.

[0146] As a possible implementation manner, the processing unit is specifically used for:

[0147] According to the default value rule, determine the data default ranges corresponding to the respective sub-data;

[0148] According to the data default ranges respectively corresponding to the sub-data, the sub-data are processed respectively to obtain the standard data, and the sub-data of the standard data belong to the corresponding data default ranges respectively.

[0149] As a possible implementation, the data processing rule is obtained in the following manner:

[0150] Acquire a rule model and a plurality of the data requirements, wherein the rule model is used to generate data processing rules corresponding to the data requirements;

[0151] The plurality of data requirements are respectively input into the rule model, and data processing rules corresponding to the respective data requirements are generated through the rule model.

[0152] The present application also provides a computer device, see Figure 3 , which shows a structural diagram of a computer device provided in an embodiment of the present application, such as Figure 3 As shown, the device includes a memory 310 and a processor 320:

[0153] The memory 310 is used to store program codes and transmit the program codes to the processor;

[0154] The processor 320 is used to execute any one of the loan data processing methods provided in the above embodiments according to the instructions in the program code.

[0155] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is used to execute any one of the loan data processing methods provided in the above embodiments.

[0156] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the loan data processing method provided in various optional implementations of the above aspects.

[0157] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0158] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0159] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0160] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0161] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0162] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing multi-file data, characterized in that: The method comprises: Obtain a file list and n file blocks from m original files, wherein the original file includes at least one file block, and the file list is used to represent the attribution relationship between the n file blocks and the m original files, wherein n is a positive integer, and m is a positive integer less than or equal to n; Merging the n file blocks according to the file list to obtain the m original files; Processing each sub-data in the target original file according to the bank data quality rule to obtain standard data, wherein the target original file is one of the m original files, and the bank data quality rule is used to identify abnormal fields in the sub-data and modify the abnormal fields, and the abnormal fields are not included in each sub-data included in the standard data; The m original files are respectively used as the target original files, and the standard data corresponding to the m original files are obtained; Obtaining a data requirement table for satisfying a plurality of data requirements, the data requirement table comprising a plurality of data processing rules, the data processing rules being used to process data into data satisfying the data requirements corresponding to the data processing rules, different data processing rules corresponding to different data requirements; Obtain target data requirements; The standard data is processed according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

2. The method according to claim 1, characterized in that: The bank data quality rules include the head office data quality rules and the branch bank data quality rules. The sub-data in the target original file are processed according to the bank data quality rules to obtain standard data, including: Verifying the branch bank data quality rule according to the head office data quality rule to obtain a verification result, wherein the verification result is used to indicate whether the branch bank data quality rule conflicts with the head office data quality rule; If the verification result indicates no conflict, each sub-data in the target original file is processed according to the head office data quality rule and the branch bank data quality rule to obtain standard data; If the verification result indicates a conflict, a first warning message is issued, wherein the first warning message includes the head office data quality rule corresponding to the verification result and the branch bank data quality rule corresponding to the verification result.

3. The method according to claim 1, characterized in that The method further comprises: The m original files are stored in the standard database, and corresponding relationships between the original files and the corresponding standard data are established respectively; In response to receiving a backtracking request, determining, according to the corresponding relationship, original data corresponding to the abnormal standard data in the backtracking request; The raw data is displayed.

4. The method according to claim 1, characterized in that The method further comprises: Acquire a report requirement table for satisfying a plurality of report requirements, wherein the report requirement table includes a plurality of report processing rules, wherein the report processing rules are used to process a report into a report satisfying the report requirements corresponding to the report processing rules, and different report processing rules correspond to different report requirements; Obtain target report requirements; Processing the data report according to the report requirement table and the target report requirement to obtain a final data report that meets the target report requirement; Submit the final data report.

5. The method according to claim 1, characterized in that: The bank data quality rules include data association rules and default value rules. The data association rules are used to process sub-data with association relationships respectively, and the default value rules are used to process sub-data with data default ranges in the original file.

6. The method according to claim 5, characterized in that If the bank data quality rule is the data association rule, and the data association rule includes the association relationship of the sub-data, the sub-data data processing order and the processing rules corresponding to each of the sub-data, then the sub-data in the target original file are processed according to the bank data quality rule to obtain standard data, including: Determine, according to the data association rule, the first sub-data in the target original file and the second sub-data in the target original file, the first sub-data and the second sub-data have an association relationship, and the processing order of the first sub-data is prior to the processing order of the second sub-data; performing a first process on the first sub-data according to the data association rule; If the first processing is successful, performing a second processing on the second sub-data according to the data association rule to obtain the standard data; If the first processing fails or the second processing fails, a second alarm message is issued, wherein the second alarm message includes the data association rule, and the first sub-data and the second sub-data corresponding to the data association rule.

7. The method according to claim 5, characterized in that If the bank data quality rule is the default value rule, then the sub-data in the target original file are processed according to the bank data quality rule to obtain standard data, including: According to the default value rule, determine the data default ranges corresponding to the respective sub-data; According to the data default ranges respectively corresponding to the sub-data, the sub-data are processed respectively to obtain the standard data, and the sub-data of the standard data belong to the corresponding data default ranges respectively.

8. The method according to claim 1, characterized in that The data processing rules are obtained in the following way: Acquire a rule model and a plurality of the data requirements, wherein the rule model is used to generate data processing rules corresponding to the data requirements; The plurality of data requirements are respectively input into the rule model, and data processing rules corresponding to the respective data requirements are generated through the rule model.

9. A multi-file data processing device, characterized in that: The device comprises: an acquisition unit, a merging unit and a processing unit; The acquisition unit is used to acquire a file list and n file blocks from m original files, wherein the original files include at least one file block, and the file list is used to represent the attribution relationship between the n file blocks and the m original files, wherein n is a positive integer, and m is a positive integer less than or equal to n; The merging unit is used to merge the n file blocks according to the file list to obtain the m original files; The processing unit is used to process each sub-data in the target original file according to the bank data quality rule to obtain standard data, the target original file is one of the m original files, the bank data quality rule is used to identify abnormal fields in the sub-data and modify the abnormal fields, and the abnormal fields are not included in each sub-data included in the standard data; The processing unit is further used to use the m original files as the target original files respectively, and obtain the standard data corresponding to the m original files respectively; The acquisition unit is further used to acquire a data requirement table for satisfying multiple data requirements, wherein the data requirement table includes multiple data processing rules, and the data processing rules are used to process data into data that satisfies the data requirements corresponding to the data processing rules, and different data processing rules correspond to different data requirements; The acquisition unit is also used to acquire target data requirements; The processing unit is further used to process the standard data according to the data requirement table and the target data requirement to obtain a data report table that meets the target data requirement.

10. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 7 according to the computer program.

Images