A data processing method and device

Abstract

The present invention discloses a data processing method and device, which relates to the field of data processing, wherein the method comprises: initiating N data extraction, conversion and loading ETL processes for a source end distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in a data warehouse; when there are M failed ETL processes in the execution process of the N ETL processes, the N ETL processes are re-initiated; when the re-initiated N ETL processes are successfully executed, the first target table generated by the re-initiated N ETL processes in the data warehouse is obtained, and the first result table generated by statistically summarizing the first target table is obtained; when there is data missing in the first result table, the first result table is repaired. The method can be applied to financial technology (Fintech).

Description

Technical Field

[0001] The present invention relates to the field of data processing in financial technology (Fintech), and in particular to a data processing method and device. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed computing, blockchain, artificial intelligence, etc.) are being applied in the financial field, and the traditional financial industry is gradually transforming into financial technology (Fintech). At present, in the field of financial technology, due to the rapid development of information technologies such as mobile Internet, cloud computing, and the Internet of Things, more and more data are being generated and are increasingly used in various information processing. Taking banks as an example, the traditional core system of banks stores account data, transaction flows, etc. in a database node. When these data are provided to big data through the process of data extraction, transformation and loading (ETL), the data is only verified in the ETL process, and no attention is paid to the data after the ETL is completed.

[0003] This method in the existing technology is feasible when the data volume is not large, but with the advent of the big data era, facing the demand for ETL of data at the terabyte (TB) or even petabyte (PB) level, the probability of errors in the process of data passing through ETL and after the completion of ETL will increase with the increase in the data volume level. At this time, it is particularly important to have a more sufficient verification mechanism.

[0004] Therefore, according to the existing ETL working methods, when faced with TB or even PB-level data, how to ensure the integrity and accuracy of the data is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a data processing method and device, which solve the problem in the prior art that there is no data inspection and repair after the ETL processing is completed, resulting in increased risks in data use and increased difficulty in troubleshooting erroneous data.

[0006] The present application provides a data processing method, which specifically includes:

[0007] Initiate N ETL processes for the source distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in the data warehouse;

[0008] When there are M failed ETL processes during the execution of the N ETL processes, re-initiate the N ETL processes;

[0009] When the N re-initiated ETL processes are successfully executed, obtaining the first target table generated in the data warehouse by the N re-initiated ETL processes, and a first result table generated by statistically aggregating the first target table;

[0010] When data is missing in the first result table, data is repaired in the first result table.

[0011] In a possible implementation, when data is missing from the first result table, repairing data in the first result table includes:

[0012] When the table-level partition data of the first result table is missing, query the corresponding first target table;

[0013] When there is no missing data in the first target table, re-collect and summarize the data according to the first target table to generate table-level partitions corresponding to the first result table, and store the data in the table-level partitions corresponding to the first result table;

[0014] Alternatively, when data is missing from the first target table, the ETL process of the source distributed architecture database corresponding to the first target table is re-initiated.

[0015] In a possible implementation manner, the method further includes:

[0016] When there is no missing data in the table-level partition of the first result table, query the data center node (DCN)-level partition of the source distributed architecture database corresponding to the first result table, where DCN corresponds to the number of processes N in the N ETL processes;

[0017] When there is data missing in the DCN level partition of the first result table, query and generate the DCN level partition name of the first target table corresponding to the DCN level partition of the first result table according to the DCN level partition name of the first result table;

[0018] When there is no data missing in the DCN level partition of the first target table, re-collecting and aggregating the DCN level partitions of the first target table to generate the DCN level partitions corresponding to the first result table, and storing the data in the DCN level partitions corresponding to the first result table;

[0019] When there is data missing in the DCN level partition of the first target table, the ETL process of the DCN level partition of the source distributed architecture database corresponding to the DCN level partition of the first target table is re-initiated.

[0020] In a possible implementation, when data is missing from the first result table, after repairing the data in the first result table, the method further includes:

[0021] Obtaining the amount of data in the source distributed architecture database;

[0022] When the number of data in the first target table is not equal to the number of data in the source distributed architecture database, checking the first target table for duplicate items and extracting non-duplicate data in the first target table;

[0023] Merge and process the duplicate data in the first target table;

[0024] The non-duplicated data is reloaded into the first target table.

[0025] In a possible implementation, before the N ETL processes that are re-initiated are successfully executed, the method further includes:

[0026] When there are M failed ETL processes in the execution process of the N ETL processes, the processes that have been executed in the N ETL processes generate a second target table in the data warehouse;

[0027] Obtaining the second target table;

[0028] Clear the second target table.

[0029] The present application provides a data processing device, which specifically includes:

[0030] A processing unit is used to initiate N ETL processes to a source distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in a data warehouse; when M ETL processes fail during the execution of the N ETL processes, the N ETL processes are re-initiated;

[0031] an acquisition unit, configured to acquire, when the N re-initiated ETL processes are successfully executed, the first target table generated in the data warehouse by the N re-initiated ETL processes, and a first result table generated by performing statistical aggregation on the first target table;

[0032] A processing unit is configured to repair data in the first result table when data is missing in the first result table.

[0033] In a possible implementation manner, the processing unit is further configured to:

[0034] When the table-level partition data of the first result table is missing;

[0035] When there is no missing data in the first target table, re-collect and summarize the data according to the first target table to generate table-level partitions corresponding to the first result table, and store the data in the table-level partitions corresponding to the first result table;

[0036] Alternatively, when data is missing from the first target table, the ETL process of the source distributed architecture database corresponding to the first target table is re-initiated.

[0037] In a possible implementation manner, the processing unit is further configured to:

[0038] When there is no missing data in the table-level partition of the first result table, query the data center node DCN-level partition of the source distributed architecture database corresponding to the first result table, where DCN corresponds to the number of processes N in the N ETL processes;

[0039] When there is data missing in the DCN level partition of the first result table, query and generate the DCN level partition name of the first target table corresponding to the DCN level partition of the first result table according to the DCN level partition name of the first result table;

[0040] When there is no data missing in the DCN level partition of the first target table, re-collecting and aggregating the DCN level partitions of the first target table to generate the DCN level partitions corresponding to the first result table, and storing the data in the DCN level partitions corresponding to the first result table;

[0041] When there is data missing in the DCN level partition of the first target table, the ETL process of the DCN level partition of the source distributed architecture database corresponding to the DCN level partition of the first target table is re-initiated.

[0042] In a possible implementation manner, the processing unit is further configured to:

[0043] Obtaining the amount of data in the source distributed architecture database;

[0044] When the number of data in the first target table is not equal to the number of data in the source distributed architecture database, checking the first target table for duplicate items and extracting non-duplicate data in the first target table;

[0045] Merge and process the duplicate data in the first target table;

[0046] The non-duplicated data is reloaded into the first target table.

[0047] In a possible implementation manner, the processing unit is further configured to:

[0048] When there are M failed ETL processes in the execution process of the N ETL processes, the processes that have been executed in the N ETL processes generate a second target table in the data warehouse;

[0049] Obtaining the second target table;

[0050] Clear the second target table.

[0051] An embodiment of the present application provides a computer device, including a program or instructions, which, when executed, enables a computer to execute a method in any possible design described above.

[0052] An embodiment of the present application provides a storage medium, including a program or an instruction, which, when executed, enables a computer to execute a method in any possible design described above.

[0053] The data processing method and device provided by the present invention have the following beneficial effects: by adding a new inspection, verification and repair link after the ETL process is completed, the anti-interference ability of data pollution and the ability to correct data afterwards are enhanced, the hidden risks of data use are eliminated, high-quality data is provided for subsequent analysis and processing, and the time and energy for processing data are saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the technical architecture of a data processing method provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the device structure of a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with the accompanying drawings and specific implementation methods of the specification. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments may be combined with each other.

[0058] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field, and the traditional financial industry is gradually transforming to financial technology (Fintech). Facing the demand for ETL of terabyte (TB) or even petabyte (PB) level data, the embodiment of the present application provides a data processing method, which can still ensure the integrity and accuracy of the data when processing such massive data. The specific technical solution is as follows:

[0059] Figure 1 A schematic diagram of the technical architecture of a data processing method provided in an embodiment of the present application. Figure 1 The technical architecture shown in the figure includes a distributed database (TDSQL) 100, an ETL script library 101, a check module 102, a data warehouse 103, a post-verification and repair module 104, a scheduling system 105, a business data platform (Business Data Platform, BDP) 106 and a data mining tool platform (A data mining platform based on Spark, BDAP) ​​107.

[0060] The distributed database (TDSQL) 100 is connected to the ETL script library 101 and the inspection module 102. It is a high-performance distributed database compatible with MySQL protocol and syntax and supporting automatic horizontal splitting, and is used to initiate N ETL processes.

[0061] The ETL script library 101 is connected to the distributed database 100, the inspection module 102, the post-verification and repair module 104 and the scheduling system 105. It is the place where the ETL script is stored and is used to push the data of the distributed database 100 to the data warehouse 103 to generate a table.

[0062] The checking module 102 is connected to the distributed database 100 and the data warehouse 103 and is used to check the status of each process during the execution of N ETL processes to see if there is any failed process.

[0063] The data warehouse 103 is connected to the inspection module 102 and the post-verification repair module 104. It is a data warehouse tool based on Hadoop, which can map structured data files into a data table and provide simple SQL query functions. Among them, Hadoop is designed for offline and large-scale data analysis, which is equivalent to the combination of data storage and data processing. The data warehouse 103 is used to store the target table generated by the ETL process initiated by the distributed database 100 pushed by the ETL script library 101 and the result table generated by the statistical summary of the target table.

[0064] The post-verification and repair module 104 is connected to the data warehouse 103, the ETL script library 101 and the scheduling system 105, and is used to check the target table and the result table stored in the data warehouse 103 for missing and number matching of verification data, and to repair erroneous data.

[0065] The scheduling system 105 is connected to the ETL script library 101 and the post-verification and repair module 104, and is used to schedule the ETL script library 101 every hour to push the distributed database 100 to initiate the ETL process; at the same time, the post-verification and repair module 104 is scheduled to work every hour.

[0066] The business data platform 106 is connected to the data warehouse 103 and is used for offline storage and analysis of the target table and result table in the data warehouse 103 .

[0067] The data mining tool platform 107 is connected to the data warehouse 103 and is used to analyze and process the target table and result table in the data warehouse 103 using a rule engine, a big data analysis algorithm, and the like.

[0068] The ETL script library 101 pushes the data of the distributed database 100 to the data warehouse 103 to generate a target table. The post-verification and repair module 104 checks the missing and number matching of the verification data and repairs the specific process of the erroneous data. The following will describe in detail.

[0069] like Figure 2 As shown, it is a flow chart of a data processing method provided in an embodiment of the present application, and the specific steps will be described in detail below.

[0070] Step 200: The source-end distributed architecture database initiates N ETL processes; N is an integer greater than 0; the N ETL processes are used to generate a first target table in the data warehouse;

[0071] Step 201: when there are M failed ETL processes in the execution process of the N ETL processes, re-initiate the N ETL processes;

[0072] Step 202: when the N ETL processes that are re-initiated are successfully executed, obtaining the first target table generated in the data warehouse by the N ETL processes that are re-initiated, and a first result table generated by statistically aggregating the first target table;

[0073] Step 203: When there is data missing in the first result table, repair the data in the first result table.

[0074] On the basis of step 200-step 202, by adding step 203, the problem of increased data hidden dangers and increased data use risks due to the lack of anti-interference ability to data pollution and the ability to correct data afterwards after the general ETL process is completed is solved. The previous processing mode is to return to trace the data source for analysis and processing only after erroneous data is found during processing and analysis. This makes it very difficult to troubleshoot the problem, which brings great inconvenience to the subsequent data analysis and processing, and also consumes a lot of precious time and energy. In this application, the method of actively checking, verifying and repairing the data table generated after the ETL process is completed can ensure the accuracy of the data while completing the ETL work of massive data, and provide high-quality data for subsequent analysis and processing, while also saving time and energy in processing data. The specific detailed process is described below.

[0075] Specifically, step 200 is that the source-end distributed cluster database simultaneously initiates N ETL processes in the form of multiple processes on multiple nodes; N is an integer greater than 0; the N ETL processes are used to generate a first target table in the data warehouse;

[0076] Step 201 is that during the execution of N ETL processes, the checking module is responsible for checking the status of each process to see if there is a failed process. When M failed processes are detected, the ETL processes that have been executed among the N ETL processes generate a second target table in the data warehouse, clear the data in the second target table, and then re-initiate the N ETL processes. Specifically, in the first ETL process, the ETL script library pushes the source distributed database to initiate multiple ETL processes on multiple nodes at the same time. During the execution process, if an ETL process fails due to network or other reasons, the processes that have been executed among the remaining ETL processes generate a second target table in the data warehouse; the unfinished processes become residual processes; and the second ETL process is re-initiated until the execution is completed and the process ends.

[0077] Step 202 is to obtain the first target table generated by the N ETL processes re-initiated in the data warehouse, and the first result table generated by statistically summarizing the first target table when the N ETL processes re-initiated are successfully executed. Among them, the first target table and the first result table in the data warehouse will be stored according to the pre-configured partitions. There are 4 levels of pre-configured storage partitions, namely: table level, date level, hour level and DCN level. The table level is classified according to the target table, the date level is classified according to the date, the hour level is classified according to the 24-hour level, and the DCN level is classified according to the logical nodes of the distributed architecture database. The first target table is stored in partitions at three levels: date level, hour level, and DCN level; the first result table is stored in partitions at the above four levels. These four levels of partitions are associated in an expanded dimension manner from high to low. The four-level partition storage allows for quick and easy query of the stored data at any level of partition. Moreover, when data errors are detected, only the partition data of the lowest dimension in the corresponding level needs to be processed, without processing the entire result table or target table, thereby improving processing efficiency and more efficient and accurate data repair.

[0078] Step 203 is to check whether there is any missing data in the table-level partition of the first result table through a software program. If it is found that there is data missing in the table-level partition of the first result table, the table-level partition of the first result table is repaired.

[0079] Specifically, step 203 is to check whether there is any missing table-level partition of the first result table through a software program.

[0080] If data is found to be missing in the table-level partition of the first result table, the first target table corresponding to the table-level partition generating the first result table is queried through a software program according to the table-level partition name of the first result table.

[0081] Check whether there are any missing items in the first target table. If there are no missing items, re-statistic and summarize the first target table to generate table-level partitions corresponding to the first result table, and store them in the table-level partitions corresponding to the first result table; if there are missing items in the first target table, the post-verification and repair module will trigger the missing error information to inform the ETL script library, and the ETL script library will push the data in the source database to re-initiate the ETL process.

[0082] If it is found that there is no data missing in the table-level partition of the first result table, the DCN-level partition of the first result table is queried, where DCN is a logical node of the distributed architecture database, corresponding to the number of processes N in the N ETL processes.

[0083] When it is found that there is data missing in the DCN-level partition of the first result table, query and generate the DCN-level partition name of the first target table corresponding to the DCN-level partition of the first result table according to the DCN-level partition name of the first result table.

[0084] Check whether there is any missing data in the DCN-level partition of the first target table. If there is no missing data, re-statistically summarize and generate the DCN-level partition corresponding to the first result table according to the DCN-level partition of the first target table, and store it in the DCN-level partition corresponding to the first result table;

[0085] When there is data missing in the DCN-level partition of the first target table, the post-verification and repair module will trigger and inform the ETL script library of this missing error message, and the ETL script library will push the corresponding DCN-level partition in the source database to restart the ETL process.

[0086] More specifically, taking the current time as the latest time, take the Y hours before the latest time as the time range for checking partitions. Generally, Y is an integer greater than or equal to 6. Assume that the current time is date A and hour B. If B >= Y, the time range for checking partitions is A, X, where X is [B - Y, B); if B < Y, the time range for partitions to be verified is A, X1 and A - 1, X2. Among them, X1 is [1, B), and X2 is (24 - (Y - B), 24]. And in the order of date and hour, check whether there is any missing data in the DCN-level partition of the first result table for each hour in the above time range from the back to the front.

[0087] More specifically, the scheduling system initiates an ETL process of the current latest time every hour; when an ETL process lasts for more than 1 hour but has not been completed due to network or other reasons at this time, the ETL script library will accept the scheduling of the scheduling system to initiate an ETL process of the current latest time, and will not restart the previously uncompleted process. At this time, there will be a missing problem in the time partition target table in the data warehouse corresponding to the uncompleted process.

[0088] Through the above detailed description of step 203, the missing check and repair work for the target table and result table stored in the data warehouse have been completed. After that, the post-verification and repair module can also perform matching verification and data repair on the number of data in the first target table and the number of data in the source database.

[0089] Specifically, the ETL script library is called to obtain the number of data in the source database and then send it to the post-verification and repair module. The number of data in the first target table is obtained by the post-verification and repair module from the data warehouse, and the two are compared.

[0090] If the number of data in the first target table is not equal to the number of data in the source database, the result will trigger the ETL script library, which will obtain specific data from the source database and the data warehouse respectively. The ETL script library will first extract the non-duplicate data in the first target table, merge the duplicate data in the first target table, and then reload the non-duplicate data into the first target table. The updated first target table is then fed back to the data warehouse and re-stored in the corresponding partition.

[0091] Through the above detailed description, the missing check and repair work of the first target table and the first result table stored in the data warehouse, as well as the matching check and repair work of the data number of the first target table and the data number of the source database have been completed. After that, the first result table can be checked normally, or additionally checked from the business level according to different business needs.

[0092] Among them, general verification may include:

[0093] 1. Check the quantity corresponding to the four levels, such as the table level quantity: according to the number of tables; the date level quantity: according to the number of days; the hour level quantity: according to the number of 24 hours; the DCN level quantity: according to the number of databases in the distributed cluster.

[0094] 2. Verification of the incremental table, that is, there cannot be any intersection at each level, for example, at the date level: the previous day and the next day cannot have any intersection; at the hour level: the previous hour and the next hour cannot have any intersection; at the DCN level: each DCN cannot have any intersection.

[0095] 3. Verify the number of records and regularly recalculate the number of result tables and the number of data in the target table from the source database to prevent data modification in the source database or data contamination in the target table due to account reversal or other reasons.

[0096] 4. Compare with another isomorphic table every day and compare the difference in numbers.

[0097] 5. When comparing the full table, a configurable error ratio is allowed with the source data table.

[0098] 6. Compare the number of result records of the current verification with the number of result records of the last verification.

[0099] Additional verification, taking the banking system as an example, may include:

[0100] 1. Compare the total amount of debit and credit results in the transaction details table in the data warehouse with the total amount of debit and credit results in the source database.

[0101] 2. Compare the number of bad debt status records in the target table with the number of bad debt status records in the source database.

[0102] 3. Compare the total hash value of a field with the total hash value of the source table.

[0103] At this point, all missing checks, data number matching checks, data repairs, and general or additional verifications of the first result table and the first target table in the data warehouse have been completed, and the integrity and accuracy of the first result table and the first target table have been guaranteed. At the same time, in order to further ensure the continuous integrity and accuracy of the data, the scheduling system will schedule the post-verification and repair module every hour for the above-mentioned inspections, repairs, and verifications until a period of time has passed before the monitoring verification and repair of the data is completed. This period of time can be set to 8 hours.

[0104] Afterwards, the first result table and the first target table will be used for query processing of the business data platform BDP and the data mining tool platform BDAP.

[0105] Figure 3 A device schematic diagram of a data processing method provided in an embodiment of the present application includes a processing unit 300 and an acquisition unit 301, which are described in detail as follows.

[0106] The processing unit 300 is used to initiate N ETL processes to the source distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in the data warehouse; when there are M failed ETL processes during the execution of the N ETL processes, the N ETL processes are re-initiated;

[0107] The acquisition unit 301 is used for acquiring the first target table generated by the N ETL processes that are re-initiated in the data warehouse and the first result table generated by performing statistics and aggregation on the first target table when the N ETL processes that are re-initiated are successfully executed;

[0108] The processing unit 300 is configured to repair data in the first result table when data is missing in the first result table.

[0109] The processing unit 300 is further configured to:

[0110] When the table-level partition data of the first result table is missing;

[0111] When there is no missing data in the first target table, re-collect and summarize the data according to the first target table to generate table-level partitions corresponding to the first result table, and store the data in the table-level partitions corresponding to the first result table;

[0112] Alternatively, when data is missing from the first target table, the ETL process of the source distributed architecture database corresponding to the first target table is re-initiated.

[0113] The processing unit 300 is further configured to:

[0114] When there is no missing data in the table-level partition of the first result table, query the DCN-level partition of the source distributed architecture database corresponding to the first result table, where DCN corresponds to the number of processes N in the N ETL processes;

[0115] When there is data missing in the DCN level partition of the first result table, query and generate the DCN level partition name of the first target table corresponding to the DCN level partition of the first result table according to the DCN level partition name of the first result table;

[0116] When there is no data missing in the DCN level partition of the first target table, re-collecting and aggregating the DCN level partitions of the first target table to generate the DCN level partitions corresponding to the first result table, and storing the data in the DCN level partitions corresponding to the first result table;

[0117] When there is data missing in the DCN level partition of the first target table, the ETL process of the DCN level partition of the source distributed architecture database corresponding to the DCN level partition of the first target table is re-initiated.

[0118] The processing unit 300 is further configured to:

[0119] Obtaining the amount of data in the source distributed architecture database;

[0120] When the number of data in the first target table is not equal to the number of data in the source distributed architecture database, checking the first target table for duplicate items and extracting non-duplicate data in the first target table;

[0121] Merge and process the duplicate data in the first target table;

[0122] The non-duplicated data is reloaded into the first target table.

[0123] The processing unit 300 is further configured to:

[0124] When there are M failed ETL processes in the execution process of the N ETL processes, the processes that have been executed in the N ETL processes generate a second target table in the data warehouse;

[0125] Obtaining the second target table;

[0126] Clear the second target table.

[0127] Finally, it should be noted that: those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: Initiate N data extraction, conversion and loading ETL processes for the source distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in the data warehouse; When there are M failed ETL processes during the execution of the N ETL processes, re-initiate the N ETL processes; When the N re-initiated ETL processes are successfully executed, obtaining the first target table generated in the data warehouse by the N re-initiated ETL processes, and a first result table generated by statistically aggregating the first target table; When the table-level partition data of the first result table is missing; When there is no missing data in the first target table, re-collect and summarize the data according to the first target table to generate table-level partitions corresponding to the first result table, and store the data in the table-level partitions corresponding to the first result table; Alternatively, when data is missing from the first target table, re-initiating the ETL process of the source distributed architecture database corresponding to the first target table; When there is no missing data in the table-level partition of the first result table, query the data center node DCN-level partition of the source distributed architecture database corresponding to the first result table, where DCN corresponds to the number of processes N in the N ETL processes; When there is data missing in the DCN level partition of the first result table, query and generate the DCN level partition name of the first target table corresponding to the DCN level partition of the first result table according to the DCN level partition name of the first result table; When there is no data missing in the DCN level partition of the first target table, re-collecting and aggregating the DCN level partitions of the first target table to generate the DCN level partitions corresponding to the first result table, and storing the data in the DCN level partitions corresponding to the first result table; When there is data missing in the DCN level partition of the first target table, the ETL process of the DCN level partition of the source distributed architecture database corresponding to the DCN level partition of the first target table is re-initiated.

2. The method according to claim 1, characterized in that When there is data missing in the first result table, after repairing the data in the first result table, the method further includes: Obtaining the amount of data in the source distributed architecture database; When the number of data in the first target table is not equal to the number of data in the source distributed architecture database, checking the first target table for duplicate items and extracting non-duplicate data in the first target table; Merge and process the duplicate data in the first target table; The non-duplicated data is reloaded into the first target table.

3. The method according to claim 1, characterized in that Before the re-initiated N ETL processes are successfully executed, the process further includes: When there are M failed ETL processes in the execution process of the N ETL processes, the processes that have been executed in the N ETL processes generate a second target table in the data warehouse; Obtaining the second target table; Clear the second target table.

4. A data processing device, characterized in that: include: A processing unit is used to initiate N ETL processes to a source distributed architecture database; N is an integer greater than 0; the N ETL processes are used to generate a first target table in a data warehouse; when M ETL processes fail during the execution of the N ETL processes, the N ETL processes are re-initiated; an acquisition unit, configured to acquire, when the N re-initiated ETL processes are successfully executed, the first target table generated in the data warehouse by the N re-initiated ETL processes, and a first result table generated by performing statistical aggregation on the first target table; a processing unit, configured to repair data in the first result table when data is missing from the first result table; The processing unit is further used for: When the table-level partition data of the first result table is missing; When there is no missing data in the first target table, re-collect and summarize the data according to the first target table to generate table-level partitions corresponding to the first result table, and store the data in the table-level partitions corresponding to the first result table; Alternatively, when data is missing from the first target table, re-initiating the ETL process of the source distributed architecture database corresponding to the first target table; The processing unit is further used for: When there is no missing data in the table-level partition of the first result table, query the data center node DCN-level partition of the source distributed architecture database corresponding to the first result table, where DCN corresponds to the number of processes N in the N ETL processes; When there is data missing in the DCN level partition of the first result table, query and generate the DCN level partition name of the first target table corresponding to the DCN level partition of the first result table according to the DCN level partition name of the first result table; When there is no data missing in the DCN level partition of the first target table, re-collecting and aggregating the DCN level partitions of the first target table to generate the DCN level partitions corresponding to the first result table, and storing the data in the DCN level partitions corresponding to the first result table; When there is data missing in the DCN level partition of the first target table, the ETL process of the DCN level partition of the source distributed architecture database corresponding to the DCN level partition of the first target table is re-initiated.

5. The device according to claim 4, characterized in that The processing unit is further used for: Obtaining the amount of data in the source distributed architecture database; When the number of data in the first target table is not equal to the number of data in the source distributed architecture database, checking the first target table for duplicate items and extracting non-duplicate data in the first target table; Merge and process the duplicate data in the first target table; The non-duplicated data is reloaded into the first target table.

6. The device according to claim 4, characterized in that The processing unit is further used for: When there are M failed ETL processes in the execution process of the N ETL processes, the processes that have been executed in the N ETL processes generate a second target table in the data warehouse; Obtaining the second target table; Clear the second target table.

7. A computer device, characterized in that: The method comprises a program or an instruction. When the program or the instruction is executed, the method according to any one of claims 1 to 3 is performed.

8. A storage medium, characterized in that: The method comprises a program or an instruction. When the program or the instruction is executed, the method according to any one of claims 1 to 3 is performed.

Images