A data cleaning method and device for data table

Abstract

The present invention provides a data cleaning method and device for a data table, and relates to the field of big data technology. The method comprises: establishing a list of data tables to be cleaned; splitting the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rules; wherein the table list splitting rules are preset; traversing each sub-list, screening the partitions of each data table based on the data cleaning rules, and obtaining a partition list; wherein the data cleaning rules are preset; generating a cleaning script based on the partition list, and executing the cleaning script to complete data cleaning. The device is used to execute the above method. The data cleaning method and device for a data table provided in the embodiments of the invention improve the efficiency of data cleaning.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data cleaning method and device for a data table. Background Art

[0002] At present, with the advent of the era of big data, distributed systems based on Hadoop technology are widely used in various fields. As an important component of distributed systems, Hive tables have also been widely used.

[0003] In the era of big data, the scale of data has expanded rapidly, and storage space has increased accordingly. Due to the large volume of data and high storage costs, companies will regularly clean up data after completing data analysis and mining. In the prior art, data is mainly cleaned manually, but due to the large volume of data, manual data cleaning is costly and inefficient. Moreover, due to the consideration of later projects, while cleaning the data, it is also necessary to retain some data from the huge amount of cleaned data, and manual screening of retained data is also inefficient. Therefore, how to propose a data cleaning method that can quickly clean up Hive table data is an important issue that needs to be solved in this field. Summary of the invention

[0004] In view of the problems in the prior art, an embodiment of the present invention provides a data cleaning method and device for a data table, which can at least partially solve the problems in the prior art.

[0005] In one aspect, the present invention provides a data cleaning method for a data table, comprising:

[0006] Create a list of data tables to be cleaned up;

[0007] According to the table list splitting rule, the data table list to be cleaned up is split into multiple sub-lists; wherein the table list splitting rule is preset;

[0008] Traversing each sub-list, screening the partitions of each data table based on the data cleaning rule, and obtaining a partition list; wherein the data cleaning rule is preset;

[0009] A cleanup script is generated based on the partition list, and the cleanup script is executed to complete data cleanup.

[0010] In another aspect, the present invention provides a data cleaning device for a data table, comprising:

[0011] Establish a module for establishing a list of data tables to be cleaned;

[0012] A splitting module, used to split the data table list to be cleaned into multiple sub-lists according to the table list splitting rule; wherein the table list splitting rule is preset;

[0013] A traversal module, used to traverse each sub-list, screen the partitions of each data table based on a data cleaning rule, and obtain a partition list; wherein the data cleaning rule is preset;

[0014] The cleaning module is used to generate a cleaning script based on the partition list and execute the cleaning script to complete data cleaning.

[0015] On the other hand, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data cleaning method for the data table described in any of the above embodiments are implemented.

[0016] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data cleaning method for a data table described in any of the above embodiments.

[0017] The data cleaning method and device for a data table provided in an embodiment of the invention can establish a list of data tables to be cleaned, split the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rules, traverse each sub-list, screen the partitions of each data table based on the data cleaning rules, obtain a partition list, generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning, thereby improving the efficiency of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 It is a flowchart of a data cleaning method for a data table provided in one embodiment of the present invention.

[0020] Figure 2 It is a flowchart of a data cleaning method for a data table provided by another embodiment of the present invention.

[0021] Figure 3 It is a flowchart of a data cleaning method for a data table provided by yet another embodiment of the present invention.

[0022] Figure 4It is a structural schematic diagram of a data cleaning device for a data table provided by an embodiment of the present invention.

[0023] Figure 5 It is a structural schematic diagram of a data cleaning device for a data table provided by another embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily.

[0026] In order to facilitate the understanding of the technical solution provided by the present application, the relevant contents of the technical solution of the present application are first described below. The present invention provides a data cleaning method for a data table, which obtains a list of tables to be cleaned, and simultaneously traverses all partitions in the Hadoop cluster concurrently, screens out data that meets the cleaning range to generate a cleaning list, and after screening out the retained data from the cleaning list, generates a corresponding cleaning script, which can concurrently execute the cleaning script to complete data cleaning, thereby improving the efficiency of enterprise data cleaning and reducing the cost of data cleaning. The data cleaning method for a data table provided in an embodiment of the present invention can be applied to a distributed system based on Hadoop technology to clean up Hive table data.

[0027] The following takes the cleaning server as an example to illustrate the implementation process of the data table data cleaning method provided by the embodiment of the present invention. It is understandable that the execution subject of the data table data cleaning method provided by the embodiment of the present invention includes but is not limited to the cleaning server.

[0028] Figure 1 FIG. 1 is a flow chart of a method for cleaning a data table according to an embodiment of the present invention. Figure 1 As shown, the data cleaning method of the data table provided by the embodiment of the present invention includes:

[0029] S101, establishing a list of data tables to be cleaned up;

[0030] Specifically, the cleaning server can access the database or Hadoop cluster, obtain relevant information of all data tables, and establish a list of data tables to be cleaned. The list of data tables to be cleaned may include information such as data table name, data table size, and data table partitions, which is set according to actual needs and is not limited in the embodiment of the present invention.

[0031] S102, splitting the data table list to be cleaned into multiple sub-lists according to a table list splitting rule; wherein the table list splitting rule is preset;

[0032] Specifically, since the amount of data is large, in order to facilitate subsequent screening of the data to be cleaned, the cleaning server may split the to-be-cleaned data table list into multiple sub-lists according to the table list splitting rule.

[0033] For example, the table list splitting rule is: split the list of data tables to be cleaned into m sub-lists, and the number of data tables corresponding to each sub-list does not exceed n. The cleaning server can count the total number of data tables corresponding to the list of data tables to be cleaned, and then divide it by m to obtain a quotient k. If the quotient k is an integer, then the quotient k can be used as the number of data tables corresponding to each sub-list; if the quotient k is not an integer, then the quotient k can be rounded up to obtain p, and p is used as the number of data tables corresponding to m-1 sub-lists, and the remaining sub-list corresponds to the remaining number of data tables. Among them, m and n are set according to actual conditions, and the embodiment of the present invention does not limit them.

[0034] For example, the table list splitting rule is: split the data table list to be cleaned into m sub-lists, the number of data tables corresponding to each sub-list does not exceed n, and the sum of the sizes of the data tables corresponding to each sub-list does not exceed a preset value. The cleaning server can arrange the data tables corresponding to the list of data tables to be cleaned from large to small according to the size of the data tables to obtain a sorting result, and then distribute the data tables to m sub-lists according to the sorting result, so that the sum of the sizes of the data tables corresponding to each data table is as consistent as possible, the first sub-list can correspond to the data tables with serial numbers 1, 2m, 2m+1, 4m, 4m+1, 6m+1, 8m... in the sorting result, the second sub-list can correspond to the data tables with serial numbers 2, 2m-1, 2m+2, 4m-1, 6m+2, 8m-1... in the sorting result, the third sub-list can correspond to the data tables with serial numbers 3, 2m-2, 2m+3, 4m-2, 6m+3, 8m-2... in the sorting result, the fourth sub-list can correspond to the data tables with serial numbers 4, 2m-3, 2m+4, 4m-3, 6m+4, 8m-1... in the sorting result, and so on.

[0035] S103, traversing each sub-list, screening the partition of each data table based on the data cleaning rule, and obtaining a partition list; wherein the data cleaning rule is preset;

[0036] Specifically, after splitting into multiple sub-lists, the cleaning server traverses each sub-list to screen the partitions of each data table according to the data cleaning rules, screens out the partitions of each data table corresponding to each sub-list that need to be cleaned, and then merges the partitions of each data table that need to be cleaned to obtain a partition list. The partition list includes the partition range of the data table that needs to be cleaned. Among them, the data cleaning rules are preset, which set which partitions need to be cleaned.

[0037] For example, when traversing each sub-list, the traversal may be performed one by one; however, in order to improve efficiency, multiple sub-lists may be traversed at the same time, that is, each sub-list may be traversed concurrently.

[0038] When traversing each sub-list, obtain the data table name from the sub-list in turn, obtain all partitions included in the corresponding data table according to the data table name, determine which partitions meet the data cleaning rules, and use the partitions that meet the data cleaning rules as the partitions that need to be cleaned.

[0039] S104: Generate a cleanup script based on the partition list, and execute the cleanup script to complete data cleanup.

[0040] Specifically, after obtaining the partition list, the cleaning server may generate a cleaning script based on the partition list, and then execute the cleaning script to delete the data to be cleaned from the corresponding data table to complete data cleaning.

[0041] The data cleaning method for a data table provided in an embodiment of the present invention can establish a list of data tables to be cleaned, split the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rules, traverse each sub-list, screen the partitions of each data table based on the data cleaning rules, obtain a partition list, generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning, thereby improving the efficiency of data cleaning.

[0042] On the basis of the above embodiments, further, the table list splitting rule includes:

[0043] Based on the total number of data tables corresponding to the list of data tables to be cleared and the number of preset sub-lists, the list of data tables to be cleared is evenly divided according to the number of preset sub-lists; or

[0044] The list of data tables to be cleaned is comprehensively split based on the total number of data tables corresponding to the list of data tables to be cleaned and the size of each data table, so that the number of data tables corresponding to each sub-list is less than a first threshold and the total size of the data tables corresponding to each sub-list is less than a second threshold.

[0045] Specifically, the cleaning server can obtain the total number Q of data tables corresponding to the list of data tables to be cleaned according to the number of data table names included in the list of data tables to be cleaned. The number of sub-lists is preset to m, then k=Q / m is calculated. If the value of k is an integer, then each sub-list corresponds to k data tables; if the value of k is a decimal, then k can be rounded up to obtain p. If each sub-list corresponds to p data tables, then there will be a lack of mp-Q data tables. The mp-Q sub-lists can correspond to p-1 data tables, and the remaining m-(mp-Q) sub-lists can correspond to p data tables. K can also be rounded down to obtain s. If each sub-list corresponds to s data tables, then there will be Q-ms data tables more. The Q-ms sub-lists can correspond to s+1 data tables, and the remaining m-(ms-Q) sub-lists can correspond to s data tables. The above process realizes the uniform splitting of the list of data tables to be cleaned.

[0046] The cleaning server can obtain the total number Q of data tables corresponding to the list of data tables to be cleaned according to the number of data table names included in the list of data tables to be cleaned, and can also obtain the size of each data table through the list of data tables to be cleaned. The cleaning server sorts the data tables corresponding to the list of data tables to be cleaned from large to small according to the size of the data tables to obtain a sorting result. The cleaning server sequentially allocates the data tables corresponding to each sub-list according to the sorting result and the preset number of sub-lists m. The first sub-list corresponds to the data tables with serial numbers 1, 2m, 2m+1, 4m, 4m+1, 6m+1, 8m... in the sorting results; the second sub-list corresponds to the data tables with serial numbers 2, 2m-1, 2m+2, 4m-1, 6m+2, 8m-1... in the sorting results; the third sub-list corresponds to the data tables with serial numbers 3, 2m-2, 2m+3, 4m-2, 6m+3, 8m-2... in the sorting results; the fourth sub-list corresponds to the data tables with serial numbers 4, 2m-3, 2m+4, 4m-3, 6m+4, 8m-1... in the sorting results; and so on. The mth sub-list corresponds to the data tables with serial numbers m, m+1, 3m, 3m+1, 7m, 7m+1... in the sorting results.

[0047] The cleaning server can calculate k=Q / m based on the total number Q of data tables corresponding to the data table list to be cleaned and the preset number of sub-lists m. If k is an integer, then each sub-list corresponds to k data tables, k is less than a first threshold, and the sum of the sizes of the data tables corresponding to each sub-list is less than a second threshold.

[0048] If the k value is a decimal, then k can be rounded up to obtain p. If each sub-list corresponds to p data tables, then there will be a lack of mp-Q data tables. Finally, there are mp-Q sub-lists corresponding to p-1 data tables, and the remaining m-(mp-Q) sub-lists correspond to p data tables. p is less than the first threshold, and the sum of the sizes of the data tables corresponding to each sub-list is less than the second threshold.

[0049] If the k value is a decimal, k can also be rounded down to obtain s. If each sub-list corresponds to s data tables, there will be Q-ms additional data tables. Finally, there are Q-ms sub-lists corresponding to s+1 data tables, and the remaining m-(ms-Q) sub-lists correspond to s data tables. s is less than the first threshold, and the sum of the sizes of the data tables corresponding to each sub-list is less than the second threshold.

[0050] For example, suppose that the list of data tables to be cleaned corresponds to 9 data tables: data table A, data table B, data table C, data table D, data table E, data table F, data table G, data table H, and data table I. The above 9 data tables are sorted according to the size as shown in Table 1.

[0051] Table 1

[0052] Sorting number Data Sheet 1 Data Table A 2 Data Table B 3 Data Table C 4 Data Table D 5 Data Sheet E 6 Data Sheet F 7 Data Table G 8 Data Sheet H 9 Data Sheet I

[0053] The number of preset sub-lists is 3, and the three sub-lists are named n1, n2, and n3 respectively. In order to ensure that the data volume of each sub-list is consistent as much as possible, the cleaning server allocates the data table corresponding to each sub-list in turn according to the sorting results in Table 1 and the preset number of sub-lists 3. The process is as follows: in the first round of allocation, the data table A corresponding to the sequence number 1 is allocated to the sub-list n1, the data table B corresponding to the sequence number 2 is allocated to the sub-list n2, and the data table C corresponding to the sequence number 3 is allocated to the sub-list n3; in the second round of allocation, the data table F corresponding to the sequence number 6 is allocated to the sub-list n1, the data table E corresponding to the sequence number 5 is allocated to the sub-list n2, and the data table D corresponding to the sequence number 4 is allocated to the sub-list n3; in the third round of allocation, the data table G corresponding to the sequence number 7 is allocated to the sub-list n1, the data table H corresponding to the sequence number 8 is allocated to the sub-list n2, and the data table I corresponding to the sequence number 9 is allocated to the sub-list n3.

[0054] After the above-mentioned list of data tables to be cleaned is split, the final split results are: sub-list n1 includes data table A, data table F and data table G; sub-list n2 includes data table B, data table E and data table H; sub-list n3 includes data table C, data table D and data table I.

[0055] On the basis of the above embodiments, further, traversing each sub-list and screening the partitions of each data table based on the data cleaning rule includes:

[0056] Concurrently traverse each sub-list and filter the partitions of each data table based on the data cleansing rules.

[0057] Specifically, in order to improve the screening efficiency of each sub-list, each sub-list may be traversed concurrently, and the partition of each data table may be screened based on the data cleaning rule to obtain the partition of each data table corresponding to each sub-list that needs to be cleaned.

[0058] For example, if there are 20 sublists, 5 threads can be started to traverse each sublist concurrently, with each thread processing 4 sublists.

[0059] On the basis of the above embodiments, further, the data cleaning rules include:

[0060] Clean up the data in the preset partition segment.

[0061] Specifically, the data cleaning rule includes cleaning data in a preset partition segment, and the partition segment can be represented by a partition identifier. The preset partition segment is set according to actual needs and is not limited in the embodiment of the present invention.

[0062] For example, the partition can be identified by time, such as a partition is 2020-01-01. The preset partition segment is 2020-01-01 to 2020-06-30, that is, the data corresponding to the partition identifier 2020-01-01 to the partition identifier 2020-06-30 in the data table is cleaned.

[0063] Figure 2 FIG. 1 is a flow chart of a data table data cleaning method provided by another embodiment of the present invention. Figure 2 As shown, based on the above embodiments, further, generating a cleanup script based on the partition list includes:

[0064] S201, updating the partition list according to a data retention condition; wherein the retention condition is preset;

[0065] Specifically, the retention condition is preset, setting which data tables cannot delete data or which partitions of which tables cannot delete data. The cleaning server can remove the data tables that cannot delete data or the partitions of the data tables that cannot delete data from the partition list according to the data retention condition, update the partition list, and obtain an updated partition list.

[0066] For example, the data retention condition includes a list of data tables that need to be retained, that is, a retained table list. The cleaning server reads the data table name from the retained table list and removes the corresponding data table from the partition list.

[0067] For example, the data retention condition includes a list of partitions of the data table that needs to be retained, that is, a retained partition list. The cleaning server reads the data table name and the corresponding partition identifier from the retained partition list, and removes the corresponding partition of the corresponding data table from the partition list according to the data table name and the corresponding partition identifier.

[0068] S202: Generate a cleanup script according to the updated partition list.

[0069] Specifically, after obtaining the updated partition list, the cleaning server generates corresponding cleaning statements for the data tables and partitions of the data tables included in the updated partition list, and each generated cleaning statement constitutes a cleaning script.

[0070] For example, the partition list includes the data table name ecs.dcm_ecs_tb2020_s and the corresponding partition identifier 2020-01-01. The generated cleanup statement is: alter table ecs.dcm_ecs_tb2020_s drop partition(pt_dt='2020-01-01'), which means deleting the data corresponding to the partition identifier 2020-01-01 from the data table ecs.dcm_ecs_tb2020_s.

[0071] On the basis of the above embodiments, further, generating a cleanup script based on the partition list includes:

[0072] A plurality of cleaning sub-scripts are generated according to the partition list, each sub-script including no more than a preset number of cleaning statements.

[0073] Specifically, the cleaning server may generate a plurality of cleaning sub-scripts according to the partition list, each cleaning sub-script including no more than a preset number of cleaning statements, wherein the preset number is set according to actual needs and is not limited in the embodiment of the present invention.

[0074] For example, the preset number is 20,000. When the cleaning server generates cleaning statements according to the partition list, every time 20,000 cleaning statements are generated, the 20,000 cleaning statements are grouped into a cleaning sub-script until the partition list can no longer generate cleaning statements.

[0075] On the basis of the above embodiments, further, executing the cleaning script to complete data cleaning includes:

[0076] Concurrently execute each cleaning sub-script to complete data cleaning.

[0077] Specifically, in order to improve the execution efficiency of the cleaning script, the cleaning server can execute each cleaning sub-script concurrently to complete data cleaning. Concurrently executing each cleaning sub-script can improve the efficiency of data cleaning.

[0078] For example, there are 50 cleanup sub-scripts, which are executed in parallel by 10 threads, with each thread executing 5 cleanup sub-scripts.

[0079] Figure 3 FIG. 1 is a flow chart of a data table data cleaning method provided by another embodiment of the present invention. Figure 3 As shown, the specific implementation process of the data table data cleaning method provided by the embodiment of the present invention is as follows:

[0080] Step 1: Create a list of data tables to be cleaned. The cleanup server Hadoop cluster obtains the table names and sizes of all data tables to create a list of data tables to be cleaned.

[0081] Step 2: Split the list of data tables to be cleaned. The cleaning server performs a comprehensive splitting of the list of data tables to be cleaned based on the total number of data tables corresponding to the list of data tables to be cleaned and the size of each data table so that the number of data tables corresponding to each sub-list is less than a first threshold and the total size of the data tables corresponding to each sub-list is less than a second threshold, and obtains N sub-lists.

[0082] Step 3: Concurrently traverse each sublist. The cleaning server concurrently traverses N sublists, and screens the partitions of the data table of each sublist based on the data cleaning rules to obtain the partitions that need to be cleaned for each sublist, and obtains the cleaned partition list of each sublist. The cleaned partition list includes the partitions that need to be cleaned in the data table.

[0083] Step 4: Merge to obtain a partition list. The cleanup server merges the cleanup partition lists of each sub-list to obtain a partition list.

[0084] Step 5: Screen out the reserved data. The cleaning server updates the partition list according to the data retention condition to obtain an updated partition list.

[0085] Step 6: Generate multiple cleaning sub-scripts. The cleaning server generates multiple cleaning sub-scripts according to the partition list.

[0086] Step 7: Execute multiple cleaning sub-scripts concurrently. The cleaning server executes each cleaning sub-script concurrently to complete data cleaning.

[0087] The data cleaning method for a data table provided in an embodiment of the present invention can improve the data cleaning efficiency of the data table and reduce the data cleaning cost by setting the range of the cleaning partition and the retained tables and partitions, concurrently traversing the sub-lists and concurrently executing the cleaning sub-scripts.

[0088] Figure 4 FIG. 1 is a schematic diagram of a data table data cleaning device provided by an embodiment of the present invention. Figure 4 As shown, the data cleaning device for a data table provided by an embodiment of the present invention includes a creation module 401, a splitting module 402, a traversal module 403 and a cleaning module 404, wherein:

[0089] The establishment module 401 is used to establish a list of data tables to be cleaned; the splitting module 402 is used to split the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rules; wherein the table list splitting rules are preset; the traversal module 403 is used to traverse each sub-list, and screen the partitions of each data table based on the data cleaning rules to obtain a partition list; wherein the data cleaning rules are preset; the cleaning module 404 is used to generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning.

[0090] Specifically, the establishment module 401 can access the database or Hadoop cluster, obtain relevant information of all data tables, and establish a list of data tables to be cleaned up. The list of data tables to be cleaned up may include information such as data table name, data table size, and data table partitions, which is set according to actual needs and is not limited in the embodiment of the present invention.

[0091] Since the amount of data is large, in order to facilitate the subsequent screening of the data to be cleaned, the splitting module 402 can split the data table list to be cleaned into multiple sub-lists according to the table list splitting rules.

[0092] After splitting into multiple sub-lists, the traversal module 403 traverses each sub-list to screen the partitions of each data table according to the data cleaning rule, screens out the partitions of each data table corresponding to each sub-list that need to be cleaned, and then merges the partitions of each data table that need to be cleaned to obtain a partition list. The partition list includes the partition range of the data table that needs to be cleaned. The data cleaning rule is preset, which sets which partitions need to be cleaned.

[0093] After obtaining the partition list, the cleaning module 404 may generate a cleaning script based on the partition list, and then execute the cleaning script to delete the data to be cleaned from the corresponding data table to complete data cleaning.

[0094] The data cleaning device for a data table provided in an embodiment of the present invention can establish a list of data tables to be cleaned, split the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rules, traverse each sub-list, screen the partitions of each data table based on the data cleaning rules, obtain a partition list, generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning, thereby improving the efficiency of data cleaning.

[0095] On the basis of the above embodiments, further, the table list splitting rule includes:

[0096] Based on the total number of data tables corresponding to the list of data tables to be cleared and the number of preset sub-lists, the list of data tables to be cleared is evenly divided according to the number of preset sub-lists; or

[0097] The list of data tables to be cleaned is comprehensively split based on the total number of data tables corresponding to the list of data tables to be cleaned and the size of each data table, so that the number of data tables corresponding to each sub-list is less than a first threshold and the total size of the data tables corresponding to each sub-list is less than a second threshold.

[0098] On the basis of the above embodiments, further, the traversal module 403 is specifically used for:

[0099] Concurrently traverse each sub-list and filter the partitions of each data table based on the data cleansing rules.

[0100] On the basis of the above embodiments, further, the data cleaning rules include:

[0101] Clean up the data in the preset partition segment.

[0102] Figure 5 FIG. 1 is a schematic diagram of a data table data cleaning device provided by another embodiment of the present invention. Figure 5 As shown, based on the above embodiments, further, the cleaning module 404 includes an updating unit 4041 and a generating unit 4042, wherein:

[0103] The updating unit 4041 is used to update the partition list according to the data retention condition; wherein the retention condition is preset; the generating unit 4042 is used to generate a cleanup script according to the updated partition list.

[0104] On the basis of the above embodiments, further, the cleaning module 404 is specifically used for:

[0105] A plurality of cleaning sub-scripts are generated according to the partition list, each sub-script including no more than a preset number of cleaning statements.

[0106] On the basis of the above embodiments, further, the cleaning module 404 is also used for:

[0107] Concurrently execute each cleaning sub-script to complete data cleaning.

[0108] The embodiments of the apparatus provided in the embodiments of the present invention can be specifically used to execute the processing flow of the above-mentioned method embodiments, and its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0109] It should be noted that the data table data cleaning method and device provided in the embodiment of the present invention can be used in the financial field, and can also be used in any technical field other than the financial field. The embodiment of the present invention does not limit the application field of the data table data cleaning method and device.

[0110] Figure 6 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604. The processor 601 may call the logic instructions in the memory 603 to execute the following method: establish a list of data tables to be cleaned; split the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rule; wherein the table list splitting rule is preset; traverse each sub-list, screen the partitions of each data table based on the data cleaning rule, and obtain a partition list; wherein the data cleaning rule is preset; generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning.

[0111] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0112] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: establishing a list of data tables to be cleaned; splitting the list of data tables to be cleaned into multiple sub-lists according to a table list splitting rule; wherein the table list splitting rule is preset; traversing each sub-list, screening the partitions of each data table based on the data cleaning rule, and obtaining a partition list; wherein the data cleaning rule is preset; generating a cleaning script based on the partition list, and executing the cleaning script to complete data cleaning.

[0113] The present embodiment provides a computer-readable storage medium, which stores a computer program, and the computer program enables the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: establishing a list of data tables to be cleaned; splitting the list of data tables to be cleaned into multiple sub-lists according to the table list splitting rule; wherein the table list splitting rule is preset; traversing each sub-list, screening the partitions of each data table based on the data cleaning rule, and obtaining a partition list; wherein the data cleaning rule is preset; generating a cleaning script based on the partition list, and executing the cleaning script to complete data cleaning.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A data cleaning method for a data table, characterized in that: include: Create a list of data tables to be cleaned up; According to the table list splitting rule, the data table list to be cleaned up is split into multiple sub-lists; wherein the table list splitting rule is preset; Traversing each sub-list, screening the partitions of each data table based on the data cleaning rule, and obtaining a partition list; wherein the data cleaning rule is preset; Generate a cleanup script based on the partition list, and execute the cleanup script to complete data cleanup; Among them, each sub-list is traversed, and the partitions of each data table are screened based on the data cleaning rules. The partition list obtained includes: Traverse each sub-list, filter the partitions of each data table according to the data cleaning rules, filter out the partitions that need to be cleaned of each data table corresponding to each sub-list, and then merge the partitions that need to be cleaned of each data table to obtain a partition list; the partition list includes the partition range of the data table that needs to be cleaned.

2. The method according to claim 1, characterized in that The table list splitting rules include: Based on the total number of data tables corresponding to the list of data tables to be cleared and the number of preset sub-lists, the list of data tables to be cleared is evenly divided according to the number of preset sub-lists; or The list of data tables to be cleaned is comprehensively split based on the total number of data tables corresponding to the list of data tables to be cleaned and the size of each data table, so that the number of data tables corresponding to each sub-list is less than a first threshold and the total size of the data tables corresponding to each sub-list is less than a second threshold.

3. The method according to claim 1, characterized in that The traversing of each sub-list and screening the partitions of each data table based on the data cleaning rule includes: Concurrently traverse each sub-list and filter the partitions of each data table based on the data cleansing rules.

4. The method according to claim 1, characterized in that The data cleaning rules include: Clean up the data in the preset partition segment.

5. The method according to claim 1, characterized in that The generating of the cleanup script based on the partition list comprises: Updating the partition list according to a data retention condition; wherein the retention condition is preset; Generate a cleanup script based on the updated partition manifest.

6. The method according to any one of claims 1 to 5, characterized in that: The generating of the cleanup script based on the partition list comprises: A plurality of cleaning sub-scripts are generated according to the partition list, each sub-script including no more than a preset number of cleaning statements.

7. The method according to claim 6, characterized in that Executing the cleaning script to complete data cleaning includes: Concurrently execute each cleaning sub-script to complete data cleaning.

8. A data cleaning device for a data table, characterized in that: include: Establish a module for establishing a list of data tables to be cleaned; A splitting module, used to split the data table list to be cleaned into multiple sub-lists according to the table list splitting rule; wherein the table list splitting rule is preset; A traversal module, used to traverse each sub-list, screen the partitions of each data table based on a data cleaning rule, and obtain a partition list; wherein the data cleaning rule is preset; A cleaning module, used to generate a cleaning script based on the partition list, and execute the cleaning script to complete data cleaning; Among them, the traversal module is specifically used to traverse each sub-list, screen the partitions of each data table according to the data cleaning rules, screen out the partitions that need to be cleaned of each data table corresponding to each sub-list, and then merge the partitions that need to be cleaned of each data table to obtain a partition list; the partition list includes the partition range of the data table that needs to be cleaned.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images