Data cleaning method and device

Abstract

The present application provides a data cleaning method and device, which are applied to a target cleaning node corresponding to any data warehouse in a multi-level data warehouse. The method comprises: determining the target data to be cleaned from any data warehouse and creating a cleaning task for the target data, wherein the cleaning task includes the target data information and a cleaning rule; determining a source data warehouse of the target data, and determining a target cleaning sub-node corresponding to the source data warehouse according to a mapping relationship between the data warehouse and the cleaning node; synchronizing the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

Description

Technical Field

[0001] The present application relates to the field of data warehouses, and in particular to a data cleaning method and device. Background Art

[0002] Data cleaning is the process of re-examining and verifying data, with the aim of deleting duplicate information, correcting existing errors, and processing invalid and missing values. In the related art, in the scenario of multi-level data warehouses, data warehouses at different levels have their own cleaning nodes to control the data cleaning process of the data warehouse at this level. Cleaning nodes at different levels cannot be linked, resulting in inconsistent and non-uniform cleaning rules used by data warehouses at all levels. Summary of the invention

[0003] In view of this, the present application provides a data cleaning method and device for cleaning data.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] According to a first aspect of the present application, a data cleaning method is proposed, which is applied to a target cleaning node corresponding to any data warehouse in a multi-level data warehouse, and the method comprises:

[0006] Determine the target data to be cleaned from any of the data warehouses and create a cleaning task for the target data, wherein the cleaning task includes the target data information and cleaning rules;

[0007] Determine the source data warehouse of the target data, and determine the target cleaning sub-node corresponding to the source data warehouse according to the mapping relationship between the data warehouse and the cleaning node;

[0008] The cleaning task is synchronized to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

[0009] According to a second aspect of the present application, a data cleaning device is proposed, which is applied to any cleaning node in a multi-level data warehouse, and the device includes:

[0010] A creation unit, which determines target data to be cleaned from any of the data warehouses and creates a cleaning task for the target data, wherein the cleaning task includes the target data information and cleaning rules;

[0011] A determination unit, which determines a source data warehouse of the target data, and determines a target cleaning sub-node corresponding to the source data warehouse according to a mapping relationship between the data warehouse and the cleaning node;

[0012] The synchronization unit synchronizes the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

[0013] According to a third aspect of the present application, an electronic device is provided, including:

[0014] processor;

[0015] a memory for storing processor-executable instructions;

[0016] The processor implements the method described in the embodiment of the first aspect above by running the executable instructions.

[0017] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the embodiments of the first aspect above are implemented.

[0018] It can be seen from the technical solution provided by the present application that in the present application, the cleaning node corresponding to any data warehouse in the multi-level data warehouse can determine the source data warehouse of the target data that needs to be cleaned, and then, in combination with the mapping relationship between the data warehouse and the cleaning node, the cleaning task is sent to the cleaning sub-node corresponding to the determined source data warehouse. Since the cleaning task contains cleaning rules, through the method of the present application, the cleaning rules of any data warehouse and the cleaning rules of the source data warehouse that provides data to this data warehouse will be consistent, thereby improving the uniformity of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 is an architecture diagram of a multi-level data warehouse according to an exemplary embodiment of the present application;

[0021] Figure 2 is a relationship diagram between a data warehouse and a cleaning node according to an exemplary embodiment of the present application;

[0022] Figure 3 is a flow chart of a data cleaning method according to an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a data cleaning electronic device according to an exemplary embodiment of the present application;

[0024] Figure 5It is a block diagram of a data cleaning device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0026] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] Next, the embodiments of the present application are described in detail.

[0029] Figure 1 The following is an architecture diagram of a multi-level data warehouse according to an exemplary embodiment of the present application. It includes three-level data warehouses. In the multi-level database warehouse in the present application, the first-level data warehouse is called the upper-level data warehouse of the second-level and third-level data warehouses, the second-level data warehouse is called the upper-level data warehouse of the third-level data warehouse, and so on. Figure 1 This is for illustration only, and the present application does not limit the number of levels of the multi-level data warehouse.

[0030] Figure 2 FIG. 1 is a diagram showing a relationship between a data warehouse and a cleaning node according to an exemplary embodiment of the present application. Figure 2 The cleaning node corresponding to the primary data warehouse 11 is the primary cleaning node 1A, the secondary data warehouse 21 corresponds to the secondary cleaning node 2A, and the secondary data warehouse 22 corresponds to the secondary cleaning node 2B. The cleaning nodes corresponding to the data warehouse can control the data cleaning process of the corresponding data warehouse.

[0031] Figure 3 FIG. 1 is a flow chart of a data cleaning method according to an exemplary embodiment of the present application. Figure 3 As shown, the method is applied to a target cleaning node corresponding to any data warehouse in a multi-level data warehouse, and the method may include the following steps:

[0032] Step 302: Determine target data to be cleaned from any data warehouse and create a cleaning task for the target data, wherein the cleaning task includes the target data information and cleaning rules.

[0033] In one embodiment, if Figure 1 The scenario shown is in a multi-level data warehouse, in which the lower-level data warehouse can aggregate data to the upper-level data warehouse corresponding to itself. For example, the third-level data warehouses 31 and 32 can aggregate the data they maintain to the second-level data warehouse 21, and the second-level data warehouses 21 and 22 can aggregate the data they maintain to the first-level data warehouse 11. It is worth noting that the lower-level data warehouse can aggregate all the data it maintains to the upper-level data warehouse, and can also aggregate part of its own data to the upper-level data warehouse. In addition to maintaining the data aggregated by the lower-level data warehouse, the upper-level data warehouse can also maintain other data that is not related to the lower-level data warehouse, and this application does not limit this. In the process of data aggregation, the management tool of the multi-level data warehouse can call the ETL (Extract transform load, data warehouse technology) tool to extract specific data from the lower-level database according to the preset data exchange task and aggregate it to the upper-level database.

[0034] In one embodiment, a target cleaning node corresponding to any data warehouse in a multi-level data warehouse can determine target data to be cleaned from any data warehouse, wherein any data warehouse has a subordinate data warehouse that provides data to it, and each data warehouse has its own corresponding cleaning node, and the cleaning node is used to control the data cleaning process of the corresponding data warehouse, such as issuing cleaning tasks to the corresponding data warehouse, triggering data cleaning, etc. Data warehouses at the same level can be controlled by different cleaning nodes or by the same cleaning node.

[0035] Specifically, the target cleaning node can determine the target data that needs to be cleaned from its corresponding data warehouse. The target data can be all the data in the above data warehouse or part of the data, and this application does not limit this. The above target data can form an ODS (Operational Data Store) table in the target cleaning node. The target cleaning node can generate a cleaning task for the above ODS table. The above cleaning task contains the cleaning rules needed to clean the target data, which are used to indicate what kind of processing the target data needs to be performed on this cleaning task, for example, sorting the target data in a certain way, or deduplicating and correcting the target data, etc. The same cleaning task can contain one or more cleaning rules. In addition, the cleaning task also contains information about the target data, so that when the cleaning node obtains the cleaning task and sends it to the corresponding data warehouse, the data warehouse can determine what data needs to be cleaned in this data warehouse based on the information of the target data.

[0036] In one case, the cleaning rules can be configured by the management device of the multi-level database and sent to the target cleaning node, and the cleaning rules can be flexibly adjusted according to the different requirements of the cleaning data. The target cleaning node combines the obtained cleaning rules with the target data to be cleaned to generate a cleaning task for the target data.

[0037] Considering the wide variety of cleaning rules, and in a multi-level data warehouse, the data of the upper data warehouse may be composed of data aggregated from multiple lower data warehouses. When the cleaning rule includes the aggregation processing of data across data warehouses, the cleaning task generated for such cleaning rules will not trigger the cleaning process in this application, but will be processed according to other cleaning processes. For example, when the target data is composed of a secondary data warehouse 21 and a secondary data warehouse 22, and the cleaning rule is to sort the entire target data, or to perform a deduplication operation on the target data as a whole, then such cleaning rules are said to include aggregation processing of data across data warehouses. In order to cope with the above situation, when applying the above method, the target cleaning node can analyze the cleaning rules it obtains. If it includes the above-mentioned aggregation processing of data across data warehouses, the cleaning process in this application will not be triggered, but will be processed according to other cleaning processes; when it is confirmed that the cleaning rule does not include aggregation processing of data across data warehouses, the subsequent process can be continued.

[0038] Step 304: Determine the source data warehouse of the target data, and determine the target cleaning sub-node corresponding to the source data warehouse according to the mapping relationship between the data warehouse and the cleaning node.

[0039] In one embodiment, when performing data cleaning, the manager of the upper-level data warehouse wants to send the cleaning rules specified by itself to the lower-level data warehouse to ensure that the cleaning rules used by the upper and lower-level data warehouses are consistent when performing data cleaning. Therefore, by performing a blood relationship analysis on the target data provided by any data warehouse, the source data warehouse obtained should be the lower-level data warehouse of any data warehouse, that is, the data warehouse that provides data to any data warehouse.

[0040] refer to Figure 1 , assuming that the target data comes from the secondary data warehouse 21, the secondary data warehouse 22 and the tertiary data warehouse 31, the source data warehouse of the target data, i.e., the secondary data warehouse 21, the secondary data warehouse 22 and the tertiary data warehouse 31, can be determined by performing a lineage analysis on the target data. In the technical solution of the present application, when the target data exists in the form of an ODS table, the lineage analysis of the target data can be obtained by the similarity of the table name and table structure information of each data table in the multi-level data warehouse, wherein the table structure information may include field name, field type, field length, etc.; or it can also be analyzed through a variety of tools such as DRUID or Spark's Logicplan, and a corresponding data model is established to form a data lineage map. In fact, the data lineage analysis methods in the relevant technology can all be used in the technical solution of the present application, and the present application does not limit this. By performing a lineage analysis on the above-mentioned ODS table, the source of the data table can be traced back, the original data table corresponding to the above-mentioned ODS table in the multi-level data warehouse can be determined, and then the access path of the data warehouse that provides data to the above-mentioned ODS table can be determined. Of course, the source data warehouse of the target data can be one or more, and this application does not limit this.

[0041] In one embodiment, the target cleaning node maintains in advance the mapping relationship between the data warehouses at all levels and the cleaning nodes corresponding to each data warehouse. The above association relationship can be established by the target cleaning node, which only includes the association between the cleaning sub-nodes under the target cleaning node and their corresponding data warehouses; or the above association relationship can be established by the management party of the multi-level data warehouse, which includes the association relationship between all the data warehouses in the multi-level data warehouse and the cleaning nodes corresponding to each data warehouse. Figure 2, the data in the primary data warehouse 11 comes from the secondary data warehouse 21 and the secondary data warehouse 22. The cleaning node corresponding to the primary data warehouse 11 is the primary cleaning node 1A, the secondary data warehouse 21 corresponds to the secondary cleaning node 2A, and the secondary data warehouse 22 corresponds to the secondary cleaning node 2B. Using the above mapping relationship, the corresponding cleaning node can be queried through each data warehouse. For example, when the target cleaning node is the primary cleaning node 1A, it is assumed that it determines through the blood relationship analysis that the source data warehouse of the target data is the secondary data warehouse 21 and the secondary data warehouse 22, and then the target cleaning node can use the access path of the secondary data warehouse 21 and the secondary data warehouse 22 to query in the above mapping relationship, and obtain the secondary cleaning node 2A corresponding to the secondary data warehouse 21 and the secondary cleaning node 2B corresponding to the secondary data warehouse 22. 2A and 2B are called target cleaning sub-nodes.

[0042] In one case, the above mapping relationship may only record the corresponding relationship between the data warehouse and each cleaning node, wherein each cleaning node and the data warehouse are represented by an access path. After the target cleaning node obtains the access path of the source data warehouse through lineage analysis, the access path of the target cleaning sub-node corresponding to the source data warehouse is directly obtained by querying the above mapping relationship, and the target cleaning node can interact with the target cleaning sub-node through the above access path.

[0043] In another case, when the hierarchy of the multi-level data warehouse is more complex, in addition to recording the corresponding relationship between the data warehouse and each cleaning node, the above mapping relationship can also record the hierarchical relationship of the cleaning nodes at all levels. The above hierarchical relationship reflects the jurisdiction between the upper and lower cleaning nodes. For example, the target cleaning node can send the cleaning task to its lower cleaning sub-node. After the lower cleaning sub-node receives the cleaning task, it can also query whether the cleaning task is triggered by the upper cleaning node that has jurisdiction over it according to the above mapping relationship. Only the upper cleaning node that has jurisdiction over the cleaning sub-node can trigger it to create a cleaning task. When the above mapping relationship records the hierarchical relationship of the cleaning nodes at all levels, the above mapping relationship can be stored in the form of a mapping relationship tree. This application does not limit the specific organizational form of the mapping relationship. The hierarchical relationship of the cleaning nodes at all levels contained in the above mapping relationship can be established by the target cleaning node after being authenticated by the upper and lower cleaning nodes, or the target cleaning node can also be established by obtaining the hierarchical data maintained by the management party of the multi-level data warehouse.

[0044] Step 306: Synchronize the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

[0045] In one embodiment, after the target cleaning node determines the target cleaning subnode, it can establish a connection with the target cleaning subnode and synchronize the cleaning task to the target cleaning subnode through the established connection. The target cleaning subnode can send the cleaning task to its corresponding source data warehouse, control the source data warehouse to parse the cleaning task, identify the data that needs to be cleaned from its own stored data according to the information of the target data contained in the cleaning task, and clean the data that needs to be cleaned according to the cleaning rules recorded in the cleaning task to generate an STD table recording the cleaned data.

[0046] In one embodiment, after receiving the cleaning task sent by the target cleaning node, the target cleaning sub-node can directly convert the cleaning task into the SQL (Structured Query Language) language that can be recognized by the data warehouse and send it to the corresponding source data warehouse. Since the target data may come from multiple different data warehouses, the source data warehouse can parse the data cleaning task and filter out part of the target data stored in the data warehouse according to the target data information contained therein, and clean this part of the target data according to the cleaning rules contained in the cleaning task. Alternatively, a certain target cleaning sub-node can first parse the cleaning task, determine which target data is stored in its corresponding data warehouse, and create a data cleaning sub-task, which contains information about part of the target data in the above data warehouse, as well as cleaning rules, so that the corresponding data warehouse can clean the above part of the target data according to the above cleaning rules.

[0047] In one embodiment, the target cleaning node can synchronize the cleaning task and the execution cycle of the cleaning task to the target cleaning subnode, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning subnode according to the execution cycle. For example, the cleaning task can be performed periodically. Assuming that the cleaning cycle is 1 day, the execution frequency of this cleaning task is once a day. If the target cleaning subnode needs to be re-determined and the cleaning task needs to be re-issued each time the cleaning task is executed, it may cause a waste of resources. In order to solve the above problem, in this embodiment, in addition to synchronizing the cleaning task, the target cleaning node can also synchronize the execution cycle of the cleaning task to the target cleaning subnode. After the target cleaning subnode obtains the task execution cycle, it can adjust the cleaning process of the corresponding source database according to the task execution cycle. It is worth noting that periodic cleaning can only be for incremental data, that is, part of the data newly added within the time range corresponding to each cycle. For the upper-level data warehouse, the target cleaning node can create a data cleaning task for part of the incremental data summarized by the lower-level data warehouse in each cleaning cycle, so as to link the lower-level cleaning node to perform data cleaning for this part of the incremental data.

[0048] In one embodiment, the target cleaning node can call the extraction tool to obtain the cleaned data generated by each source data warehouse, and synchronize the cleaned data to any of the data warehouses. After the source data warehouse completes the cleaning of the target data, the target cleaning node can call the extraction tool (such as ETL tool) API (Application Programming Interface) interface related to data extraction to synchronize the cleaned data to the upper database corresponding to the target cleaning node. More specifically, the target cleaning node generates a lineage of the STD table after cleaning by the lineage of the ODS table recorded on this node. The lineage of the STD table expresses which upper data warehouse the cleaned data should be aggregated to. Each target cleaning sub-node can call the ETL tool to copy the preset data exchange task about the ODS table mentioned above, and replace the table name of the above data exchange task with the corresponding STD table. By completing the cleaning task at the target cleaning sub-node, the above ETL tool is periodically triggered to upload and aggregate the cleaned STD table to the STD table of the upper data warehouse corresponding to the target cleaning node.

[0049] Furthermore, in order to better control the cleaning process of the multi-level data warehouse, the target cleaning node can not only send the execution cycle of the cleaning task to the target cleaning sub-node, but also further control the latest time point for the completion of the cleaning and data synchronization process. For example, taking the execution cycle as 1 day as an example, the target cleaning node can further set the source data warehouse to synchronize the cleaned data to the upper-level data warehouse at 8 o'clock every morning. The target cleaning sub-node can determine the time to trigger data cleaning and trigger data synchronization based on the execution capability of the corresponding source data warehouse and the volume of the target data to be cleaned.

[0050] In one embodiment, the target cleaning node can filter the uncleaned data in any of the data warehouses; send a data cleaning task for the uncleaned data to any of the data warehouses, and control any of the data warehouses to clean the uncleaned data according to the data cleaning task for the uncleaned data. Considering that the source data warehouse will synchronize the cleaned data to the upper-level data warehouse, the upper-level data warehouse corresponding to the target cleaning node does not need to clean this part of the data again, otherwise it will cause a waste of data processing resources. Therefore, the target cleaning node can filter out the data in the upper-level data warehouse that has not been cleaned except the target data, and generate a new data cleaning task based on the above-mentioned uncleaned data. For example, when filtering the uncleaned data, the target cleaning node can filter according to the partition information of the lower-level database, and the above-mentioned partition information can be extracted by the target cleaning node from the partition field in the above-mentioned ODS table. The cleaning rules contained in the above-mentioned new data cleaning task can be the same as the cleaning rules of the source data warehouse, and it contains data information of the data that has not been cleaned. Then, after the target cleaning node sends the new data cleaning task to the upper-level data warehouse, the upper-level data warehouse can clean the data that has not been cleaned according to the above-mentioned data cleaning task. As for the target data, the cleaned data synchronized from the source data warehouse can be directly used as the cleaned data.

[0051] It can be seen from the technical solution provided by the present application that in the present application, the cleaning node corresponding to any data warehouse in the multi-level data warehouse can determine the source data warehouse of the target data that needs to be cleaned, and then combine the mapping relationship between the data warehouse and the cleaning node to send the cleaning task to the cleaning sub-node corresponding to the determined source data warehouse. Since the cleaning task contains cleaning rules, through the method of the present application, the cleaning rules of any data warehouse and the cleaning rules of the source data warehouse that provides data to this data warehouse will be consistent, which improves the uniformity of data cleaning. At the same time, compared with the method of extracting all the data in the lower-level data warehouse to the upper-level data warehouse for the same cleaning, the technical solution provided by the present application can also share the pressure of cleaning data to the lower-level data warehouse, reduce the cleaning pressure of the upper-level data warehouse, and improve the efficiency of data cleaning. In addition, the present application can also realize the functions of controlling the execution cycle of data cleaning, synchronizing the cleaned data to the upper-level data warehouse, filtering the cleaned data to prevent secondary cleaning, etc., which improves the convenience of the data cleaning process of the multi-level data warehouse and enriches the functions of the data cleaning process of the data warehouse.

[0052] Corresponding to the above method embodiment, this specification also provides an embodiment of a device.

[0053] Figure 4is a schematic diagram of a data cleaning electronic device according to an exemplary embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other services. The processor 402 reads the corresponding computer program from the non-volatile memory 410 into the memory 408 and then runs it, forming a device for solving data cleaning of multi-level data warehouses at the logical level. Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0054] Figure 5 is a block diagram of a data cleaning device according to an exemplary embodiment of the present application. Figure 5 The device is applied to a target cleaning node corresponding to any data warehouse in a multi-level data warehouse synchronization unit library, and the device includes: a creation unit 502, a determination unit 504, and a synchronization unit 506.

[0055] A creating unit 502 determines target data to be cleaned from any data warehouse and creates a cleaning task for the target data, wherein the cleaning task includes information about the target data and cleaning rules;

[0056] The determining unit 504 determines the source data warehouse of the target data, and determines the target cleaning sub-node corresponding to the source data warehouse according to the mapping relationship between the data warehouse and the cleaning node;

[0057] The synchronization unit 506 synchronizes the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

[0058] Optionally, the device further includes: a lineage analysis unit 508, wherein determining the source data warehouse of the target data includes:

[0059] A lineage analysis is performed on the target data to determine a source data warehouse of the target data.

[0060] Optionally, in the multi-level data warehouse, any one of the data warehouses includes a parent data warehouse of the source data warehouse.

[0061] Optionally, the device further comprises: a mapping relationship establishing unit 510, wherein the mapping relationship is established by the target cleaning node, and the mapping relationship at least comprises an association between each cleaning sub-node of the target cleaning node and its corresponding data warehouse.

[0062] Optionally, the device further includes: a periodic synchronization unit 512, wherein synchronizing the cleaning task to the target cleaning sub-node so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node includes:

[0063] The cleaning task and the execution cycle of the cleaning task are synchronized to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the execution cycle based on the cleaning task issued by the corresponding target cleaning sub-node.

[0064] Optionally, the device further includes: a data synchronization unit 514, configured to call an extraction tool to obtain cleaned data generated by each source data warehouse, and synchronize the cleaned data to any of the data warehouses.

[0065] Optionally, the device also includes: a screening unit 516, used to screen the uncleaned data in any of the data warehouses; send a data cleaning task for the uncleaned data to any of the data warehouses, and control any of the data warehouses to clean the uncleaned data according to the data cleaning task for the uncleaned data.

[0066] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0067] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0068] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions. The instructions may be executed by a processor of a data cleaning device to implement any of the methods described in the above embodiments. For example, the method may include:

[0069] Determine the target data to be cleaned from any of the data warehouses and create a cleaning task for the target data, wherein the cleaning task includes the target data information and cleaning rules; determine the source data warehouse of the target data, and determine the target cleaning sub-node corresponding to the source data warehouse according to the mapping relationship between the data warehouse and the cleaning node; synchronize the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node.

[0070] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0071] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data cleaning method, characterized in that: A target cleaning node for controlling a data cleaning process corresponding to any data warehouse in a multi-level data warehouse, wherein any data warehouse in the multi-level data warehouse has a lower-level data warehouse that provides data sources to the target cleaning node, the method comprising: Determine target data to be cleaned from any data warehouse in the multi-level data warehouse and create a cleaning task for the target data, wherein the cleaning task includes the target data information and cleaning rules; Determine the source data warehouse of the target data, and determine the target cleaning sub-node corresponding to the source data warehouse according to the mapping relationship between the data warehouse and the cleaning node; Synchronizing the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node; Calling an extraction tool to obtain cleaned data generated by a source data warehouse of each of the target data, and synchronizing the cleaned data to any data warehouse in the multi-level data warehouse; Filtering the uncleaned data in any data warehouse in the multi-level data warehouse; A data cleaning task for the uncleaned data is issued to any data warehouse in the multi-level data warehouse, and any data warehouse in the multi-level data warehouse is controlled to clean the uncleaned data according to the data cleaning task for the uncleaned data.

2. The method according to claim 1, characterized in that: The step of determining the source data warehouse of the target data includes: A lineage analysis is performed on the target data to determine a source data warehouse of the target data.

3. The method according to claim 1, characterized in that: Any data warehouse in the multi-level data warehouse includes a parent data warehouse of the source data warehouse.

4. The method according to claim 1, characterized in that: The mapping relationship is established by the target cleaning node, and the mapping relationship at least includes the association between each cleaning sub-node of the target cleaning node and its corresponding data warehouse.

5. The method according to claim 1, characterized in that The step of synchronizing the cleaning task to the target cleaning sub-node so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node includes: The cleaning task and the execution cycle of the cleaning task are synchronized to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the execution cycle based on the cleaning task issued by the corresponding target cleaning sub-node.

6. A data cleaning device, characterized in that: A target cleaning node for controlling a data cleaning process corresponding to any data warehouse in a multi-level data warehouse, wherein any data warehouse in the multi-level data warehouse has a lower-level data warehouse that provides data sources to the target cleaning node, the device comprises: A creating unit, which determines target data to be cleaned from any data warehouse in the multi-level data warehouse and creates a cleaning task for the target data, wherein the cleaning task includes the target data information and a cleaning rule; A determination unit, which determines a source data warehouse of the target data, and determines a target cleaning sub-node corresponding to the source data warehouse according to a mapping relationship between the data warehouse and the cleaning node; A synchronization unit, synchronizing the cleaning task to the target cleaning sub-node, so that the source data warehouse cleans the target data according to the cleaning task issued by the corresponding target cleaning sub-node; The synchronization unit further calls an extraction tool to obtain the cleaned data generated by the source data warehouse of each of the target data, and synchronizes the cleaned data to any data warehouse in the multi-level data warehouse; A screening unit, screening the uncleaned data in any data warehouse in the multi-level data warehouse; The cleaning unit sends a data cleaning task for the uncleaned data to any data warehouse in the multi-level data warehouse, and controls any data warehouse in the multi-level data warehouse to clean the uncleaned data according to the data cleaning task for the uncleaned data.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 5 by running the executable instructions.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images