Data cleaning method and device, storage medium and electronic equipment

Abstract

The application discloses a data cleaning method and device, a storage medium and an electronic device. The method comprises the following steps: in response to an operation command of a target object to a plurality of operation nodes, a first flowchart is generated, wherein each operation node is used for executing at least one operation of a target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution sequence of the plurality of operation nodes; according to the first flowchart, a cleaning rule of the target data cleaning task is determined, wherein the cleaning rule at least comprises a running rule of the plurality of operation nodes; based on the cleaning rule, a data cleaning process is performed on a source data set corresponding to the target data cleaning task to obtain a target cleaning result, wherein the target cleaning result represents whether the data cleaning process is successful. The application solves the technical problem of low execution efficiency of a data cleaning task in the prior art by using a code writing method for data cleaning.

Description

Technical Field

[0001] This invention relates to the field of computer software control technology, and more specifically, to a data cleaning method, apparatus, storage medium, and electronic device. Background Technology

[0002] As data awareness increases among employees at all levels within enterprises, traditional data preparation and report development tools designed for developers are time-consuming, inefficient, and slow to respond. More and more business personnel are starting to use self-service analysis systems. However, in many cases, the original data source cannot support the requirements of the final analysis scenario, and most self-service business intelligence analysis tools do not include data cleaning functions. They still require professional data engineers to process the data by writing SQL statements and code, which results in high development costs and significant communication costs. Data engineers also spend a lot of time debugging and implementing the code, while business personnel spend considerable time verifying and providing feedback on the final data, which greatly reduces real-time performance and leads to low efficiency in data cleaning tasks.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a data cleaning method, apparatus, storage medium, and electronic device to at least solve the technical problem of low execution efficiency of data cleaning tasks in the prior art, which uses code-based data cleaning.

[0005] According to one aspect of the present invention, a data cleaning method is provided, comprising: responding to an operation command from a target object to multiple operation nodes, generating a first flowchart, wherein each operation node is used to execute at least one operation of a target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of the multiple operation nodes; determining cleaning rules for the target data cleaning task based on the first flowchart, wherein the cleaning rules include at least the operation rules of the multiple operation nodes; and performing data cleaning processing on the source dataset corresponding to the target data cleaning task based on the cleaning rules to obtain a target cleaning result, wherein the target cleaning result represents whether the data cleaning processing was successful.

[0006] Furthermore, based on the cleaning rules, the source dataset corresponding to the target data cleaning task is cleaned to obtain the target cleaning result, including: determining the node status of each operation node, wherein the node status is one of the following: unmodified state or modified state; obtaining the operation rules of multiple target operation nodes from the operation rules of each operation node to generate the first cleaning rule, wherein the node status of multiple target operation nodes is modified state; and based on the first cleaning rule, the source dataset corresponding to the target data cleaning task is cleaned to obtain the target cleaning result.

[0007] Further, based on the first cleaning rule, data cleaning processing is performed on the source dataset corresponding to the target data cleaning task to obtain the target cleaning result, including: based on the first cleaning rule, determining the first execution command and the second execution command from the execution commands corresponding to each target operation node, wherein the execution order of the first execution command takes precedence over the execution order of the second execution command; executing the first execution command to perform data cleaning processing on the source dataset corresponding to the target data cleaning task to generate the first dataset; executing the second execution command to perform data cleaning processing on the first dataset to generate the second dataset, wherein the data standardization of the second dataset is higher than that of the first dataset; and generating the target cleaning result based on the second dataset.

[0008] Further, based on the second dataset, the target cleaning result is generated, including: determining the execution status of the first operation node, wherein the execution status is one of the following: executing state, not executing state; if the execution status is not executing state, updating the execution status to executing state, and obtaining the running rules of the first operation node, wherein the running rules of the first operation node include at least the output mode of the second dataset, wherein the output mode is one of the following: first output mode, second output mode, and the data volume corresponding to the second output mode is less than the data volume corresponding to the first output mode; if the output mode of the second dataset is the first output mode, determining whether the data structure of the second dataset has changed; if the data structure of the second dataset has changed, updating the data structure of the second dataset; after updating the data structure of the second dataset, or if the data structure of the second dataset has not changed, creating a first data table, and writing the data of the second dataset from the current data table into the first data table, generating a write result, wherein the write result indicates whether the data of the second dataset was successfully written from the current data table into the first data table; and generating the target cleaning result based on the write result.

[0009] Furthermore, based on the writing result, a target cleaning result is generated, including: if the writing result indicates that the data of the second dataset was successfully written from the current data table to the first data table, the names of the current data table and the first data table are swapped, and the current data table with the name of the first data table is deleted; the target dataset is obtained based on the first data table with the name of the current data table, and a data cleaning success result is generated; if the writing result indicates that the data of the second dataset was not successfully written from the current data table to the first data table, the first data table is deleted, and a data cleaning failure result is generated.

[0010] Furthermore, the data cleaning method also includes: when the output mode of the second dataset is the second output mode, deleting the data in the target area and writing the target data in the second dataset from the current data table into the target area; after writing the target data in the second dataset from the current data table into the target area, obtaining the target dataset based on the target area, and generating a successful data cleaning result.

[0011] Furthermore, the data cleaning method also includes: after generating a first flowchart in response to the target object's operation commands to multiple operation nodes, calculating the lineage relationship between the multiple operation nodes to obtain a first lineage relationship set; calculating the lineage relationship between the target fields in the multiple operation nodes to obtain a second lineage relationship set; storing the first lineage relationship set and the second lineage relationship set to generate the lineage relationship set corresponding to the target data cleaning task.

[0012] Furthermore, the data cleaning method also includes: after performing data cleaning processing on the source dataset corresponding to the target data cleaning task based on cleaning rules to obtain the target cleaning result, sending the target cleaning result to the target interface and rendering it on the target interface; and displaying the target cleaning result to the target object through the target interface.

[0013] According to another aspect of the present invention, a data cleaning apparatus is also provided, comprising: a first processing module, configured to generate a first flowchart in response to an operation command from a target object to multiple operation nodes, wherein each operation node is configured to perform at least one operation of a target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of the multiple operation nodes; a determining module, configured to determine cleaning rules for the target data cleaning task based on the first flowchart, wherein the cleaning rules include at least the running rules of the multiple operation nodes; and a second processing module, configured to perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on the cleaning rules to obtain a target cleaning result, wherein the target cleaning result represents whether the data cleaning processing was successful.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described data cleaning method at runtime.

[0015] According to another aspect of the present invention, an electronic device is also provided, the electronic device including one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the above-described data cleaning method during runtime.

[0016] In this embodiment of the invention, a first flowchart is first generated in response to the target object's operation commands to multiple operation nodes. Then, based on the first flowchart, cleaning rules for the target data cleaning task are determined. Following these cleaning rules, data cleaning processing is performed on the source dataset corresponding to the target data cleaning task to obtain the target cleaning result. Each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart. The first flowchart represents the execution order of multiple operation nodes, the cleaning rules include at least the operating rules of multiple operation nodes, and the target cleaning result represents whether the data cleaning processing was successful.

[0017] In the above process, by responding to the target object's operation commands on multiple operation nodes, a first flowchart can be generated. This process collects user commands such as dragging and editing operation nodes on the target interface, obtaining a topological relationship that represents the link situation of the target data cleaning task. This provides an accurate data foundation for subsequent data cleaning, enabling ordinary business personnel to easily and quickly construct data cleaning tasks by dragging and dropping operation nodes. This reduces the communication costs between business personnel and developers and improves the response rate to data analysis needs. Based on the first flowchart, the cleaning rules of the target data cleaning task can be determined. Therefore, based on the cleaning rules, data cleaning processing can be performed on the source dataset corresponding to the target data cleaning task to obtain the target cleaning result. There is no need to write SQL statements and code, reducing development costs and thus improving the execution efficiency of the cleaning task.

[0018] Therefore, the technical solution of this invention achieves the goal of providing ordinary business personnel with an easy-to-operate, low-threshold, and intelligent data cleaning and processing tool, thereby improving the execution efficiency of cleaning tasks and solving the technical problem of low execution efficiency of data cleaning tasks in the prior art, which uses code writing for data cleaning. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart of an optional data cleaning method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of an optional core engine of a data cleaning system according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of an optional node type according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of an optional data cleaning system technical architecture according to an embodiment of the present invention;

[0024] Figure 5 This is a flowchart of an optional node execution according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of an optional node executing a command according to an embodiment of the present invention;

[0026] Figure 7 This is a flowchart illustrating the execution of an optional output node according to an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of an optional data cleaning apparatus according to an embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this invention are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0032] Example 1

[0033] According to an embodiment of the present invention, an embodiment of a data cleaning method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 1 This is a flowchart of an optional data cleaning method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0035] Step S101: Respond to the target object's operation commands to multiple operation nodes and generate a first flowchart, wherein each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of multiple operation nodes.

[0036] In the above steps, the target object's operation commands to multiple operation nodes can be responded to by devices such as application systems, processors, and electronic devices. Optionally, the data cleaning system can respond to the target object's operation commands to multiple operation nodes. The target object can be a user of the data cleaning system, such as a general business person or a non-technical person. The operation commands can be commands such as drag and drop, or editing node configurations. The first flowchart can be a topology diagram showing the link relationship of the target data cleaning task.

[0037] Optionally, the data cleaning system includes a graphical interface for receiving operation commands from the target object to multiple operation nodes. It displays the target data cleaning task, composed of these operation nodes, to the target object. For example, business personnel can arrange various types of operation nodes by dragging and dropping, or double-click a node to enter its configuration editing interface and set specific operations. Multiple operation nodes form a directed acyclic graph, displayed from left to right. The starting node can be the source dataset, located on the far left. Operation nodes can be connected to its right for subsequent cleaning steps, and the final node is the output node (located on the far right) used to generate a new dataset. Except for the output node, each operation node can have multiple operation nodes connected to its right. Each operation node can create a common successor node with its non-predecessor and non-successor operation nodes, enabling operations such as union and association. The interface also allows setting scheduled tasks, manually executing and stopping tasks, viewing task execution flow and status information, and viewing historical execution records.

[0038] Figure 2 This is a schematic diagram of an optional core engine of a data cleaning system according to an embodiment of the present invention, such as... Figure 2 As shown, the core engine of the data cleaning system includes an overall control engine, an analysis engine, and a lineage engine. Optionally, the overall control engine responds to the target object's operation commands on multiple operation nodes, generating a first flowchart, which involves receiving and saving user operation records.

[0039] Optionally, the main functions of the overall control engine include: receiving and saving user operation records; generating execution plans (i.e., cleaning rules for target data cleaning tasks) based on the operation records and providing them to the analysis engine for execution; overall scheduling the execution of operation nodes and the output of output nodes; obtaining the execution results of the analysis engine and rendering them on the interface; and providing metadata to the lineage engine for calculation.

[0040] Optionally, to improve processing efficiency and stability, the overall control engine mainly adopts stateless technology and concurrent execution technology. Stateless technology means that no state data is stored in memory, which supports arbitrary horizontal scaling. Concurrent execution technology means that the execution and output of nodes are executed concurrently using task queues and custom thread pools to ensure processing speed and reduce the pressure on the analysis engine.

[0041] Optionally, the main functions of the analytics engine include: querying node data and outputting data to a dataset. Optionally, to improve performance and enhance the ability to clean massive amounts of data, the analytics engine primarily employs the following techniques:

[0042] (1) Read-write separation. The analysis engine is deployed in a cluster mode, with read and write operations performed on different instances. The cluster can be dynamically scaled horizontally according to the amount of business data.

[0043] (2) Rule-Based Optimization (RBO). The backend provides a series of optimization rules tailored to engine characteristics, such as automatically adjusting the association mode for related nodes, automatically creating indexes and partitions for commonly used fields, and automatically creating materialized views for nodes with excessive depth. Furthermore, an extension interface is provided, allowing users to add custom optimization rules based on experience.

[0044] (3) Incremental output and parallel output. When outputting data, incremental partitioning is supported. When only some business data changes, incremental output can be used to reduce the amount of output data and speed up the output efficiency. Batch output function can also be used to output multiple nodes in parallel.

[0045] Optionally, the main functions of the lineage engine include: calculating and recording the lineage relationships between nodes and between node fields; querying the lineage of process nodes, which will be calculated sequentially from front to back according to the lineage of nodes during process execution; and querying the lineage relationship of node fields, which is used to determine whether there are circular dependencies or breaks in the fields, so as to avoid the formation of a loop graph that would cause the cleaning task to fail.

[0046] Figure 3 This is a schematic diagram of an optional node type according to an embodiment of the present invention, such as... Figure 3 As shown, the operation nodes include input nodes, cleanup nodes, aggregation nodes, association nodes, row-to-column nodes, column-to-row nodes, union nodes, and output nodes.

[0047] Figure 4 This is a schematic diagram of an optional data cleaning system technical architecture according to an embodiment of the present invention, such as... Figure 4 As shown, the technical architecture of the data cleaning system includes an interactive interface, data services, core functions, and data storage. Optionally, this invention provides web pages and backend interfaces for functions such as process management, data cleaning, data preview, lineage calculation, data output, and task scheduling. These functions are implemented through the aforementioned core engines. For example, process management and task scheduling functions are implemented through the overall control engine, data cleaning functions are implemented through the analysis engine, and lineage calculation functions are implemented through the lineage engine.

[0048] Optionally, the interactive interface includes HTML for data processing settings, HTML for viewing result sets, and HTML for analyzing result sets. Optionally, process management provides functions such as creating, modifying, deleting, and querying processes, managing process tags, and importing / exporting processes. Data cleaning provides functions for creating, modifying, deleting, and querying process nodes, as well as node configuration, adding, rolling back cleaning operation commands, and executing process nodes. Data preview provides real-time preview and sorting of node data. Lineage calculation provides functions for calculating and querying the lineage relationship of nodes and fields. Data output provides functions for incremental and full output of data to the dataset. Task scheduling provides functions for adding, modifying, querying, deleting, and adding scheduled output tasks.

[0049] It should be noted that this invention features no-code functionality, meaning all data cleaning operations are performed through a graphical interface using drag-and-drop methods, eliminating the need for coding; real-time preview capabilities, allowing users to see the results of each step in real time; support for massive datasets, utilizing the high-performance ClickHouse analytics engine to process datasets of hundreds of millions of records with sub-second response times; operation rollback functionality, displaying all data cleaning operation records on the interface, allowing users to undo and roll back each step for optimal cleaning results; and data lineage functionality, providing field-level data lineage, enabling users to view the lineage of each field for easy tracing and identification of erroneous fields.

[0050] Step S102: Based on the first flowchart, determine the cleaning rules for the target data cleaning task, wherein the cleaning rules include at least the operation rules for multiple operation nodes.

[0051] In the above steps, according to the first flowchart, configuration information of multiple operation nodes is obtained, that is, configuration information edited by the user by double-clicking a node. Then, based on the configuration information, the cleaning rules can be determined. Optionally, the cleaning rules consist of operation steps corresponding to the cleaning commands generated based on the configuration information of the operation nodes, and the execution rules can be rules formed by the cleaning commands generated based on the configuration information of the operation nodes.

[0052] Step S103: Based on the cleaning rules, perform data cleaning processing on the source dataset corresponding to the target data cleaning task to obtain the target cleaning result, wherein the target cleaning result represents whether the data cleaning processing is successful.

[0053] In the above steps, the analysis engine can perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on cleaning rules to obtain the target cleaning result.

[0054] Based on the scheme defined in steps S101 to S103 above, it can be understood that in this embodiment of the invention, firstly, a first flowchart is generated in response to the target object's operation commands to multiple operation nodes. Then, based on the first flowchart, the cleaning rules for the target data cleaning task are determined. Finally, based on the cleaning rules, data cleaning processing is performed on the source dataset corresponding to the target data cleaning task to obtain the target cleaning result. Each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart. The first flowchart represents the execution order of multiple operation nodes. The cleaning rules include at least the running rules of multiple operation nodes. The target cleaning result represents whether the data cleaning processing was successful.

[0055] It is noteworthy that in the above process, by responding to the target object's operation commands on multiple operation nodes, a first flowchart can be generated. This enables the collection of user commands such as dragging and editing operation nodes on the target interface, obtaining a topological relationship that represents the link situation of the target data cleaning task. This provides an accurate data foundation for subsequent data cleaning, allowing ordinary business personnel to easily and quickly construct data cleaning tasks by dragging and dropping operation nodes, reducing the communication costs between business personnel and developers, and improving the response rate to data analysis needs. Based on the first flowchart, the cleaning rules of the target data cleaning task can be determined, enabling data cleaning processing of the source dataset corresponding to the target data cleaning task based on the cleaning rules, obtaining the target cleaning result without writing SQL statements and code, reducing development costs, and thus improving the execution efficiency of the cleaning task.

[0056] Therefore, the technical solution of this invention achieves the goal of providing ordinary business personnel with an easy-to-operate, low-threshold, and intelligent data cleaning and processing tool, thereby improving the execution efficiency of cleaning tasks and solving the technical problem of low execution efficiency of data cleaning tasks in the prior art, which uses code writing for data cleaning.

[0057] In one optional embodiment, during the process of cleaning the source dataset corresponding to the target data cleaning task based on cleaning rules to obtain the target cleaning result, the node state of each operation node is first determined. Then, the operation rules of multiple target operation nodes are obtained from the operation rules of each operation node to generate a first cleaning rule. Finally, based on the first cleaning rule, the source dataset corresponding to the target data cleaning task is cleaned to obtain the target cleaning result. The node state can be one of the following: unmodified, modified, or the node state of multiple target operation nodes is modified.

[0058] Figure 5 This is a flowchart of an optional node execution according to an embodiment of the present invention, such as... Figure 5 As shown, the process first checks if the modification flag is 0, thus determining the node status of each operation node. If the modification flag is 0, the node is considered unmodified, its status is "unmodified," and the process ends. If the modification flag is not 0, for example, if it is 1, the node status is "modified." Further, a context is constructed, the node execution program is initialized, and the running rules (i.e., all execution commands corresponding to the nodes) for multiple target operation nodes (i.e., nodes with a modified status) are obtained. Commands of the same type are merged to generate the first cleaning rule. Optionally, user cleaning operations on nodes are abstracted into a single execution command in the background, such as deleting a column or adding a calculated column. Merging commands of the same type avoids redundant calculations. For example, the execution commands for node A might include command 1 to change the node name to B and command 2 to change the node name to C. By merging commands of the same type, only the last configured command to change the node name needs to be executed, according to the configured execution time.

[0059] Furthermore, based on the first cleaning rule, the source dataset corresponding to the target data cleaning task is cleaned to obtain the target cleaning result.

[0060] In one optional embodiment, during the process of cleaning the source dataset corresponding to the target data cleaning task based on the first cleaning rule to obtain the target cleaning result, firstly, based on the first cleaning rule, a first execution command and a second execution command are determined from the execution commands corresponding to each target operation node. Then, the first execution command is executed to clean the source dataset corresponding to the target data cleaning task, generating a first dataset. Then, the second execution command is executed to clean the first dataset, generating a second dataset. Finally, the target cleaning result is generated based on the second dataset. The execution order of the first execution command takes precedence over the execution order of the second execution command, and the data standardization level of the second dataset is higher than that of the first dataset.

[0061] Optionally, the first command to execute can be a pre-command, and the second command to execute can be a post-command, such as... Figure 5 As shown, after the merge command obtains the first cleaning rule, the pre-processing commands are executed to clean the source dataset corresponding to the target data cleaning task, generating the first dataset. Specifically, the commands are executed in the order of their creation time. For example, commands such as adding or deleting columns are pre-processing commands, while formatting modification commands such as deleting spaces are post-processing commands.

[0062] Optional, such as Figure 5As shown, metadata for the operation nodes is stored, including at least node information, field information, execution records, and data lineage. Then, a physical table is created to store the first dataset obtained after cleaning the source dataset by executing pre-processing commands. A view is then generated in the analysis engine based on the query plan.

[0063] Furthermore, such as Figure 5 As shown, the post-execution command (i.e., the second execution command) cleanses the first dataset, generating the second dataset. For example, executing a formatting modification command such as deleting spaces corrects the first dataset, resulting in a corrected second dataset. Optionally, after executing the second execution command, the modification flag is reset to 0, i.e., the node state is set to the unmodified state.

[0064] Figure 6 This is a schematic diagram of an optional node executing commands according to an embodiment of the present invention, such as... Figure 6 As shown, node execution commands (node ​​operation commands) include general commands, cleanup commands, and rule-based optimizations. General commands include data filtering, adding calculated columns, adding serial number columns, removing columns, restoring columns, deleting columns, changing field types, changing display names, splitting columns, and sorting fields. Cleanup commands include deleting spaces, deleting special characters, converting to uppercase, converting to lowercase, and replacing values. Rule-based optimizations include left-association rule optimization and materialized view rule optimization. Optionally, an operation node includes at least one execution command. For example, a user can double-click to edit operation node A and configure commands such as deleting a column or changing the field type for operation node A.

[0065] Furthermore, based on the second dataset, target cleaning results are generated.

[0066] In one optional embodiment, during the process of generating the target cleaning result based on the second dataset, the execution status of the first operation node is first determined. If the execution status is not executed, it is updated to be executed, and the running rules of the first operation node are obtained. Then, if the output mode of the second dataset is the first output mode, it is determined whether the data structure of the second dataset has changed. If the data structure of the second dataset has changed, it is updated. After updating the data structure of the second dataset, or if the data structure of the second dataset has not changed, a first data table is created, and the data of the second dataset is written from the current data table to the first data table, generating a write result. Then, based on the write result, the target cleaning result is generated. The execution status is one of the following: executing or not executed. The running rules of the first operation node include at least one of the output modes of the second dataset, which is one of the following: first output mode or second output mode. The data volume corresponding to the second output mode is less than the data volume corresponding to the first output mode. The write result indicates whether the data of the second dataset was successfully written from the current data table to the first data table.

[0067] Optionally, the first operation node can be an output node, the first output mode can be full output, and the second output mode can be incremental output. Figure 7 This is a flowchart illustrating the execution of an optional output node according to an embodiment of the present invention, such as... Figure 7 As shown, the process first determines whether the output node is currently outputting, i.e., determines the execution status of the first operation node. If the execution status is currently executing, the process ends; if the execution status is not executing, the execution status is updated to currently executing to prevent duplicate output, and the running rules of the first operation node are obtained.

[0068] Furthermore, it is determined whether the output mode is full output. If the output mode of the second dataset is the first output mode (i.e., full output), it is determined whether the data structure of the second dataset has changed. If the data structure of the second dataset has changed, the data structure of the second dataset is updated (i.e., the metadata structure is updated).

[0069] Furthermore, after updating the data structure of the second dataset, or if the data structure of the second dataset remains unchanged, a first data table is created (i.e., a new physical table is created), and data from the second dataset is written from the current data table to the first data table (i.e., the new physical table), generating a write result.

[0070] Furthermore, based on the writing results, the target cleaning results are generated.

[0071] In one optional embodiment, during the process of generating the target cleaning result based on the writing result, if the writing result indicates that the data of the second dataset was successfully written from the current data table to the first data table, the names of the current data table and the first data table are swapped, and the current data table with the name of the first data table is deleted. Then, the target dataset is obtained based on the first data table with the name of the current data table, and a data cleaning success result is generated. If the writing result indicates that the data of the second dataset was not successfully written from the current data table to the first data table, the first data table is deleted, and a data cleaning failure result is generated.

[0072] Optional, such as Figure 7 As shown, the process determines whether there are any abnormalities in the data writing. If the writing result indicates that the data of the second dataset has been successfully written from the current data table to the first data table, that is, there are no abnormalities in the writing, then the names of the current data table and the first data table are swapped (that is, the new table and the old table are renamed), and the current data table with the name of the first data table is deleted (that is, the new table is deleted). Then, based on the first data table with the name of the current data table, the target dataset is obtained, and a data cleaning success result is generated, and the status is changed to output success.

[0073] Optionally, if the write result indicates that the data of the second dataset was not successfully written from the current data table to the first data table, i.e., there is an error in the write, then the first data table is deleted (i.e., a new table is deleted), and a data cleaning failure result is generated, and the status is changed to output failure.

[0074] In one optional embodiment, when the output mode of the second dataset is the second output mode, the data in the target region is deleted, and the target data in the second dataset is written from the current data table to the target region. After the target data in the second dataset is written from the current data table to the target region, the target dataset is obtained according to the target region, and a data cleaning success result is generated.

[0075] Optionally, the target region can be an incremental partition, and the target data can be incremental data. Optionally, if the output mode of the second dataset is the second output mode (i.e., incremental output), then the data in the incremental partition is deleted, and the incremental data from the second dataset in the current data table is written to the incremental partition. Then, based on the data in the incremental partition, the target dataset is obtained, and a data cleaning success result is generated.

[0076] In one optional embodiment, after generating a first flowchart in response to the target object's operation commands to multiple operation nodes, the lineage relationships between the multiple operation nodes are calculated to obtain a first lineage relationship set; the lineage relationships between target fields in the multiple operation nodes are calculated to obtain a second lineage relationship set; the first lineage relationship set and the second lineage relationship set are stored to generate a lineage relationship set corresponding to the target data cleaning task.

[0077] Optionally, the lineage engine calculates the lineage relationships between multiple operation nodes to obtain a first lineage relationship set, and calculates the lineage relationships between target fields in multiple operation nodes to obtain a second lineage relationship set. The first and second lineage relationship sets are then stored to generate the lineage relationship set corresponding to the target data cleaning task.

[0078] It should be noted that this embodiment provides data lineage query functions at the node level and field level. Users can view the lineage chain of each node and each field in the operation node layout from left to right through a graphical interface. That is, they can trace the lineage relationship of each node and each field on the graphical interface. They can also trace the data processing flow used from any analysis scenario. In this way, when anomalies occur during data cleaning, the problematic node or field can be quickly queried based on the lineage relationship, thereby further improving the execution efficiency of the cleaning task.

[0079] In one optional embodiment, after cleaning the source dataset corresponding to the target data cleaning task based on the cleaning rules and obtaining the target cleaning result, the target cleaning result is sent to the target interface and rendered on the target interface, and the target cleaning result is displayed to the target object through the target interface.

[0080] Optionally, the target interface can be the aforementioned graphical interface. The analysis engine sends the target cleaning results to the graphical interface for rendering and displays the target cleaning results to the user through the graphical interface.

[0081] It's important to note that before data analysis, processing and cleaning are often necessary to ensure reliable data quality. Current technologies typically require specialized data engineers to handle this through SQL statements or code. This embodiment, based on ClickHouse technology, enables real-time cleaning of massive datasets. It's particularly suitable for non-data engineers to perform ETL processing on existing massive datasets based on business needs, outputting new datasets for business analysis. This solves the problem that only data engineers can perform data processing and cleaning, allowing ordinary business personnel to perform a series of operations such as aggregation, row and column transformation, union, association, filtering, and cleaning through a graphical interface, generating datasets suitable for specific analytical scenarios. The graphical interface clearly and intuitively expresses data processing operations such as aggregation, row and column transformation, union, association, filtering, and cleaning, allowing anyone without programming experience to easily get started. The entire data processing workflow can be orchestrated and output through simple drag-and-drop operations, reducing communication costs between business personnel and developers and improving the response speed to data analysis needs.

[0082] Furthermore, using a graph database (e.g., Neo4j) to record the relationships between nodes allows the directed acyclic graph (DAG) formed during the orchestration process to be read and displayed instantly. This solves the data black-box problem inherent in traditional SQL or ETL data processing and cleaning. Moreover, using ClickHouse to process data operations in the background transforms the orchestration process displayed on the graphical interface into a series of efficient ClickHouse operations, enabling the generation of cleaning results from the beginning to that node immediately after setting any node. Additionally, the final output node allows the processed and cleaned data to be directly output to the analysis system, enabling immediate analysis upon output. This solves the problem of data not being usable immediately after processing.

[0083] Therefore, the technical solution of this invention achieves the goal of providing ordinary business personnel with an easy-to-operate, low-threshold, and intelligent data cleaning and processing tool, thereby improving the execution efficiency of cleaning tasks and solving the technical problem of low execution efficiency of data cleaning tasks in the prior art, which uses code writing for data cleaning.

[0084] Example 2

[0085] According to an embodiment of the present invention, a data cleaning apparatus is provided, wherein, Figure 8 This is a schematic diagram of an optional data cleaning apparatus according to an embodiment of the present invention, such as... Figure 8As shown, the device includes: a first processing module 801, used to respond to operation commands from a target object to multiple operation nodes and generate a first flowchart, wherein each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of the multiple operation nodes; a determining module 802, used to determine the cleaning rules of the target data cleaning task according to the first flowchart, wherein the cleaning rules include at least the running rules of the multiple operation nodes; and a second processing module 803, used to perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on the cleaning rules to obtain a target cleaning result, wherein the target cleaning result represents whether the data cleaning processing was successful.

[0086] It should be noted that the first processing module 801, the determining module 802 and the second processing module 803 mentioned above correspond to steps S101 to S103 in the above embodiments. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment 1.

[0087] Optionally, the second processing module includes: a first determining unit, used to determine the node state of each operation node, wherein the node state is one of the following: unmodified state or modified state; a first obtaining unit, used to obtain the operation rules of multiple target operation nodes from the operation rules of each operation node, and generate a first cleaning rule, wherein the node state of the multiple target operation nodes is modified state; and a first processing unit, used to perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on the first cleaning rule, and obtain the target cleaning result.

[0088] Optionally, the first processing unit includes: a first determining submodule, used to determine a first execution command and a second execution command from the execution commands corresponding to each target operation node based on a first cleaning rule, wherein the execution order of the first execution command takes precedence over the execution order of the second execution command; a first execution submodule, used to execute the first execution command to perform data cleaning processing on the source dataset corresponding to the target data cleaning task, generating a first dataset; a second execution submodule, used to execute the second execution command to perform data cleaning processing on the first dataset, generating a second dataset, wherein the data standardization level of the second dataset is higher than that of the first dataset; and a third execution submodule, used to generate a target cleaning result based on the second dataset.

[0089] Optionally, the third execution submodule includes: a first determining subunit, used to determine the execution status of the first operation node, wherein the execution status is one of the following: executing state, not executing state; a first obtaining subunit, used to update the execution status to executing state when the execution status is not executing state, and obtain the running rules of the first operation node, wherein the running rules of the first operation node include at least the output mode of the second dataset, wherein the output mode is one of the following: first output mode, second output mode, and the data volume corresponding to the second output mode is less than the data volume corresponding to the first output mode; a first judging subunit, used to judge whether the data structure of the second dataset has changed when the output mode of the second dataset is the first output mode; a first updating subunit, used to update the data structure of the second dataset when the data structure of the second dataset has changed; a first writing subunit, used to create a first data table after updating the data structure of the second dataset, or when the data structure of the second dataset has not changed, and write the data of the second dataset from the current data table to the first data table, generating a writing result, wherein the writing result indicates whether the data of the second dataset has been successfully written from the current data table to the first data table; and a first generating subunit, used to generate the target cleaning result based on the writing result.

[0090] Optionally, the first generation subunit includes: a swapping processing subunit, used to swap the names of the current data table and the first data table, and delete the current data table whose name is the same as the name of the first data table, if the write result indicates that the data of the second dataset has been successfully written from the current data table to the first data table; a second generation subunit, used to obtain the target dataset based on the first data table whose name is the same as the name of the current data table, and generate a data cleaning success result; and a third generation subunit, used to delete the first data table and generate a data cleaning failure result if the write result indicates that the data of the second dataset has not been successfully written from the current data table to the first data table.

[0091] Optionally, the data cleaning apparatus further includes: a deletion module, used to delete data in the target area when the output mode of the second dataset is the second output mode, and write the target data in the second dataset from the current data table into the target area; and a generation module, used to obtain the target dataset according to the target area after writing the target data in the second dataset from the current data table into the target area, and generate a data cleaning success result.

[0092] Optionally, the data cleaning device further includes: a first calculation module for calculating the lineage relationships between multiple operation nodes to obtain a first lineage relationship set; a second calculation module for calculating the lineage relationships between target fields in multiple operation nodes to obtain a second lineage relationship set; and a storage module for storing the first and second lineage relationship sets to generate a lineage relationship set corresponding to the target data cleaning task.

[0093] Optionally, the data cleaning device further includes: a sending module for sending the target cleaning results to the target interface and rendering them on the target interface; and a display module for displaying the target cleaning results to the target object through the target interface.

[0094] Example 3

[0095] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described data cleaning method at runtime.

[0096] Example 4

[0097] According to another aspect of the present invention, an electronic device is also provided, wherein, Figure 9 This is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as... Figure 9 As shown, the electronic device includes one or more processors; and a memory for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to run the programs, wherein the programs are configured to execute the aforementioned data cleaning method during runtime.

[0098] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0099] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0105] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data cleaning method, characterized in that, include: In response to the target object's operation commands to multiple operation nodes, a first flowchart is generated, wherein each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of the multiple operation nodes. Based on the first flowchart, the cleaning rules for the target data cleaning task are determined, wherein the cleaning rules include at least the operation rules of the plurality of operation nodes; Based on the cleaning rules, the source dataset corresponding to the target data cleaning task is subjected to data cleaning processing to obtain the target cleaning result, wherein the target cleaning result represents whether the data cleaning processing is successful; Specifically, based on the cleaning rules, the source dataset corresponding to the target data cleaning task is subjected to data cleaning processing to obtain the target cleaning result, including: Determine the node state of each operation node, wherein the node state is one of the following: unmodified state, modified state; From the operation rules of each operation node, the operation rules of multiple target operation nodes are obtained, and a first cleaning rule is generated, wherein the node status of the multiple target operation nodes is the modified status; Based on the first cleaning rule, the source dataset corresponding to the target data cleaning task is subjected to the data cleaning process to obtain the target cleaning result; Specifically, based on the first cleaning rule, the source dataset corresponding to the target data cleaning task is subjected to the data cleaning process to obtain the target cleaning result, including: Based on the first cleaning rule, a first execution command and a second execution command are determined from the execution commands corresponding to each target operation node, wherein the execution order of the first execution command takes precedence over the execution order of the second execution command; Execute the first execution command to perform the data cleaning process on the source dataset corresponding to the target data cleaning task, and generate the first dataset. The second execution command is executed to perform the data cleaning process on the first dataset to generate a second dataset, wherein the data standardization of the second dataset is higher than that of the first dataset; Based on the second dataset, the target cleaning result is generated; The generation of the target cleaning result based on the second dataset includes: Determine the execution status of the first operation node, wherein the execution status is one of the following: executing state, not executing state; When the execution state is the non-execution state, the execution state is updated to the executing state, and the running rules of the first operation node are obtained. The running rules of the first operation node include at least the output mode of the second dataset. The output mode is one of the following: a first output mode, a second output mode, and the amount of data corresponding to the second output mode is less than the amount of data corresponding to the first output mode. If the output mode of the second dataset is the first output mode, determine whether the data structure of the second dataset has changed; If the data structure of the second dataset changes, update the data structure of the second dataset. After updating the data structure of the second dataset, or if the data structure of the second dataset has not changed, a first data table is created, and the data of the second dataset is written from the current data table to the first data table, generating a write result, wherein the write result indicates whether the data of the second dataset was successfully written from the current data table to the first data table; Based on the writing results, the target cleaning result is generated.

2. The method according to claim 1, characterized in that, Based on the writing result, the target cleaning result is generated, including: If the write result indicates that the data of the second dataset has been successfully written from the current data table to the first data table, the names of the current data table and the first data table are swapped, and the current data table with the name of the first data table is deleted. Based on the first data table whose name is the same as the current data table, the target dataset is obtained, and a data cleaning success result is generated; If the write result indicates that the data of the second dataset was not successfully written from the current data table to the first data table, the first data table is deleted, and a data cleaning failure result is generated.

3. The method according to claim 1, characterized in that, The method further includes: When the output mode of the second dataset is the second output mode, delete the data in the target area and write the target data in the second dataset from the current data table into the target area; After writing the target data from the second dataset into the target region from the current data table, the target dataset is obtained based on the target region, and a data cleaning success result is generated.

4. The method according to claim 1, characterized in that, After generating the first flowchart in response to the target object's operation commands for multiple operation nodes, the method further includes: Calculate the lineage relationships among the multiple operation nodes to obtain a first lineage relationship set; Calculate the lineage relationships between the target fields in the plurality of operation nodes to obtain a second lineage relationship set; The first bloodline set and the second bloodline set are stored to generate the bloodline set corresponding to the target data cleaning task.

5. The method according to claim 1, characterized in that, After performing data cleaning processing on the source dataset corresponding to the target data cleaning task based on the cleaning rules to obtain the target cleaning result, the method further includes: The target cleaning result is sent to the target interface and rendered on the target interface; The target cleaning results are displayed to the target object through the target interface.

6. A data cleaning apparatus, characterized in that, include: The first processing module is used to respond to the operation commands of the target object to multiple operation nodes and generate a first flowchart, wherein each operation node is used to execute at least one operation of the target data cleaning task corresponding to the first flowchart, and the first flowchart represents the execution order of the multiple operation nodes. The determination module is used to determine the cleaning rules of the target data cleaning task according to the first flowchart, wherein the cleaning rules include at least the operation rules of the plurality of operation nodes; The second processing module is used to perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on the cleaning rules, and obtain the target cleaning result, wherein the target cleaning result indicates whether the data cleaning processing is successful. The second processing module includes: a first determining unit, used to determine the node status of each operation node, wherein the node status is one of the following: unmodified state or modified state; a first obtaining unit, used to obtain the operation rules of multiple target operation nodes from the operation rules of each operation node, and generate a first cleaning rule, wherein the node status of the multiple target operation nodes is modified state; and a first processing unit, used to perform data cleaning processing on the source dataset corresponding to the target data cleaning task based on the first cleaning rule, and obtain the target cleaning result. The first processing unit includes: a first determining submodule, used to determine a first execution command and a second execution command from the execution commands corresponding to each target operation node based on a first cleaning rule, wherein the execution order of the first execution command takes precedence over the execution order of the second execution command; a first execution submodule, used to execute the first execution command to perform data cleaning processing on the source dataset corresponding to the target data cleaning task, generating a first dataset; a second execution submodule, used to execute the second execution command to perform data cleaning processing on the first dataset, generating a second dataset, wherein the data standardization level of the second dataset is higher than that of the first dataset; and a third execution submodule, used to generate the target cleaning result based on the second dataset. The third execution submodule includes: a first determining subunit, used to determine the execution status of the first operation node, wherein the execution status is one of the following: executing state, not executing state; a first acquiring subunit, used to update the execution status to executing state when the execution status is not executing state, and to acquire the running rules of the first operation node, wherein the running rules of the first operation node include at least the output mode of the second dataset, wherein the output mode is one of the following: first output mode, second output mode, and the data volume corresponding to the second output mode is less than the data volume corresponding to the first output mode; a first judging subunit, used to judge whether the data structure of the second dataset has changed when the output mode of the second dataset is the first output mode; a first updating subunit, used to update the data structure of the second dataset when the data structure of the second dataset has changed; a first writing subunit, used to create a first data table after updating the data structure of the second dataset, or when the data structure of the second dataset has not changed, and to write the data of the second dataset from the current data table into the first data table, generating a writing result, wherein the writing result indicates whether the data of the second dataset has been successfully written from the current data table into the first data table; and a first generating subunit, used to generate the target cleaning result based on the writing result.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data cleaning method according to any one of claims 1 to 5 when it is run.

8. An electronic device, characterized in that, The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to be configured to run the programs, wherein the programs are configured to perform the data cleaning method as described in any one of claims 1 to 5 at runtime.

Images