Data cleaning methods and apparatus, electronic devices, storage media

Abstract

This invention discloses a data cleaning method, apparatus, electronic device, and storage medium, relating to the field of data processing technology or other related fields. The method includes: receiving a data cleaning request submitted by a user and parsing the request to obtain N cleaning tasks, each corresponding to a preset cleaning rule. Each cleaning task includes at least a task object field, task execution conditions, and task execution order, where N is a positive integer. For each cleaning task, a filter template is matched according to the preset cleaning rule, and filter entity code is generated by combining the filter template, task object field, and task execution conditions. According to the task execution order, the N filter entity codes are organized into a chained execution structure to generate chained execution code. The chained execution code is injected into an execution engine, and the execution engine is run to obtain the data cleaning result. This invention solves the technical problem of low efficiency in traditional data cleaning methods in related technologies.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data cleaning method and apparatus, electronic device, and storage medium. Background Technology

[0002] In the digital age, data has become a core asset for corporate decision-making. With the continuous expansion of the domestic civil aviation business, increasing business complexity, and growing user base, user demands are also gradually rising. During data processing, raw data commonly suffers from noise, missing values, and duplicate records. For example, data may generate outliers due to system interactions; manually entered information often has inconsistent formats (such as incorrect ID numbers) or is submitted repeatedly. Low-quality data directly leads to biased analysis results, wasting valuable time in the data cleaning stage.

[0003] In the context of big data, current data cleaning technologies in the civil aviation sector generally suffer from the following limitations: 1. They primarily rely on traditional manual verification and correction of data or custom scripts. Using scripts for data cleaning is inefficient and has a high error rate. 2. Due to the heterogeneous nature of data sources, such as databases, log files, and API interfaces, issues like naming conflicts and unit differences need to be addressed uniformly (e.g., inconsistent timestamp formats in flight dynamic data). Each data cleaning operation relies on a single rule and cannot support flexible combinations of multiple rules, resulting in a significant reduction in data cleaning efficiency. 3. Changes in business scenarios (e.g., the addition of passenger privacy protection rules) require dynamic adjustments to the cleaning logic. However, existing tools often require downtime for updates, making it difficult to meet real-time cleaning needs and requiring waiting in queues, indirectly leading to reduced data cleaning efficiency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data cleaning method, apparatus, electronic device, and storage medium to at least solve the technical problem of low efficiency in traditional data cleaning methods in related technologies.

[0006] According to one aspect of the present invention, a data cleaning method is provided, comprising: receiving a data cleaning request submitted by a user, parsing the data cleaning request to obtain N cleaning tasks, wherein each cleaning task corresponds to a preset cleaning rule, and the cleaning task includes at least a task object field, task execution conditions, and task execution order, where N is a positive integer; for each cleaning task, matching a filter template according to the preset cleaning rule, and generating filter entity code by combining the filter template, the task object field, and the task execution conditions; organizing the N filter entity codes into a chained execution structure according to the task execution order to generate chained execution code, wherein the output of each filter entity code in the chained execution structure serves as the input of the next filter entity code; injecting the chained execution code into an execution engine, and running the execution engine to obtain a data cleaning result.

[0007] Further, the steps of receiving data cleaning requests submitted by users and parsing the data cleaning requests to obtain N cleaning tasks include: receiving a request form submitted by the user through a configuration interface, parsing the request form to obtain cleaning task information, wherein the request form is used to record the data cleaning requests in a first data format, and the cleaning task information is used to record the data cleaning requests in a second data format; determining N cleaning tasks and the preset cleaning rule corresponding to each cleaning task from the cleaning task information based on a preset cleaning rule form, and determining the task execution order according to the cleaning task information; and determining the task object field and the task execution conditions for each cleaning task according to the configuration information of the preset cleaning rule.

[0008] Furthermore, after receiving the data cleaning request submitted by the user and parsing the data cleaning request to obtain N cleaning tasks, the process further includes: for each cleaning task, determining the target source table information that the cleaning task will act on based on the cleaning task information, wherein the target source table information includes at least: a source table field list; verifying whether each task object field of the cleaning task exists in the source table field list to obtain a field existence verification result; verifying whether the task execution conditions contain all required parameter items according to the list of required parameters indicated by the configuration information of the preset cleaning rules to obtain a parameter rationality verification result; verifying whether the execution order of all tasks is continuous and without repetition to obtain an order validity verification result; if all verification results pass, matching a filter template according to the preset cleaning rules; otherwise, stopping the data cleaning process and reporting an error to the configuration interface.

[0009] Further, for each cleaning task, the step of matching a filter template according to the preset cleaning rules and generating filter entity code by combining the filter template, the task object field, and the task execution conditions includes: for each cleaning task, determining a filter type according to the preset cleaning rules, wherein the filter type includes: filtering type, transformation type, correction type, and deduplication type; retrieving the filter template from a preset template library according to the filter type, wherein the filter template is a reusable code template pre-encapsulated for each preset cleaning rule; configuring the input / output data structure in the filter template based on the task object field, and configuring the rule parameters in the filter template based on the task execution conditions; and determining the filter entity code corresponding to each cleaning task after configuration.

[0010] Further, the step of organizing the N filter entity codes into a chain execution structure and generating chain execution code according to the task execution order includes: for the cleaning task with task execution order 1, configuring the input data of the filter entity code based on the task object field, and representing the output data as a first parameter placeholder; for the cleaning task with task execution order i, configuring the input data of the filter entity code as the (i-1)th parameter placeholder, and configuring the output data as the i-th placeholder, where 2≤i<N; for the cleaning task with task execution order N, configuring the input data of the filter entity code as the (N-1)th parameter placeholder, and obtaining the chain execution code after configuration.

[0011] Furthermore, the data cleaning method further includes: performing a logical syntax check on the chained execution code before injecting it into the execution engine to determine that the input data source of each filter entity code is not empty; otherwise, pausing the data cleaning process and triggering a data source missing alarm; during the operation of the execution engine, monitoring the execution status of each filter entity code, and pausing the data cleaning process and triggering a timed retry if the execution status fails.

[0012] Further, the step of running the execution engine to obtain the data cleaning results includes: running the execution engine, locating the task object data in the operation data storage layer according to the input data of the filter entity code with task execution order 1, and extracting the task object data; injecting the task object data into the execution engine, performing data flow in the chained execution structure, and obtaining the output data of the filter entity code with task execution order N after the flow is completed; and writing the output data into the data warehouse detail layer specified by the data cleaning requirements, wherein the data warehouse detail layer is responsible for storing all the data cleaning results.

[0013] According to another aspect of the present invention, a data cleaning apparatus is also provided, comprising: a receiving unit, configured to receive a data cleaning request submitted by a user, and parse the data cleaning request to obtain N cleaning tasks, wherein each cleaning task corresponds to a preset cleaning rule, and the cleaning task includes at least a task object field, task execution conditions, and task execution order, where N is a positive integer; a first generation unit, configured to, for each cleaning task, match a filter template according to the preset cleaning rule, and generate filter entity code by combining the filter template, the task object field, and the task execution conditions; a second generation unit, configured to, according to the task execution order, organize the N filter entity codes into a chain execution structure to generate chain execution code, wherein the output of each filter entity code in the chain execution structure serves as the input of the next filter entity code; and a running unit, configured to inject the chain execution code into an execution engine and run the execution engine to obtain data cleaning results.

[0014] Further, the receiving unit includes: a parsing module, configured to receive a user-submitted request form through a configuration interface, and parse the request form to obtain data cleaning task information, wherein the request form records the data cleaning request in a first data format, and the data cleaning task information records the data cleaning request in a second data format; a first determining module, configured to determine N cleaning tasks and the preset cleaning rule corresponding to each cleaning task from the data cleaning task information based on a preset cleaning rule form, and determine the task execution order according to the data cleaning task information; and a second determining module, configured to determine the task object field and the task execution condition of each cleaning task according to the configuration information of the preset cleaning rule.

[0015] Furthermore, the data cleaning device further includes: a third determining module, used to, after receiving a user-submitted data cleaning request and parsing the data cleaning request to obtain N cleaning tasks, determine the target source table information to be applied to each cleaning task based on the cleaning task information, wherein the target source table information includes at least: a source table field list; a first verification module, used to verify whether each task object field of the cleaning task exists in the source table field list, and obtain a field existence verification result; a second verification module, used to verify whether the task execution conditions contain all required parameter items according to the list of required parameters indicated by the configuration information of the preset cleaning rules, and obtain a parameter rationality verification result; a third verification module, used to verify whether the execution order of all tasks is continuous and without repetition, and obtain an order validity verification result; and a stopping module, used to match a filter template according to the preset cleaning rules if all verification results pass, otherwise, stop the data cleaning process and report an error to the configuration interface.

[0016] Further, the first generation unit includes: a fourth determining module, configured to determine a filter type for each cleaning task according to the preset cleaning rules, wherein the filter type includes: a filtering type, a transformation type, a correction type, and a deduplication type; a retrieving module, configured to retrieve the filter template from a preset template library according to the filter type, wherein the filter template is a reusable code template pre-encapsulated for each preset cleaning rule; a first configuration module, configured to configure the input / output data structure in the filter template based on the task object field, and to configure the rule parameters in the filter template based on the task execution conditions; and a fifth determining module, configured to determine the filter entity code corresponding to each cleaning task after configuration.

[0017] Further, the second generation unit includes: a second configuration module, configured as input data of the filter entity code based on the task object field for the cleaning task with task execution order 1, and representing the output data as a first parameter placeholder; a third configuration module, configured as the (i-1)th parameter placeholder and the output data as the i-th placeholder for the cleaning task with task execution order i, where 2≤i<N; and a fourth configuration module, configured as the (N-1)th parameter placeholder for the cleaning task with task execution order N, and obtaining the chained execution code after configuration.

[0018] Furthermore, the data cleaning device further includes: a checking module, used to perform logical syntax checks on the chained execution code before injecting the chained execution code into the execution engine, to determine that the input data source of each filter entity code is not empty, otherwise suspending the data cleaning process and triggering a data source missing alarm; and a monitoring module, used to monitor the execution status of each filter entity code during the operation of the execution engine, and to suspend the data cleaning process and trigger a timed retry if the execution status fails.

[0019] Further, the running unit includes: a running module, used to run the execution engine, locate task object data in the operation data storage layer according to the input data of the filter entity code with task execution order 1, and extract the task object data; an injection module, used to inject the task object data into the execution engine, perform data flow in the chained execution structure, and obtain the output data of the filter entity code with task execution order N after the flow is completed; and a storage module, used to write the output data into the data warehouse detail layer specified by the data cleaning requirement, wherein the data warehouse detail layer is responsible for storing all the data cleaning results.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described data cleaning methods.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data cleaning methods.

[0022] This invention proposes a data cleaning method. First, it receives data cleaning requests submitted by users and parses these requests to obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule and includes at least a task object field, task execution conditions, and task execution order. N is a positive integer. Then, for each cleaning task, a filter template is matched according to the preset cleaning rule, and filter entity code is generated by combining the filter template, task object field, and task execution conditions. Next, according to the task execution order, the N filter entity codes are organized into a chained execution structure to generate chained execution code. The output of each filter entity code in the chained execution structure serves as the input of the next filter entity code. Finally, the chained execution code is injected into the execution engine, and the execution engine is run to obtain the data cleaning result.

[0023] This invention employs a modular chain-based construction approach, using intelligent parsing and template-based generation of filter entity code to achieve efficient and automated processing of multi-stage data cleaning needs. This results in improved flexibility and traceability in the data cleaning process, eliminating the problem of rigid rules failing to adapt to dynamic business needs, and ultimately solving the technical problem of low efficiency in traditional data cleaning technologies.

[0024] Specifically, this invention receives and parses user-submitted data cleaning requests, breaking them down into multiple independent cleaning tasks that are customized according to preset cleaning rules. For each cleaning task, a corresponding filter template is automatically matched based on the preset cleaning rules. Dedicated filter entity code is generated by combining the specific object fields and execution conditions of the task, greatly reducing the user's configuration difficulty and improving code generation efficiency, while also reducing potential programming errors. Subsequently, a chain execution structure is constructed according to the task execution order to generate chain execution code, ensuring smooth data flow between filters. The output of one filter seamlessly connects to the input of the next, forming a pipeline-like data cleaning process. Finally, the chain execution code is injected into the data processing execution engine, which runs the code to achieve batch data cleaning, significantly improving the speed and quality of data cleaning. The entire process is highly automated, supporting flexible combinations of various cleaning rules, ensuring both accuracy and efficiency in data cleaning. When dealing with large-scale datasets, it effectively reduces data cleaning time consumption and improves the overall value conversion rate of business data. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0026] Figure 1 This is an architecture diagram of an optional modular chain filtering engine according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of an optional data cleaning method according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of an optional data cleaning apparatus according to an embodiment of the present invention;

[0029] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) for performing a data cleaning method according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0033] A data warehouse is a subject-oriented, integrated, relatively stable collection of data that reflects historical changes and is used to support management decisions.

[0034] Data cleaning refers to the process of detecting, identifying, and processing (such as filtering, transforming, correcting, and deduplicating) the raw data collected during data warehouse construction to ensure data quality issues meet data warehouse standards and are usable.

[0035] SQL, or Structured Query Language, is a standardized computer language used to interact with relational databases. It enables operations such as data querying, insertion, updating, deletion, and database structure management.

[0036] Hive, or Honeycomb Data Warehouse, is used to query data from large datasets using the SQL query language (HiveQL), and can also convert SQL queries into MapReduce tasks for execution on Hadoop clusters.

[0037] CTE, Common Table Expression, is a SQL feature supported by Hive used to define temporary result sets that can be referenced multiple times in SQL statements, simplifying complex query scenarios.

[0038] The Filter Pattern is a structural design pattern that allows a collection of objects to be filtered and processed step by step through a series of filtering conditions (filters) to obtain results that meet specific criteria.

[0039] ODS, Operational Data Store, is the transition layer between the data warehouse and the business system. It is mainly used to store the original data synchronized from the business system in its original form or with light cleaning, preserving the original form and details of the data.

[0040] DWD, or Data Warehouse Detail Layer, is a layer that transforms raw data into clearly structured and semantically clear detailed data by performing operations such as cleaning, de-identification, dimensional correlation, and business logic processing on the basis of ODS layer data.

[0041] The following embodiments of the present invention can be applied to various systems / applications / devices requiring multi-stage data cleaning and governance, enabling efficient data cleaning based on a modular chained filtering engine. The present invention uses an intelligent parsing module to receive and understand cleaning rules, then transforms these rules into specific executable filter entity code, better adapting to changing cleaning needs and data quality checks. Through a chained scheduling mechanism, the present invention can execute cleaning tasks in an orderly and efficient manner, ensuring smooth data transmission between filters. Simultaneously, through monitoring by the cleaning execution module, fine-grained control of the cleaning process is achieved. Finally, the result storage module completes standardized data archiving, significantly improving the automation level of data cleaning and the flexibility of data governance.

[0042] In terms of data processing, compared with traditional static cleaning rules, the core advantage of this invention lies in its dynamic chain construction capability, that is, cleaning rules can be encapsulated as independent CTE Blocks and chained together in an "input-output" manner to achieve hot-swappable and collaborative work of rules.

[0043] The present invention will now be described in detail with reference to various embodiments.

[0044] Example 1

[0045] According to an embodiment of the present invention, a data cleaning method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0046] Examples of embodiments of the present invention Figure 1 The modular chain filtering engine shown utilizes Hive CTE features combined with filter design patterns to achieve the data cleaning process. Each cleaning rule (filtering / transformation / correction / deduplication) acts as a filter, corresponding to each CTE Block query statement in Hive SQL. Data in the ODS layer table flows sequentially between filters, with the output of the previous filter serving as the input of the next filter. This process completes the cleaning rule processing for each stage, ultimately achieving multi-stage cleaning and saving the data to the DWD layer.

[0047] Examples of embodiments of the present invention Figure 2 The multi-stage data cleaning method based on a modular chained filtering engine is illustrated. The main implementation body of this method is the data cleaning and governance system. Combining the modular chained filtering engine and Hive CTE technology, it is used for civil aviation business data cleaning scenarios. In particular, it solves the problem of difficulty in coordinating and dynamically adjusting cleaning rules caused by multi-source heterogeneous data. It achieves efficient, accurate, and flexible data cleaning and standardized governance by intelligently parsing cleaning rules, constructing chained filtering logic, dynamically selecting execution engines and monitoring execution status, and strictly verifying cleaning results. Specifically, it includes rule definition and parsing, filter generation, chained scheduling and CTE splicing, cleaning execution and monitoring, and result storage and verification steps.

[0048] Figure 1 This is an architecture diagram of an optional modular chain filtering engine according to an embodiment of the present invention, such as... Figure 1 As shown, based on Hive CTE features and filter design patterns, a full-process data cleaning system of "rules → parsing → filtering → scheduling → execution → storage" is constructed. The architecture includes five core modules: cleaning rule definition and parsing module, filter module, chain scheduling module, cleaning execution module, and result storage module. Each module works together to achieve multi-stage data cleaning.

[0049] Figure 2 This is a flowchart of an optional data cleaning method according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0050] Step S201: Receive the data cleaning request submitted by the user and parse the data cleaning request to obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. The cleaning task includes at least a task object field, task execution conditions and task execution order. N is a positive integer.

[0051] It should be noted that data cleaning requirements refer to user-submitted requests for processing raw data. The purpose can be to eliminate noise, missing values, duplicate records, and other issues in the data, thereby improving data quality. In this embodiment of the invention, data cleaning requirements can be submitted by the user in JSON format, including the source of the data to be cleaned (such as the table name in the ODS layer), the set of cleaning rules to be executed, the fields involved, and the order in which the rules are executed. For example, a user might request a NOT NULL check on a certain field, a time format conversion on another field, and deduplication of the entire dataset; these specific requirements constitute the data cleaning requirements.

[0052] Upon receiving a data cleaning request, this implementation system can parse the original request into a series of specific cleaning tasks, each corresponding to a user-defined preset cleaning rule. N represents the total number of cleaning tasks, indicating that the data cleaning process can include multiple independent but logically related data processing steps. For example, the first cleaning task might be responsible for non-empty filtering, the second for date format conversion, and the third for data deduplication.

[0053] It should be further noted that the preset cleaning rules refer to the set of rules built into the implementation system that can perform specific cleaning operations on the data. In this embodiment of the invention, these rules are designed as reusable filter templates. Users only need to specify the type and parameters of the rules, and the implementation system can automatically match the corresponding templates and generate cleaning tasks. In this embodiment of the invention, the preset cleaning rules include, but are not limited to: filtering (such as non-empty checks), transformation (such as data format adjustment), correction (such as outlier handling), and deduplication.

[0054] Furthermore, the task object field refers to the specific data field targeted by the cleaning task during the data cleaning process. For example, the task object field for a non-empty filtering task might be "department name" or "submitter name." These fields are checked for emptiness to determine whether the data record is retained. Additionally, it's important to explain the task execution conditions. Each cleaning task has specific execution conditions that define when and how the task is executed. For example, for a non-empty filtering task, the execution condition requires that the field value cannot be NULL; for a date conversion task, the execution condition involves converting the date field from a string to a timestamp type, and it must follow specific formatting rules.

[0055] In the series of cleaning tasks in this invention embodiment, the execution order of the tasks determines the logical flow and efficiency of the data cleaning process. The task execution order is defined by the user when submitting the data cleaning request, and the implementation system follows this order to ensure that the cleaning rules can be executed sequentially according to the preset logic. For example, non-empty filtering is performed first, followed by date format conversion, and finally data deduplication, ensuring the continuity of data cleaning and improving data quality.

[0056] Further, the steps of receiving and parsing user-submitted data cleaning requests to obtain N cleaning tasks include: receiving a user-submitted request form through a configuration interface, parsing the request form to obtain cleaning task information, wherein the request form is used to record data cleaning requests in a first data format, and the cleaning task information is used to record data cleaning requests in a second data format; determining N cleaning tasks and the corresponding preset cleaning rules for each cleaning task from the cleaning task information based on a preset cleaning rule form, and determining the task execution order according to the cleaning task information; and determining the task object field and task execution conditions for each cleaning task according to the configuration information of the preset cleaning rules.

[0057] In this embodiment of the invention, the configuration interface serves as the interaction point between the user and the implementation system. Users submit data cleaning requests through this interface. The interface design should prioritize user experience, providing intuitive field selection, condition setting, and rule sorting functions, enabling users without a technical background to easily define cleaning rules. For example, the interface may include a drag-and-drop control for selecting the fields to be cleaned, a drop-down menu for selecting the cleaning rule type, and a sorting panel for defining the execution order of the rules.

[0058] A requirement form is a form that collects user interaction information from all functional controls on the configuration interface. It records data cleaning requirements in a primary data format (such as JSON) and may include the following information: the data source to be cleaned, a list of fields to be processed, specified cleaning rules, and the rule execution order. An example requirement form is: {"data_source":"ods_table_x","task_rules":[{"rule_type":"filter","field":"department","condition":"notnull"},{"rule_type":"convert","field":"date_field","format":"yyyy-MM-dd"},{"rule_type":"deduplicate","field":"main_key"}],"order":[1,2,3]}, where 'order' defines the rule execution sequence number.

[0059] The data cleaning task information is obtained by the system after parsing the requirement form, and is represented in a second data format (such as a structured internal data model). The second data format is easier for the system to process and store internally, and is used to transform user requirements into executable instructions, including but not limited to task ID, source data table name, target data table name, field list, cleaning rule type, rule parameters, and execution order.

[0060] The preset cleaning rule form is a collection of rules built into the system to guide the execution of data cleaning tasks. Rules are stored categorized by type, such as filtering (not null, range checks, etc.), transformation (format adjustment, type conversion, etc.), correction (outlier handling), and deduplication. Each rule has detailed configuration information, including SQL templates, parameter placeholders, and execution strategies. The configuration information for preset cleaning rules includes the rule's execution logic (SQL statement template), parameter requirements (such as field names and filtering conditions), data type conversion rules (such as string to date), and dependencies (upstream rule output as input for this rule). For example, for a date format conversion rule, the configuration information may include the conversion function (such as str_to_date), target time format (such as 'yyyy-MM-dd HH:mm:ss'), input field names, and output field names.

[0061] The above steps encapsulate the cleaning rules into independent CTE Blocks, design a modular chain filtering engine, support multi-rule collaborative work and dynamic adjustment, ultimately improve the accuracy and efficiency of data cleaning, reduce the complexity and cost of data governance, and ensure the quality and reliability of data cleaning results.

[0062] Furthermore, after receiving and parsing the user's data cleaning request to obtain N cleaning tasks, the process also includes: for each cleaning task, determining the target source table information for which the cleaning task operates based on the cleaning task information, wherein the target source table information includes at least: a list of source table fields; verifying whether each task object field of the cleaning task exists in the list of source table fields, obtaining a field existence verification result; verifying whether the task execution conditions contain all required parameter items according to the list of required parameters indicated by the configuration information of the preset cleaning rules, obtaining a parameter rationality verification result; verifying whether the execution order of all tasks is continuous and without repetition, obtaining an order validity verification result; if all verification results pass, matching the filter template according to the preset cleaning rules; otherwise, stopping the data cleaning process and reporting an error to the configuration interface.

[0063] In the data cleaning process of this invention, it is necessary not only to receive and parse the cleaning requests submitted by users, but also to ensure the legality and feasibility of the cleaning tasks. To this end, the implementation system performs a series of verification steps on each cleaning task after parsing the data cleaning requests to ensure the smooth execution of the data cleaning process.

[0064] In this embodiment of the invention, before each cleaning task begins, the source table information to be operated on by the task needs to be determined, including the source table name and a list of all fields contained therein. The target source table information is the basis for the execution of the cleaning task, ensuring that the system knows which data table and data structure to operate on. Next, the field list is obtained from the target source table for field existence verification of the task object. It should be noted that the field list should be up-to-date and reflect the true structure of the source table.

[0065] Furthermore, before executing a cleaning task, the implementation system needs to check whether the fields of each task (i.e., task object fields) actually exist in the source table's field list. If any task object field is missing, the cleaning task cannot be executed correctly, the cleaning process should be stopped immediately, and an error should be reported to the user, indicating the name of the illegal field. Additionally, each preset cleaning rule has specific configuration information, including a list of required parameters. For example, for a cleaning rule that converts a date type, the required parameters include the target field name and the date format. The implementation system needs to verify whether the user-submitted task execution conditions include all required parameters to ensure the completeness of the rule execution. Finally, the execution order of tasks affects the logical flow and efficiency of data cleaning. The implementation system also needs to check whether the execution sequence numbers of all tasks form a continuous sequence without duplicate numbers. Once all verification steps pass, the implementation system can match the corresponding filter templates according to the user-defined task execution conditions and preset cleaning rules to generate executable cleaning logic. If any verification step fails, the data cleaning process cannot continue, and specific error information needs to be provided to the user, including field non-existence, missing parameters, or incorrect execution order, so that the user can correct the error promptly.

[0066] The above steps, through a modular chained filtering engine, enable automated and intelligent multi-stage data cleaning. Specifically, by encapsulating cleaning rules into independent CTE query blocks and utilizing Hive's execution engine, cleaning scripts can be executed efficiently. Users can also flexibly select and combine different cleaning rules through a configuration interface, while the implementation system can dynamically match filter templates and support hot-swapping of rules to meet diverse data governance needs. Strict field existence checks, parameter rationality checks, and sequence validity checks ensure that each cleaning task is executed accurately. Finally, a user-friendly configuration interface is provided, allowing users to complete complex cleaning task definitions without needing in-depth understanding of SQL syntax and Hive operation details.

[0067] In one optional implementation scenario, a user submits a request to clean ODS layer table data through a configuration interface. This request includes three cleaning tasks: non-null filtering (checking the "Department Name" field), date format conversion (converting the "Estimated Completion Time" field from string format to date format), and data deduplication (based on the "Order Number" field). After parsing the user's cleaning request, the implementation system performs the following verification steps: 1. Field Existence Verification: Checks if the "Department Name," "Estimated Completion Time," and "Order Number" fields exist in the target source table's field list. If any field is missing, the data cleaning process stops, and an error message is sent to the user informing them of the invalid field's name. 2. Parameter Reasonableness Verification: Confirms whether the user-defined cleaning tasks include all required parameters. For example, in the date format conversion task, it checks whether the target field name and date format are specified. If a required parameter is missing, the cleaning process is interrupted, and the user is notified of the missing parameter. 3. Sequence Validity Verification: Checks whether the execution order of the cleaning tasks is consecutive and without repetition, ensuring that the output of the first task can correctly become the input of the second task, and so on. If all validation steps pass, the corresponding filter templates are matched according to preset cleaning rules, such as the "Non-empty Field Filter" template, the "Date Format Converter" template, and the "Data Deduplication" template, to generate specific cleaning logic. Subsequently, the cleaning tasks are executed in a chained sequence, using Hive CTE features to ensure accurate data transfer and processing between each cleaning step. If any validation fails, the cleaning process is not initiated; instead, specific error messages are displayed to the user, allowing them to correct configuration errors promptly.

[0068] Another alternative implementation, such as Figure 1 In the modular chained filtering engine shown, the cleaning rule definition and parsing module is the entry point for the data cleaning process. It is responsible for representing the cleaning rules configured in the user interface in JSON format and converting them into executable structured instructions. It is the key hub connecting business rules and technical implementation. Core functions include rule parsing, validity verification, filter matching, and metadata generation. The specific process is as follows:

[0069] 1. JSON rule parsing: The module receives JSON-formatted rule configurations submitted by the user, as shown in the example below:

[0070] { "job_name":"Requirement Work Order Table Data Cleaning Task",

[0071] "table_name":"ods_req_order",

[0072] "columns":

[0073] ["order_id","order_name","order_no","order_type","order_content","order_status","year","dept_name","expect_complete_time","submit_user_name","submit_time","del_flag"],

[0074] "rules": [

[0075] {"rule_type": "filter","column": "dept_name","rule_name": "column_not_null_filter", "index": 0},

[0076] {"rule_type": "converter","column": "expect_complete_time","rule_name": "column_datetime_format_converter","index": 1},

[0077] {"rule_type": "duplicate","rule_name": "table_duplicate","index": 2}]

[0078] }

[0079] The structure is parsed using a JSON Schema validator to extract key information: basic information: job name (job_name), source table name (table_name), and list of source table fields (columns); rule set: the rules array is traversed to extract the type (rule_type), name (rule_name), target field (column), and execution order (index) of each rule.

[0080] To ensure rule execution, the module performs multiple layers of validity checks: Field existence check: checks if the field specified in the rule's column exists in the columns list (e.g., "dept_name" must be in the columns array in the example); Rule integrity check: verifies if each rule contains required parameters (e.g., filter and converter rules must specify a column, duplicate type rules can omit the column, indicating full table deduplication); Order validity check: checks if the indexes are consecutive and without duplicates (e.g., if the indexes are 0, 1, and 2 in the example, ensures no execution order conflicts). Specific error information is returned when checks fail.

[0081] In addition, the module has a built-in "cleaning rule-filter" mapping table, which matches the corresponding filter template based on the combination of rule_type and rule_name. The mapping logic is as follows: rule_type determines the filter category (e.g., "filter" corresponds to the filter class), and rule_name determines the specific implementation (e.g., "column_not_null_filter" corresponds to the non-null filter template).

[0082] Example match:

[0083] a) Rule 1 (rule_type="filter", rule_name="column_not_null_filter") -> matches "non-null filter";

[0084] b) Rule 2 (rule_type="converter", rule_name="column_datetime_format_converter") -> Matches "date format conversion filter";

[0085] c) Rule 3 (rule_type="duplicate", rule_name="table_duplicate") -> matches "full table deduplication filter".

[0086] After parsing and validation, the module generates standardized data for each rule, including: input / output field mappings (e.g., for date conversion rules, the input is `expect_complete_time` (string), and the output is a field with the same name (timestamp type)); rule parameter completion (e.g., for date conversion rules, the auto-completion default format is `yyyy-MM-dd HH:mm:ss`); and execution dependency tags (e.g., deduplication rules depend on the output of all preceding rules by default). Metadata is passed to the filter module in JSON format as input parameters for filter instantiation.

[0087] The above modules decouple business rules from technical implementation. Users do not need to understand HiveSQL syntax; they can complete the definition of cleaning rules simply by configuring JSON, which greatly reduces the threshold for data governance.

[0088] Step S202: For each cleaning task, match the filter template according to the preset cleaning rules, and generate filter entity code by combining the filter template, task object fields and task execution conditions.

[0089] It's important to note that the process of matching filter templates involves translating user-defined data cleaning requirements into concrete technical implementations. This involves searching for matching filter templates based on preset cleaning rules. For example, if a user-defined cleaning task requires filtering out all records where the "department name" field is empty, a template matching this non-empty check requirement can be found in the preset filter template library. This template typically contains a generic SQL snippet, which can be parameterized (e.g., field names, filter conditions) to adapt to the specific requirements of different cleaning tasks.

[0090] It's important to note that filter templates are pre-designed, generic code templates containing specific data cleaning logic. They can be parameterized to adapt to different data sources and scenarios, supporting the rapid generation of execution code for corresponding cleaning rules. For example, the "Date Format Conversion" filter template might contain SQL statement snippets to convert a string field into a date type field with a specified date format. The template will reserve parameter positions for inserting specific field names and date formats. When generating the filter entity code, the "Task Object Fields" and "Task Execution Conditions" are injected into the template as parameters. The Task Object Fields are the specific fields in the data table targeted by the cleaning task, and the Task Execution Conditions are the cleaning logic applied to these fields, such as filtering null values, converting strings to dates, and removing duplicate records.

[0091] Based on the filter template, combined with specific data table field names and cleaning conditions, SQL or program code that can be used for actual data cleaning is generated. For example, if the filter template is used to remove duplicate records, when the system receives a task to "deduplicate based on the order number field," it will insert the "order number" field name and deduplication logic into the template, generating a specific SQL statement (such as SELECT DISTINCT). The function `FROM source_table WHERE id is not null GROUP BYorder_number` is used to perform deduplication.

[0092] Furthermore, for each cleaning task, the steps of matching a filter template according to preset cleaning rules and generating filter entity code by combining the filter template, task object fields, and task execution conditions include: for each cleaning task, determining the filter type according to preset cleaning rules, where the filter type includes: filtering type, transformation type, correction type, and deduplication type; retrieving the filter template from the preset template library according to the filter type, where the filter template is a reusable code template pre-encapsulated for each preset cleaning rule; configuring the input / output data structure in the filter template based on the task object fields, and configuring the rule parameters in the filter template based on the task execution conditions; and determining the filter entity code corresponding to each cleaning task after configuration.

[0093] It's important to note that filtering filters focus on removing records from a dataset that do not meet specific conditions. For example, they remove rows where a field is empty or outside a predefined range. By performing operations similar to the WHERE clause in SQL, filters can effectively improve the quality and usability of a dataset. Transformation filters are responsible for converting field values ​​in a dataset from one format to another, such as converting text dates to a standard date and time format. Correction filters aim to correct outliers or format errors in the data, such as replacing "unknown" or other non-standard values ​​with NULL or specific default values. Deduplication filters are used to identify and remove duplicate records from a dataset, ensuring that each data record is unique. Deduplication operations are typically based on the values ​​of one or more fields, which can significantly reduce data redundancy and improve storage efficiency and query performance.

[0094] In existing technologies, data cleaning is typically hard-coded, requiring individual manual SQL writing or script customization for each cleaning task, lacking flexibility and reusability. In contrast, this invention proposes an encapsulated filter design that allows users to define cleaning tasks through parameterization, without needing to concern themselves with the underlying SQL or scripting language details, significantly lowering the learning and operational threshold for users. The encapsulated filter also supports modular hot-swapping, allowing users to easily add, delete, or adjust cleaning rules without modifying existing cleaning processes or rewriting code, which is particularly beneficial in scenarios where data rules change frequently.

[0095] Filter templates are predefined code snippets containing the basic logic of a data cleaning task, but with parameter placeholders to adapt to specific task requirements. For example, a filter type template might contain a WHERE clause where field names and conditions are placeholders. Configuring the input / output data structures involves explicitly specifying the filter's input data source (field names, data types) and the cleaned output data structure, ensuring the filtering logic is correctly applied to the target fields and that the output conforms to data warehouse standards. During the generation of the filter entity code, the parameter placeholders in the template are populated according to the specific conditions of the user-defined data cleaning task. For example, if a user-defined transformation type task requires converting the "estimated completion time" field to a timestamp type, the system will find the transformation type template, using the "estimated completion time" field name and timestamp type as parameters, to generate the specific SQL statement. After completing the selection of the filter template, parameter population, and data structure configuration, a complete, executable SQL code snippet, called the filter entity code, is generated. These code snippets can be chained together to form a complete chained data cleaning process.

[0096] The chain-style data cleaning process transforms, corrects, and deduplicates raw data, ultimately generating cleaned results that meet the specifications and requirements of a data warehouse.

[0097] The above steps utilize a modular chained filtering engine to combine and dynamically adjust multiple rules, meeting the cleaning needs of different types of data sources. Furthermore, visual configuration and automated code generation reduce the difficulty of user operation and improve data governance efficiency. Compared to traditional methods, this invention can handle more complex combinations of cleaning rules, supports hot-swapping of rules, and has significant advantages in execution efficiency and resource utilization, making it an important optimization solution for the data cleaning stage in modern data warehouse construction.

[0098] Another alternative implementation, such as Figure 1 In the modular chain-like filtering engine shown, the filter module is the core of rule encapsulation, standardizing various cleaning requirements (filtration, conversion, correction, deduplication) into "reusable filters." Each filter contains three main metadata elements:

[0099] Input / output schema: Defines the input data structure (such as ODS table field names and types) and output data structure (such as fields to be retained after filtering and new fields after transformation) of the filter to ensure data compatibility between filters;

[0100] Cleaning rule parameters: Clearly define the core configuration of the rule. For example, filter-type filters need to specify "filter field + condition" (e.g., dept_name is not null), transformation-type filters need to specify "target field + transformation function" (e.g., concat('Name:', submit_user_name)), and deduplication filters need to specify "deduplication base field" (e.g., order_id).

[0101] SQL templates: Predefined templates adapted to Hive syntax, using parameter placeholders (such as...). {prev_cte_name} represents the output of the previous filter. Dynamic generation is achieved, as shown in the example below:

[0102] Filter templates: {cte_name}AS(SELECT FROM {prev_cte_name}WHERE {filter_condition});

[0103] Conversion template: {cte_name}AS(SELECT , {convert_function}AS {target_field} FROM {prev_cte_name});

[0104] De-weighting template: {cte_name}AS(SELECTDISTINCT FROM {prev_cte_name}PARTITIONBY {dedup_field}).

[0105] Step S203: According to the task execution order, organize the N filter entity codes into a chain execution structure to generate chain execution code, wherein the output of each filter entity code in the chain execution structure is used as the input of the next filter entity code.

[0106] It's important to note that a chained execution structure is a data processing flow built according to the order in which tasks are executed. In this structure, each filter entity code is considered a link in the chain, and its output directly serves as the input data for the next link (i.e., the next filter entity code). For example, if a user defines four consecutive cleaning tasks, the chained execution structure would look like this: Filter A (NOT check) → Filter B (data type conversion) → Filter C (outlier correction) → Filter D (duplicate record removal).

[0107] Chained execution code is a composite code formed by concatenating various filter entity codes, which can be executed in a predetermined order. In the Hive CTE environment, chained execution code manifests as a series of nested SQL queries, where each CTE block represents a filter entity code, and the output of the previous CTE block is the input of the next CTE block. For example:

[0108] with filtered_data as (select from ods_data where department_nameis not null),

[0109] converted_data as (select from filtered_data where str_to_date(expected_completion_time, 'yyyy-MM-dd HH:mm') is not null),

[0110] corrected_data as (select from converted_data where case whenerror_code = 'ERR' then 'ERROR' else error_code end as corrected_error_code),

[0111] deduplicated_data as (select from corrected_data group by order_id)

[0112] select from deduplicated_data;

[0113] In the code example above, filtered_data, converted_data, corrected_data, and deduplicated_data correspond to the outputs of the four filter entity codes, forming a chain execution structure. Finally, the cleaned data is output to the DWD layer through the last CTE block.

[0114] This invention achieves flexible combination and efficient execution of data cleaning rules by constructing a chained execution structure. Users can freely define the order of multiple cleaning tasks, and the system will automatically construct the chained execution structure of filters according to this order. This ensures that each cleaning step can accurately process the output of the previous step, avoiding confusion and duplication of work in the data processing process, and improving the overall efficiency and quality of data governance.

[0115] Another alternative implementation, such as Figure 1 In the modular chained filtering engine shown, the chained scheduling module is responsible for managing the execution order of filters and resolving dependencies. The core logic includes: topology sorting: based on the dependency relationship of the filter input and output schema (such as "date format conversion" depending on the output of "department non-empty filter"), automatically generating acyclic execution sequences (filtering, conversion, correction, deduplication) to avoid logical conflicts.

[0116] Furthermore, according to the task execution order, the steps of organizing N filter entity codes into a chain execution structure and generating chain execution code include: for the cleaning task with task execution order 1, configuring the input data of the filter entity code based on the task object field, and representing the output data as the first parameter placeholder; for the cleaning task with task execution order i, configuring the input data of the filter entity code as the (i-1)th parameter placeholder, and configuring the output data as the i-th placeholder, where 2≤i<N; for the cleaning task with task execution order N, configuring the input data of the filter entity code as the (N-1)th parameter placeholder, and obtaining the chain execution code after configuration.

[0117] In one optional embodiment, the data cleaning process includes three cleaning tasks (N=3), then:

[0118] Task 1 (Filtering Type): The first filtering task is to remove all null values ​​from the dataset, specifically targeting the "Department Name" field. The system generates filter entity code for this task, taking the original data source (such as data from the ODS layer) as input and outputting cte_filtered_departments (first parameter placeholder).

[0119] Task 2 (Conversion Type): The next task is to perform date format conversion on the "Date Field," converting the dates in the data from various formats to the 'YYYY-MM-DD' format. The input to this task is the output of the previous task, cte_filtered_departments (placeholder for the (i-1)th parameter), and the output is represented as cte_converted_dates (placeholder for the i-th parameter).

[0120] Task 3 (Duplicate Removal Type): The final task is to remove duplicate records from the dataset based on the "unique identifier" field. The input data for this task will be the output of Task 2, cte_converted_dates (the (N-1)th parameter placeholder). The output is the cleaned data, which is the end point of the chained execution process and does not need to be used as input for subsequent tasks.

[0121] By linking these three filter entity codes in sequence, the following chained execution code can be generated:

[0122] with cte_filtered_departments as (select from ods_data wheredepartment is not null),

[0123] cte_converted_dates as (select , to_date(date_field, 'YYYY-MM-DD')as date_standardized from cte_filtered_departments),

[0124] cte_deduplicated as (select from cte_converted_dates group byunique_identifier)

[0125] select from cte_deduplicated;

[0126] In the code example above, cte_filtered_departments serves as the first placeholder and is the input to task 2; cte_converted_dates serves as the second placeholder and is the input to task 3; while cte_deduplicated serves as the final output of the chained execution, completing the chained structure and requiring no input to any subsequent task.

[0127] The above steps achieve multi-step, precise cleaning of the dataset through automated generation of chained execution code. This covers the entire process from user requirement analysis to automated execution of cleaning tasks, ensuring that the data meets the expected quality standards before entering the data warehouse. The chained execution code design allows each cleaning task to seamlessly connect to the previous task, and to perform the next cleaning operation based on the output of the previous task. This effectively links all cleaning tasks, forming a compact and logically clear data cleaning process.

[0128] Furthermore, the data cleaning method also includes: performing logical syntax checks on the chained execution code before injecting it into the execution engine to ensure that the input data source of each filter entity code is not empty; otherwise, pausing the data cleaning process and triggering a data source missing alarm; during the execution engine operation, monitoring the execution status of each filter entity code, and pausing the data cleaning process and triggering a timed retry if the execution status fails.

[0129] It's important to note that before injecting chained execution code into an execution engine (such as Hive), the generated SQL statements must be rigorously checked to ensure logical and syntactic correctness and prevent potential execution failures. Syntax checks include, but are not limited to, verifying that the SQL syntax conforms to standards, that field names are spelled correctly, and that function calls are appropriate. Identifying and correcting these issues early on can prevent wasting resources and time during actual execution.

[0130] Ensuring that the input data source for each filter entity is not empty is crucial to guaranteeing valid data input at every step of the data cleaning process. This is essential for maintaining the integrity and effectiveness of the chained execution structure. If the input data source for a filter is empty, the result is meaningless, subsequent cleaning tasks will fail, and the entire data cleaning process will be interrupted.

[0131] When an empty input data source is detected for a filter, the current data cleaning process is stopped, and the problem is reported to the system administrator or user through logging, email notifications, or interface alerts. The alert information includes the identifier and location of the missing data source, as well as possible cause analysis, such as upstream task execution failure, abnormal data transmission, or the data source itself being empty, to facilitate quick problem location and data recovery measures.

[0132] During the execution of the chained code by the execution engine, the system monitors the execution status of each filter entity in real time, including whether there are any task exceptions, data loss, or whether processing has been completed. If a task fails (e.g., SQL syntax errors, data format mismatches, or insufficient resource allocation), the entire cleaning process is paused and a timed retry mechanism is initiated to re-execute the failed task. The number of retries and the interval can be dynamically adjusted according to system configuration or task nature to balance the timeliness and efficiency of error handling.

[0133] Another alternative implementation, such as Figure 1 In the modular chained filtering engine shown, the cleaning execution module is responsible for monitoring the submission and execution of CTE statements, and its functions include:

[0134] Syntax validation: Call the Hive Metastore interface to validate the syntax of the CTE statement (such as field existence and function compatibility) to avoid execution failure due to syntax errors;

[0135] Resource scheduling: Hive execution resources are dynamically allocated based on the amount of data in the source table. For example, when the data volume is >100GB, the number of Reduce tasks is automatically increased (set mapreduce.job.reduces=100), and the Hive Tez execution engine is enabled to improve efficiency (data volume query example: SELECT TBL_NAME, SD_LOCATION, INPUT_FILE_NAME, TBL_SIZE FROM TBLSWHERE DB_NAME = 'your_database_name' AND TBL_NAME = 'your_table_name').

[0136] Exception handling: Capture exceptions during execution (such as SQL syntax errors, data skew, timeouts), record exception logs (including error location and data sample), and support automatic retries (default 3 times, 5-minute intervals) or manual intervention.

[0137] Step S204: Inject the chained execution code into the execution engine and run the execution engine to obtain the data cleaning results.

[0138] It's important to note that the generated chained execution code (a series of CTE statements) is passed to the data processing engine to execute the pre-built data cleaning process. This process can be divided into several key sub-steps: 1. Injecting into the execution engine: The chained execution code is input through the execution engine's interface or submission mechanism. For example, using Hive's SQL statement submission function, the entire chained execution code is submitted as a parameter to the Hive Server. 2. Running the execution engine: After receiving the chained execution code, the execution engine begins parsing and executing the corresponding SQL statements. Each CTE query block is treated as an independent execution unit, but the dependencies (input-output) between them ensure that the data flows in the order of the cleaning rules. 3. Monitoring and resource management: During execution, the progress and status are continuously monitored, and resource allocation is dynamically adjusted according to the data volume and task complexity, such as increasing the number of reduce tasks or enabling more efficient data processing modes (such as Tez), to ensure that the data cleaning task is completed efficiently and stably. 4. Result Acquisition and Storage: After the execution engine completes all cleaning tasks, it returns the cleaning results. The system writes the result data into the DWD layer (Data Detail Layer) of the data warehouse. The process includes data verification, partitioned storage, result recording, and other steps to ensure that the final result of data cleaning meets expectations and data warehouse standards.

[0139] Furthermore, the steps for obtaining data cleaning results by running the execution engine include: running the execution engine, locating the task object data in the operation data storage layer based on the input data of the filter entity code with task execution order 1, and extracting the task object data; injecting the task object data into the execution engine, performing data flow in the chained execution structure, and obtaining the output data of the filter entity code with task execution order N after the flow is completed; and writing the output data into the data warehouse detail layer specified by the data cleaning requirements, wherein the data warehouse detail layer is responsible for storing all data cleaning results.

[0140] It's important to note that in the data cleaning process, the operational data storage layer plays a crucial role in collecting and initially organizing raw data. As the first layer of the data warehouse, it directly stores raw data from various data sources (such as business systems, external APIs, log files, etc.). Data in the ODS layer is typically unwashed and untransformed, retaining its most original form and representing data generated by business activities that are quickly and accurately captured. To ensure data timeliness and integrity, the ODS layer may not perform deep data cleaning or aggregation operations, instead preserving the raw data state to provide a foundational data source for subsequent data cleaning and analysis.

[0141] The data warehouse detail layer (DWD) is the second layer in the data warehouse architecture, following the ODS layer. Its main task is to perform deep cleaning and preliminary transformation of the data in the ODS layer, converting the raw data into data with a clear structure and consistent format, while maintaining the atomicity and integrity of the data at the detail level. The data in the DWD layer has typically had noise removed, format errors corrected, and necessary data integration operations completed, preparing high-quality foundational data for subsequent data aggregation, analysis, and reporting. The DWD layer is where "fine-grained fact data" is stored in the data warehouse, ensuring data consistency and accuracy, and is a key intermediate layer in building the data warehouse.

[0142] In this embodiment of the invention, the user-defined source data location is first located in the Operational Data Storage Layer (ODS layer) according to the input data requirements of the filter entity code with task execution order 1, and then the relevant task object data is extracted. For example, if the first cleaning task is to filter all records with empty fields, then the original dataset containing all fields is extracted from the ODS layer table as input. The extracted task object data is then injected into the execution engine, which is responsible for processing this data and executing subsequent cleaning tasks. The data injection process involves steps such as setting up the execution environment, configuring resources (such as CPU and memory), and defining data access permissions to ensure that the execution engine can successfully execute the chained execution code.

[0143] Within the execution engine, data flow follows a chained execution structure pre-built according to the task execution order. Specifically: the output data of the first filter entity code (i.e., the CTE result) is passed to the second filter entity code as input, ensuring that data cleaning proceeds step by step from the source; subsequently, the output of each filter entity code automatically becomes the input of the next filter entity code, forming a closed-loop data flow mechanism. For example, data after format conversion automatically flows into the deduplication filter for deduplication processing; the last filter entity code in the task execution order takes the processed data as the final output and no longer serves as the input for any other task. At this point, the data has been processed by all cleaning rules and has achieved the predetermined cleaning goal.

[0144] Another alternative implementation, such as Figure 1 In the modular chained filtering engine shown, the result storage module is responsible for writing the cleaned data into the DWD layer. The core mechanism is as follows:

[0145] Partitioned storage: Partition the DWD table by business dimension (such as time dt, business line biz_line). Example: INSERT OVERWRITE TABLE dwd_req_order PARTITION (dt='20250101') SELECT FROM the last CTE name;

[0146] Consistency check: Perform source-destination data verification, including record count comparison (number of records after cleaning ≤ number of records in the source table) and key field verification (such as the department field dept_name having a null value rate of 0%).

[0147] Data synchronization: The cleaning task information (execution time, processing volume, filter sequence, etc.) is synchronized to the data governance platform, supporting full-chain traceability.

[0148] Through steps S201 to S204 above, the user's submitted data cleaning request can be received first, and the data cleaning request can be parsed to obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. The cleaning task includes at least a task object field, task execution conditions, and task execution order. N is a positive integer. Then, for each cleaning task, a filter template is matched according to the preset cleaning rule, and filter entity code is generated by combining the filter template, task object field, and task execution conditions. Then, according to the task execution order, the N filter entity codes are organized into a chain execution structure to generate chain execution code. The output of each filter entity code in the chain execution structure is used as the input of the next filter entity code. Finally, the chain execution code is injected into the execution engine, and the execution engine is run to obtain the data cleaning result.

[0149] In this embodiment of the invention, a modular chain construction approach is adopted. By intelligently parsing and generating filter entity code through templates, the goal of efficiently and automatically processing multi-stage data cleaning needs is achieved. This realizes the technical effects of flexibility and traceability in the data cleaning process, resolves the problem that rigid rules are difficult to adapt to dynamic business needs, and solves the technical problem of low efficiency in traditional data cleaning in related technologies.

[0150] Specifically, this invention receives and parses user-submitted data cleaning requests, breaking them down into multiple independent cleaning tasks that are customized according to preset cleaning rules. For each cleaning task, a corresponding filter template is automatically matched according to the preset cleaning rules, and specialized filter entity code is generated by combining the specific object fields and execution conditions of the task. This greatly reduces the configuration difficulty for users, while also improving code generation efficiency and reducing potential programming errors. Subsequently, a chain execution structure is constructed according to the task execution order to generate chain execution code, ensuring smooth data flow between filters. The output of one filter seamlessly connects to the input of the next filter, forming a pipeline-like data cleaning process. Finally, the chain execution code is injected into the data processing execution engine, which runs the code to achieve batch data cleaning, significantly improving the speed and quality of data cleaning. The entire process is highly automated, supports flexible combinations of various cleaning rules, ensuring both the accuracy and efficiency of data cleaning. When dealing with large-scale datasets, it can effectively reduce the time consumption of data cleaning and improve the overall value conversion rate of business data.

[0151] The invention will now be described in conjunction with another alternative embodiment.

[0152] The implementation method of this invention takes "data cleaning of demand work order table" as a typical scenario, combines Hive data warehouse architecture and modular design concept, decomposes the implementation logic into modules, and demonstrates and verifies the effect through complete process.

[0153] For the data warehouse cleansing scenario from ODS (Operational Data Storage Layer) to DWD (Data Warehouse Detail Layer), the example data table structure is as follows: ODS Layer Table (Source Table): Table name ods_req_order (requirement order table), stores the original work order data, and the fields and types are shown in Table 1 below:

[0154] Table 1

[0155]

[0156] DWD Layer Table (Target Table): Table name dwd_req_order, stores cleaned data, fields are the same as ODS Layer Table, but must meet the following requirements: dept_name is not null, expect_complete_time format is uniformly yyyy-MM-dd, no duplicate data, storage format is ORC (optimized row and column storage), compression method is SNAPPY (balancing compression ratio and query speed), partitioning strategy is the same as ODS Layer (partitioned by dt), bucketing field is order_id (10 buckets, improving query efficiency).

[0157] The core modules (cleaning rule definition and parsing module, filter module, chained scheduling module, cleaning execution module, and result storage module) need to work together. The functional positioning, inputs and outputs, technical implementation, and key processes of each module are as follows:

[0158] 1. Cleaning rule definition and parsing module: This module is responsible for receiving the cleaning rules (JSON format) configured by the user, completing the syntax verification, semantic parsing and structure transformation of the rules, providing standardized input for subsequent filter generation, and solving the maintenance difficulties caused by "hard-coded rules" in traditional cleaning.

[0159] Input: JSON format cleaning rules submitted by users through the web configuration interface or API (refer to the "Requirement Work Order Table Data Cleaning Task" rules);

[0160] Output: A structured list of RuleEntity objects (containing metadata such as rule type, associated fields, and execution order), and rule validation results (if errors exist, the error type and location must be specified).

[0161] Rule JSON structure definition: Refer to the rule examples in the technical solution to ensure the completeness and scalability of the rules. The rule parsing process adopts a three-step process of "syntax validation -> semantic validation -> structure transformation" to ensure the rules are legal and valid.

[0162] Step 1: Legality verification and rule verification.

[0163] Use JSON Schema to validate the correctness of the JSON format of the rules. For example, rule_type can only be filter / converter / correcter / duplicate, and index must be a non-negative integer. If the validation fails, return an error code (such as 1001) and a message (such as "The rule_type field value of RULE_002 is incorrect. The allowed values ​​are filter / converter / correcter / duplicate").

[0164] Validation rules must match the table structure: ① Fields in columns must exist in the source table of the ODS layer (by querying the table structure through Hive Metastore); ② rule_param must match rule_name (e.g., column_datetime_format_converter must include the target_format parameter); ③ Indexes must not have duplicate values ​​(to avoid execution order conflicts).

[0165] Step 2: Structure transformation.

[0166] Convert valid JSON rules into Java objects RuleEntity, with the following core properties:

[0167] public class RuleEntity {

[0168] private String jobId; / / Task ID

[0169] private String tableName; / / source table name

[0170] private String targetTable; / / Target table name

[0171] private List <string>columns; / / List of fields involved

[0172] private String partitionFilter; / / Partition filtering conditions

[0173] private String ruleId; / / Rule ID

[0174] private String column; / / Related field

[0175] private String ruleType; / / Rule type

[0176] private String ruleName; / / Rule name

[0177] private Map<String, String> ruleParam; / / Rule parameters

[0178] private int index; / / Execution order

[0179] }

[0180] 2. Filter Module: This module is the core execution unit of this invention. It is responsible for converting the parsed RuleEntity into "executable filter instances". Each filter corresponds to a cleaning rule (such as non-empty filtering, date conversion) and is encapsulated as a "Block fragment" of Hive CTE syntax, providing a basic unit for chained scheduling.

[0181] Based on the rule_type, filters are divided into four categories. Each type of filter must implement the Filter interface (defining the generateCTEBlock() method to generate CTE fragments), as shown in Table 2 below:

[0182] Table 2

[0183]

[0184] Filter reuse mechanism: To reduce repetitive development costs, the module is designed with a "filter template library," which encapsulates common cleaning rules into reusable templates, such as templates for "filtering fields that are not null," "converting string fields to date format," and "removing duplicates by primary key." Users only need to pass in parameters such as "target field name" and "condition value" to quickly generate filter instances without having to repeatedly define rule logic. For example, when calling the "filter not null" template, passing in the parameter "target_field=submit_user_id" will automatically generate the CTE logic "WHERE submit_user_id is not null."

[0185] 3. Chained Scheduling Module: This module is the system's "flow control hub," responsible for organizing multiple filters into a "chained execution sequence" according to their topology. It ensures that the output of one filter becomes the input of the next, fundamentally solving the problem of "how to schedule filter execution in logical order," while also supporting fault handling. After obtaining all filter instances to be executed from the filter module, the execution sequence is constructed through two steps: "dependency analysis -> priority sorting -> chained assembly."

[0186] Priority sorting: Provided that the dependencies are satisfied, filters are sorted by priority (levels 1-10), with higher priority filters being executed first (e.g., "filter invalid data" has a higher priority than "format conversion" to avoid invalid data consuming conversion resources); if the priorities are the same, they are sorted in ascending order by FilterId.

[0187] Linked assembly: A doubly linked list data structure is used to store the linked sequence. Each linked list node contains a "current filter instance", a "prev reference", and a "next reference". For example, when the sequence is "filtering filter -> transformation filter -> deduplication filter", the linked list structure is: F1 (filtering).Next = F2 (transformation), F2.Prev = F1, F2.Next = F3 (deduplication), F3.Prev = F2;

[0188] At the same time, a "chained scheduling list" is generated, which includes information such as node order, FilterId, input and output CTE names, and execution dependencies, and is pushed to the cleaning execution module.

[0189] To improve system stability, the module is designed with a "two-level fault handling mechanism":

[0190] Pre-execution verification: Before scheduling, check whether the input data source of each filter exists (such as whether the preceding filter has generated a valid CTE). If it does not exist, the scheduling is paused and a "data source missing" alarm is triggered.

[0191] Execution monitoring: Receives real-time feedback from the cleaning execution module on the filter execution status (success / failure). If a filter fails to execute (e.g., SQL syntax error, data volume is 0), a "failure log" (including FilterId, error reason, and execution time) is recorded in the filter_error_log table, and a retry is triggered (default 3 retries, 5-second interval; the number of retries is configurable).

[0192] 4. Cleaning Execution Module: This module is responsible for submitting the complete Hive CTE statements generated by the chained scheduling module to the execution engine (Hive or Tez), and monitoring the task execution process (such as progress, time consumption, and errors) to ensure the stable operation of the cleaning task. It dynamically selects the execution engine based on the data volume, balancing efficiency and resource consumption.

[0193] Small data volume (≤1GB): Select Hive local mode, no need to apply for YARN resources;

[0194] Medium data volume (1GB~10GB): Select HiveYARN mode and request default resources (20 map nodes, 10 reduce nodes).

[0195] For large data volumes (≥10GB): Select Hive Tez mode (leveraging Tez's in-memory computing advantages) and request resources.

[0196] Taking Hive YARN mode as an example, the task submission and monitoring process is as follows:

[0197] A. Hive Connection Establishment: Connect to HiveServer2 via Hive JDBC (URL format: jdbc:hive2: / / hive-server2:10000 / default;auth=noSasl), log in using a cluster user (such as hive), and ensure that you have read permissions for the source table and write permissions for the target table.

[0198] B. Task Parameter Configuration: Before submitting the CTE statement, set Hive optimization parameters to improve execution efficiency:

[0199] SET hive.exec.dynamic.partition.mode=nonstrict; -- Supports dynamic partitioning;

[0200] SET hive.exec.compress.output=true; -- Output compression;

[0201] SET hive.exec.orc.default.compress=SNAPPY; -- Compress ORC files using SNAPPY;

[0202] SET hive.auto.convert.join=true; -- Automatically convert the join type (optimizes joins on small tables);

[0203] C. Task Submission and Status Monitoring: Execute CTE statements using a Statement object, obtain the task execution status through a ResultSet, and periodically (every 10 seconds) query the YARN ResourceManager API (http: / / resourcemanager:8088 / ws / v1 / cluster / apps / {appId}) to obtain task progress (such as map completion rate and reduce completion rate). If the task execution times out (default 120 minutes), the task is actively terminated and an error log is recorded.

[0204] D. Execution Result Feedback: After the task is completed, an execution report is returned.

[0205] Success: Includes the amount of data after cleaning (e.g., "Cleaning completed, 8GB processed, 4.75 million rows output, cleaning rate 5%) and the time taken (e.g., "Total time taken 25 minutes").

[0206] Failure: Includes the error type (such as SQL syntax error, insufficient YARN resources), error stack information (to facilitate problem localization), and triggers the retry mechanism (3 retries by default, with an interval of 5 minutes between each retry).

[0207] 5. Result Storage Module: This module is responsible for writing the output data from the cleaning execution module into the target table of the DWD layer, and completing data verification, partition management, and metadata synchronization to ensure that the cleaned data conforms to data warehouse specifications. Core storage logic:

[0208] DWD table initialization (first execution): If the target table of the DWD layer does not exist, the module will automatically create the target table according to the source table structure of the ODS layer.

[0209] Data writing and partition management: Use the INSERT OVERWRITE TABLE...PARTITION syntax to write data (avoid overwriting historical partitions). Before writing, check if the target partition already exists.

[0210] If the partition does not exist: create the partition directly and write the data;

[0211] If the partition already exists: overwrite the existing data (supports incremental cleaning scenarios, such as rerunning the partition data from the previous day).

[0212] Data validation: After writing is complete, execute the following validation logic to ensure data quality.

[0213] Row count verification: Compare the row count of the target partition in the DWD layer with the row count output by the cleaning execution module. If the difference rate is >1% (configurable), mark the data as abnormal and trigger an alarm.

[0214] Field validation: Check the integrity of key fields (such as dept_name being non-empty, expect_complete_time being in the format yyyy-MM-dd).

[0215] Consistency check: Compare the sum of key fields (such as order_no work order number) between the ODS layer and the DWD layer. If the difference rate is >0.1%, check whether there are logical errors in the cleaning rules.

[0216] Data synchronization: After data verification is passed, the data is synchronized to the data governance platform's cleansing result table, and the recorded information is shown in Table 3 below:

[0217] Table 3

[0218]

[0219] The invention will now be described in conjunction with another alternative embodiment.

[0220] Example 2

[0221] The data cleaning apparatus provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0222] Figure 3 This is a schematic diagram of an optional data cleaning apparatus according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: a receiving unit 31, a first generating unit 32, a second generating unit 33, and a running unit 34.

[0223] The receiving unit 31 is used to receive the data cleaning requirements submitted by the user and parse the data cleaning requirements to obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. The cleaning task includes at least a task object field, task execution conditions and task execution order, where N is a positive integer.

[0224] The first generation unit 32 is used to match a filter template according to a preset cleaning rule for each cleaning task, and generate filter entity code by combining the filter template, task object fields and task execution conditions.

[0225] The second generation unit 33 is used to organize N filter entity codes into a chain execution structure according to the task execution order, and generate chain execution code, wherein the output of each filter entity code in the chain execution structure is used as the input of the next filter entity code.

[0226] The execution unit 34 is used to inject the chained execution code into the execution engine and run the execution engine to obtain the data cleaning results.

[0227] The aforementioned data cleaning device first receives data cleaning requests submitted by users through receiving unit 31, parses the data cleaning requests, and obtains N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. The cleaning task includes at least a task object field, task execution conditions, and task execution order, where N is a positive integer. Then, for each cleaning task, the first generation unit 32 matches a filter template according to the preset cleaning rule and generates filter entity code by combining the filter template, task object field, and task execution conditions. Then, the second generation unit 33 organizes the N filter entity codes into a chain execution structure according to the task execution order, generating chain execution code. The output of each filter entity code in the chain execution structure serves as the input of the next filter entity code. Finally, the running unit 34 injects the chain execution code into the execution engine and runs the execution engine to obtain the data cleaning result.

[0228] In this embodiment of the invention, a modular chain construction approach is adopted. By intelligently parsing and generating filter entity code through templates, the goal of efficiently and automatically processing multi-stage data cleaning needs is achieved. This realizes the technical effects of flexibility and traceability in the data cleaning process, resolves the problem that rigid rules are difficult to adapt to dynamic business needs, and solves the technical problem of low efficiency in traditional data cleaning in related technologies.

[0229] Specifically, this invention receives and parses user-submitted data cleaning requests, breaking them down into multiple independent cleaning tasks that are customized according to preset cleaning rules. For each cleaning task, a corresponding filter template is automatically matched according to the preset cleaning rules, and specialized filter entity code is generated by combining the specific object fields and execution conditions of the task. This greatly reduces the configuration difficulty for users, while also improving code generation efficiency and reducing potential programming errors. Subsequently, a chain execution structure is constructed according to the task execution order to generate chain execution code, ensuring smooth data flow between filters. The output of one filter seamlessly connects to the input of the next filter, forming a pipeline-like data cleaning process. Finally, the chain execution code is injected into the data processing execution engine, which runs the code to achieve batch data cleaning, significantly improving the speed and quality of data cleaning. The entire process is highly automated, supports flexible combinations of various cleaning rules, ensuring both the accuracy and efficiency of data cleaning. When dealing with large-scale datasets, it can effectively reduce the time consumption of data cleaning and improve the overall value conversion rate of business data.

[0230] Furthermore, the receiving unit includes: a parsing module, used to receive a user-submitted request form through a configuration interface, and parse the request form to obtain data cleaning task information, wherein the request form is used to record data cleaning requirements in a first data format, and the data cleaning task information is used to record data cleaning requirements in a second data format; a first determining module, used to determine N cleaning tasks and the corresponding preset cleaning rules for each cleaning task from the data cleaning task information based on a preset cleaning rule form, and to determine the task execution order according to the data cleaning task information; and a second determining module, used to determine the task object field and task execution conditions for each cleaning task according to the configuration information of the preset cleaning rules.

[0231] Furthermore, the data cleaning device also includes: a third determining module, used to, after receiving the data cleaning request submitted by the user and parsing the data cleaning request to obtain N cleaning tasks, determine the target source table information to be applied to each cleaning task based on the cleaning task information, wherein the target source table information includes at least: a list of source table fields; a first verification module, used to verify whether each task object field of the cleaning task exists in the list of source table fields, and obtain a field existence verification result; a second verification module, used to verify whether the task execution conditions contain all required parameter items according to the list of required parameters indicated by the configuration information of the preset cleaning rules, and obtain a parameter rationality verification result; a third verification module, used to verify whether the execution order of all tasks is continuous and without repetition, and obtain an order validity verification result; and a stopping module, used to match a filter template according to the preset cleaning rules if all verification results pass, otherwise, stop the data cleaning process and report an error to the configuration interface.

[0232] Further, the first generation unit includes: a fourth determining module, used to determine the filter type for each cleaning task according to preset cleaning rules, wherein the filter type includes: filtering type, transformation type, correction type, and deduplication type; a retrieving module, used to retrieve a filter template from a preset template library according to the filter type, wherein the filter template is a reusable code template pre-encapsulated for each preset cleaning rule; a first configuration module, used to configure the input / output data structure in the filter template based on the task object field, and to configure the rule parameters in the filter template based on the task execution conditions; and a fifth determining module, used to determine the filter entity code corresponding to each cleaning task after configuration.

[0233] Furthermore, the second generation unit includes: a second configuration module, used for configuring the input data of the filter entity code based on the task object field for a cleaning task with a task execution order of 1, and representing the output data as a first parameter placeholder; a third configuration module, used for configuring the input data of the filter entity code as the (i-1)th parameter placeholder for a cleaning task with a task execution order of i, and configuring the output data as the i-th placeholder, where 2≤i<N; and a fourth configuration module, used for configuring the input data of the filter entity code as the (N-1)th parameter placeholder for a cleaning task with a task execution order of N, and obtaining chained execution code after configuration.

[0234] Furthermore, the data cleaning device also includes: an inspection module, used to perform logical syntax checks on the chained execution code before injecting it into the execution engine, to determine that the input data source of each filter entity code is not empty, otherwise suspending the data cleaning process and triggering a data source missing alarm; and a monitoring module, used to monitor the execution status of each filter entity code during the execution engine operation, and to suspend the data cleaning process and trigger a timed retry if the execution status fails.

[0235] Furthermore, the execution unit includes: an execution module for running the execution engine, locating and extracting task object data from the operation data storage layer based on the input data of the filter entity code with task execution order 1; an injection module for injecting the task object data into the execution engine, performing data flow in a chained execution structure, and obtaining the output data of the filter entity code with task execution order N after the flow is completed; and a storage module for writing the output data into the data warehouse detail layer specified by the data cleaning requirements, wherein the data warehouse detail layer is responsible for storing all data cleaning results.

[0236] The data cleaning device described above may also include a processor and a memory. The receiving unit 31, the first generating unit 32, the second generating unit 33, the running unit 34, etc., are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0237] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the processor receives user-submitted data cleaning requests, parses these requests to obtain cleaning tasks, matches filter templates according to preset cleaning rules, and generates filter entity code by combining the filter templates, task object fields, and task execution conditions. The filter entity code is then organized into a chained execution structure according to the task execution order, generating chained execution code. This chained execution code is injected into the execution engine, which then runs to obtain the data cleaning results.

[0238] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0239] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: receiving a data cleaning request submitted by a user, parsing the data cleaning request to obtain N cleaning tasks, wherein each cleaning task corresponds to a preset cleaning rule, and the cleaning task at least includes a task object field, task execution conditions, and task execution order, where N is a positive integer; for each cleaning task, matching a filter template according to the preset cleaning rule, and generating filter entity code by combining the filter template, task object field, and task execution conditions; organizing the N filter entity codes into a chain execution structure according to the task execution order to generate chain execution code, wherein the output of each filter entity code in the chain execution structure serves as the input of the next filter entity code; injecting the chain execution code into the execution engine, and running the execution engine to obtain the data cleaning result.

[0240] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to perform any of the data cleaning methods in the first embodiment described above.

[0241] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data cleaning method of any one of the above embodiments.

[0242] Figure 4 This is a structural block diagram of an electronic device for performing a data cleaning method according to an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0243] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data cleaning method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned data cleaning method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0244] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0245] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0246] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0247] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0248] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0249] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0250] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0251] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0252] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / string>

Claims

1. A data cleaning method, characterized in that, include: Receive data cleaning requests submitted by users, parse the data cleaning requests, and obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. Each cleaning task includes at least a task object field, task execution conditions, and task execution order. N is a positive integer. For each cleaning task, a filter template is matched according to the preset cleaning rules, and filter entity code is generated by combining the filter template, the task object field, and the task execution conditions; According to the task execution order, the N filter entity codes are organized into a chain execution structure to generate chain execution code, wherein the output of each filter entity code in the chain execution structure is used as the input of the next filter entity code; The chained execution code is injected into the execution engine, and the execution engine is run to obtain the data cleaning results.

2. The data cleaning method according to claim 1, characterized in that, The steps of receiving and parsing data cleaning requests submitted by users to obtain N cleaning tasks include: The system receives a request form submitted by the user through a configuration interface, parses the request form, and obtains data cleaning task information. The request form is used to record the data cleaning request in a first data format, and the data cleaning task information is used to record the data cleaning request in a second data format. Based on the preset cleaning rule form, N cleaning tasks and the preset cleaning rule corresponding to each cleaning task are determined from the cleaning task information, and the execution order of the tasks is determined according to the cleaning task information. Based on the configuration information of the preset cleaning rules, the task object field and the task execution conditions for each cleaning task are determined.

3. The data cleaning method according to claim 2, characterized in that, After receiving the data cleaning request submitted by the user and parsing the data cleaning request to obtain N cleaning tasks, the process also includes: For each cleaning task, the target source table information to which the cleaning task is applied is determined based on the cleaning task information, wherein the target source table information includes at least: a list of source table fields; Verify whether each task object field of the cleaning task exists in the source table field list to obtain the field existence verification result; Based on the list of required parameters indicated by the configuration information of the preset cleaning rules, verify whether the task execution conditions contain all required parameter items, and obtain the parameter rationality verification result; Verify whether the execution order of all the tasks is consecutive and without repetition, and obtain the order validity verification result; If all verification results pass, the filter template is matched according to the preset cleaning rules; otherwise, the data cleaning process is stopped and an error is reported to the configuration interface.

4. The data cleaning method according to claim 1, characterized in that, For each cleaning task, the step of matching a filter template according to the preset cleaning rules and generating filter entity code by combining the filter template, the task object field, and the task execution conditions includes: For each cleaning task, the filter type is determined according to the preset cleaning rules, wherein the filter type includes: filtration type, conversion type, correction type, and deduplication type; The filter template is retrieved from the preset template library according to the filter type, wherein the filter template is a reusable code template pre-encapsulated for each preset cleaning rule; Based on the task object field, configure the input / output data structure in the filter template, and based on the task execution conditions, configure the rule parameters in the filter template; After configuration, determine the filter entity code corresponding to each cleaning task.

5. The data cleaning method according to claim 1, characterized in that, The steps of organizing the N filter entity codes into a chained execution structure and generating chained execution code according to the task execution order include: For the cleaning task with a task execution order of 1, the input data for the filter entity code is configured based on the task object field, and the output data is represented as a first parameter placeholder; For the cleaning task with execution order i, the input data of the filter entity code is configured as the (i-1)th parameter placeholder, and the output data is configured as the i-th placeholder, where 2≤i<N; For the cleaning task with an execution order of N, the input data of the filter entity code is configured as the (N-1)th parameter placeholder, and the chained execution code is obtained after the configuration is completed.

6. The data cleaning method according to claim 1, characterized in that, The data cleaning method also includes: Before injecting the chained execution code into the execution engine, a logical syntax check is performed on the chained execution code to determine that the input data source of each filter entity code is not empty; otherwise, the data cleaning process is paused and a data source missing alarm is triggered. During the execution of the execution engine, the execution status of each filter entity code is monitored, and if the execution status fails, the data cleaning process is paused and a timed retry is triggered.

7. The data cleaning method according to claim 1, characterized in that, The steps for running the execution engine to obtain data cleaning results include: Run the execution engine, locate the task object data in the operation data storage layer according to the input data of the filter entity code with the task execution order of 1, and extract the task object data; The task object data is injected into the execution engine, and data flow is performed in the chained execution structure. After the flow is completed, the output data of the filter entity code with the task execution order N is obtained. The output data is written to the data warehouse detail layer specified by the data cleaning requirement, wherein the data warehouse detail layer is responsible for storing all the data cleaning results.

8. A data cleaning apparatus, characterized in that, include: The receiving unit is used to receive data cleaning requests submitted by users, parse the data cleaning requests, and obtain N cleaning tasks. Each cleaning task corresponds to a preset cleaning rule. The cleaning task includes at least a task object field, task execution conditions, and task execution order. N is a positive integer. The first generation unit is used to match a filter template according to the preset cleaning rules for each cleaning task, and generate filter entity code by combining the filter template, the task object field and the task execution conditions. The second generation unit is used to organize the N filter entity codes into a chain execution structure according to the task execution order, and generate chain execution code, wherein the output of each filter entity code in the chain execution structure is used as the input of the next filter entity code; The execution unit is used to inject the chained execution code into the execution engine and run the execution engine to obtain the data cleaning results.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data cleaning method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data cleaning method according to any one of claims 1 to 7.

Images