Pipeline generation method and related device

Abstract

A pipeline generation method, comprising: acquiring a plurality of data cleaning operators for generating a pipeline; acquiring an attribute association relationship of data; extracting an attribute set from the plurality of data cleaning operators; grouping attributes in the attribute set on the basis of the attribute association relationship, so as to obtain a plurality of attribute groups which support concurrent computation, and the sorting of the plurality of attribute groups; on the basis of a topological sorting result of at least one attribute group among the plurality of attribute groups, determining the intra-group sorting of an operator group that corresponds to the at least one attribute group; and on the basis of the sorting of the plurality of attribute groups and the intra-group sorting of the operator group that corresponds to the at least one attribute group, orchestrating the plurality of data cleaning operators, so as to obtain a pipeline. In the method, attributes are first grouped to realize coarse-granularity pipeline recommendation, and the execution sequence of operators in a group is then intelligently orchestrated, so as to greatly reduce the number of operator combinations which require searching, thereby improving the efficiency of pipeline generation; moreover, the execution time is shortened by means of concurrency on the basis of ensuring precision, thereby improving the execution efficiency.

Description

A pipeline generation method and related device

[0001] The present application claims priority from the Chinese patent application No. 202410605239.5 filed on May 15, 2024, and entitled "A pipeline generation method and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of cloud computing technology, and in particular to a pipeline generation method, system, computing device cluster, computer readable storage medium, and computer program product. BACKGROUND

[0003] In the field of computing, a pipeline divides the entire workflow into a series of consecutive stages or tasks, and achieves efficient production or processing by taking the output of each stage as the input of the next stage. Each stage focuses on a specific task and passes its results to the next stage so that the entire process can proceed continuously.

[0004] Pipelines can be used in various large-scale data processing scenarios. For example, in the scenario of training a large language model (LLM), a large amount of data is usually required, such as gigabyte (GB) or even terabyte (TB) of data for training. The quality of the data greatly affects the effectiveness of the LLM. One way to improve data quality is to clean the data, and the LLM can achieve or even exceed the effect of using a proprietary data set by using only finely cleaned data.

[0005] However, real-world data often has defects, especially in the absence of comprehensive prior data labels and uniform standardization processes, data cleaning is often a complex and time-consuming task. There is an urgent need in the industry to provide a pipeline generation method to support automated processing, improve processing efficiency, and shorten processing time. SUMMARY

[0006] The application provides a pipeline generation method. The method extracts fine-grained data cleaning operators upwards to obtain multiple attribute groups supporting parallel computing and the ordering of the multiple attribute groups, thereby realizing coarse-grained pipeline recommendation. The method determines the intra-group ordering of the operator groups corresponding to the attribute groups according to the topological ordering results of the attribute groups, can obtain more accurate operator execution order, and improves the repair accuracy. Since the method first realizes coarse-grained pipeline recommendation by grouping attributes, and then intelligently arranges the intra-group operator execution order, the number of operator combinations that need to be searched is greatly reduced, the pipeline generation efficiency is improved, and the execution time is shortened through parallelism on the basis of ensuring accuracy, thereby improving the execution efficiency.

[0007] In a first aspect, the application provides a pipeline generation method. The method can be executed by a pipeline generation system. The pipeline generation system is a tool for generating a pipeline (such as a data cleaning pipeline). The pipeline generation system can be a software system, which can be a standalone software system or a plug-in or functional module integrated into other software systems. The software system can be provided to users in the form of a software package, and deployed by users on a computing device, such as a local computing device or a private cloud. The software system can also be provided to users in the form of a cloud service, such as an interface with pipeline generation capability provided by a cloud vendor for calling. In some examples, the pipeline generation system can be a hardware system, such as a computing device cluster with pipeline generation capability, which runs and executes the pipeline generation method of the application.

[0008] Specifically, the pipeline generation system can obtain multiple data cleaning operators for generating a pipeline, and obtain attribute association relationships of data. The pipeline is used to arrange the multiple data cleaning operators so that the multiple data cleaning operators are executed according to the arrangement result. The data cleaning operator is used to remove or repair abnormal data in a data set. The pipeline generation system can extract an attribute set of the multiple data cleaning operators, which includes attributes of input data of the multiple data cleaning operators and attributes of output data of the multiple operators. Then the pipeline generation system can group the attributes in the attribute set according to the attribute association relationships, to obtain multiple attribute groups supporting parallel computing and the ordering of the multiple attribute groups. Next, the pipeline generation system determines the intra-group ordering of the operator groups corresponding to at least one attribute group in the multiple attribute groups according to the topological ordering result of the at least one attribute group. The pipeline generation system arranges the multiple data cleaning operators according to the ordering of the multiple attribute groups and the intra-group ordering of the operator groups corresponding to the at least one attribute group, to obtain the pipeline.

[0009] In the method, the pipeline generation system abstracts fine-grained data cleaning operators into multiple attribute groups supporting parallel computing, and obtains the topological sorting of the multiple attribute groups, so as to realize coarse-grained pipeline recommendation, and according to the topological sorting result of the attribute groups, the group-in-order of the operator groups corresponding to the attribute groups is determined, so that more accurate operator execution order can be obtained, and the repair accuracy is improved. Since the method first realizes coarse-grained pipeline recommendation by grouping attributes, and then intelligently arranges the group-in-order of the operators, the number of operator combinations that need to be searched is greatly reduced, the combination explosion problem and the efficiency problem in the heuristic search stage in the related art are solved, the pipeline generation efficiency is improved, and the data cleaning pipeline generated by the method considers the dependency relationship and the order between the operators, has high accuracy, shortens the execution time on the basis of ensuring accuracy, and improves the execution efficiency.

[0010] In some possible implementation manners, the pipeline generation system can obtain the contribution degrees of the multiple data cleaning operators, the contribution degrees being used to indicate the data repair capabilities of the multiple data cleaning operators. For example, the contribution degree can be the repair rate of the data cleaning operator, and can be obtained according to the number of repairable data of the data cleaning operator / the total amount of column data estimated according to the existing operator and the representative sample. Correspondingly, the pipeline generation system can determine the group-in-order of the operator groups corresponding to the at least one attribute group according to the topological sorting result of the at least one attribute group and the contribution degrees of the multiple data cleaning operators.

[0011] The method further improves the accuracy of the sorting and the repair accuracy by sorting the data cleaning operators based on the topological sorting result and in combination with the contribution degrees of the data cleaning operators.

[0012] In some possible implementation manners, the pipeline generation system can further perform dependency detection on the connected subgraph corresponding to the at least one attribute group. When there is a cyclic dependency (also referred to as a conflict ring), the pipeline generation system can disconnect the incoming edges of the target attribute column in the connected subgraph to obtain a loop-free attribute graph. The target attribute column is an attribute column in the connected subgraph that meets the quality requirement, for example, is an attribute column with the highest quality. The pipeline generation system can perform topological sorting according to the loop-free attribute graph to obtain the topological sorting result of the at least one attribute group.

[0013] The method can improve the accuracy of the sorting by detecting and cutting off the cyclic dependency of the connected subgraph corresponding to the attribute group, and performing topological sorting based on the loop-free attribute graph.

[0014] In some possible implementation manners, the pipeline generation system can determine, from the connected subgraphs, a target attribute column according to qualities of the attribute columns recorded in the knowledge base, the target attribute column including an attribute column with the highest quality. Alternatively, the pipeline generation system can obtain, from the connected subgraphs, the target attribute column according to the primary key attribute set, the target attribute column being an attribute column included in the master data in the primary key attribute set. The master data can be data shared among the systems, for example, data to be shared between the operational / transactional application systems and the analytical systems. The primary key attribute set includes at least one primary key. The pipeline generation system can mine, based on a natural key mining algorithm, a field or a field combination with uniqueness and meaning from the data set (a set of data to be cleaned) as a candidate key, and then determine the primary key from the candidate key.

[0015] The method can determine, according to the qualities of the attribute columns recorded in the database, an attribute column with the highest quality, or determine, as the attribute column with the highest quality, an attribute column included in the master data in the primary key attribute set, and cut off the circular dependency based on this, thereby guaranteeing the accuracy of the topological sorting.

[0016] In some possible implementation manners, the pipeline generation system can identify semantics of the plurality of data cleaning operators. The semantics of the data cleaning operator can be a physical meaning expressed by the data cleaning operator, and can usually be a functional description in natural language form. Then the pipeline generation system can generate, according to the semantics of the plurality of data cleaning operators and the pipeline, an explanation of the data cleaning operator in the pipeline through the pipeline explanation template.

[0017] The method generates, based on the pipeline explanation template, the explanation of the data cleaning operator in the pipeline through the semantics of the data cleaning operator, thereby enhancing the explainability of the pipeline and helping the user to understand the role of each node (or each operator) in the pipeline.

[0018] In some possible implementation manners, the pipeline generation system can construct, according to the attribute association relationship and the attribute set, a dependency directed graph corresponding to the attribute set. In the dependency directed graph, a vertex represents an attribute in the attribute set, and an edge represents a dependency relationship between the attributes in the attribute set. The pipeline generation system can perform connectivity analysis on the dependency directed graph to obtain a plurality of connected subgraphs. The pipeline generation system can group the attributes in the attribute set according to the plurality of connected subgraphs to obtain a plurality of attribute groups supporting parallel computing.

[0019] The method constructs the dependency directed graph and performs connectivity analysis to obtain the connected subgraphs, and the connected subgraphs can be used as a basis for reasonable grouping of the attributes, thereby helping to improve the generation efficiency and execution efficiency of the pipeline.

[0020] In some possible implementation manners, the pipeline generation system can divide attributes belonging to the same connected subgraph in the multiple connected subgraphs into the same attribute group, to obtain multiple attribute groups. Then, the pipeline generation system can adjust at least one attribute group in the multiple attribute groups according to the parallelism and the number of operators corresponding to the multiple connected subgraphs.

[0021] The method further improves the accuracy of grouping by optimizing the attribute grouping result in combination of the parallelism and the number of operators corresponding to the connected subgraph (operator scale or operator magnitude), and can meet the requirement of the parallelism.

[0022] In a second aspect, the present application provides a pipeline generation system. The system comprises:

[0023] An attribute grouping subsystem is configured to obtain multiple data cleaning operators used for generating a pipeline, and obtain an attribute association relationship of data, the pipeline being configured to arrange the multiple data cleaning operators so that the multiple data cleaning operators perform according to an arrangement result, and the data cleaning operators being configured to remove or repair abnormal data in a data set.

[0024] The attribute grouping subsystem is further configured to extract an attribute set of the multiple data cleaning operators, the attribute set comprising attributes of input data of the multiple data cleaning operators and attributes of output data of the multiple operators.

[0025] The attribute grouping subsystem is further configured to group attributes in the attribute set according to the attribute association relationship, to obtain multiple attribute groups supporting parallel computing and an order of the multiple attribute groups.

[0026] An operator arrangement subsystem is configured to determine an intra-group order of an operator group corresponding to at least one attribute group in the multiple attribute groups according to a topological order result of the at least one attribute group.

[0027] The operator arrangement subsystem is further configured to arrange the multiple data cleaning operators according to the order of the multiple attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, to obtain the pipeline.

[0028] In some possible implementation manners, the operator arrangement subsystem is further configured to:

[0029] obtain contribution degrees of the multiple data cleaning operators, the contribution degrees being configured to indicate data repair capabilities of the multiple data cleaning operators.

[0030] The operator arrangement subsystem is specifically configured to:

[0031] Determine, according to the topological sorting result of at least one attribute group in the plurality of attribute groups and the contribution degree of the plurality of data cleaning operators, an intra-group order of an operator group corresponding to the at least one attribute group.

[0032] In some possible implementation manners, the operator arrangement subsystem is further configured to:

[0033] Perform dependency detection on a connected subgraph corresponding to at least one attribute group in the plurality of attribute groups;

[0034] When there is a circular dependency, disconnect an incoming edge of a target attribute column in the connected subgraph, to obtain an acyclic attribute graph, the target attribute column being an attribute column in the connected subgraph whose quality meets a requirement;

[0035] Perform topological sorting according to the acyclic attribute graph, to obtain a topological sorting result of the at least one attribute group.

[0036] In some possible implementation manners, the operator arrangement subsystem is further configured to:

[0037] According to the quality of the attribute columns recorded in the knowledge base, determine the target attribute column from the connected subgraph, the target attribute column including an attribute column with the highest quality; or

[0038] According to a primary key attribute set, obtain the target attribute column from the connected subgraph, the target attribute column being an attribute column included in the primary data of the primary key attribute set.

[0039] In some possible implementation manners, the attribute grouping subsystem is further configured to:

[0040] Identify semantics of the plurality of data cleaning operators;

[0041] The system further includes:

[0042] An explanation generation subsystem configured to generate, according to the semantics of the plurality of data cleaning operators and the pipeline, an explanation of a data cleaning operator in the pipeline by using a pipeline explanation template.

[0043] In some possible implementation manners, the attribute grouping subsystem is specifically configured to:

[0044] According to the attribute association relationship and the attribute set, construct a dependency directed graph corresponding to the attribute set, a vertex in the dependency directed graph representing an attribute in the attribute set, and an edge in the dependency directed graph representing a dependency relationship between attributes in the attribute set;

[0045] Perform connectivity analysis on the dependency directed graph, to obtain a plurality of connected subgraphs;

[0046] Grouping attributes in the attribute set according to the plurality of connected subgraphs, to obtain a plurality of attribute groups supporting parallel computation.

[0047] In some possible implementation manners, the attribute grouping subsystem is specifically configured to:

[0048] Dividing attributes belonging to the same connected subgraph in the plurality of connected subgraphs into the same attribute group, to obtain a plurality of attribute groups.

[0049] Adjusting at least one attribute group in the plurality of attribute groups according to the parallel degree and the number of operators corresponding to the plurality of connected subgraphs.

[0050] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory are in communication with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the method in the first aspect or any implementation manner of the first aspect.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores instructions. The instructions instruct the computing device or the computing device cluster to perform the method in the first aspect or any implementation manner of the first aspect.

[0052] In a fifth aspect, the present application provides a computer program product including instructions, which, when executed on a computing device or a computing device cluster, cause the computing device or the computing device cluster to perform the method in the first aspect or any implementation manner of the first aspect.

[0053] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows.

[0055] FIG. 1 is a schematic diagram of a data cleaning pipeline provided by the present application;

[0056] FIG. 2 is a flowchart of a fine-grained cleaning operator sequence generation provided by the present application;

[0057] FIG. 3 is an architecture schematic diagram of a pipeline generation system provided by the present application;

[0058] FIG. 4 is a flowchart of a pipeline generation method provided by the present application;

[0059] FIG. 5 is a flowchart of a data cleaning operator generation based on a representative sample of user-given abnormal data according to the present application;

[0060] FIG. 6 is a flowchart of pipeline generation at each stage according to the present application;

[0061] FIG. 7 is a flowchart of attribute group recommendation according to the present application;

[0062] FIG. 8 is a structural diagram of a dependency directed graph according to the present application;

[0063] FIG. 9 is a diagram of attribute group adjustment according to the present application;

[0064] FIG. 10 is a flowchart of pipeline level recommendation according to the present application;

[0065] FIG. 11 is a diagram of data clustering according to the present application;

[0066] FIG. 12 is a diagram of intra-group ordering of an operator group according to the present application;

[0067] FIG. 13 is a diagram of generating a pipeline with explainability according to the present application;

[0068] FIG. 14 is a diagram of a pipeline with explainability according to the present application;

[0069] FIG. 15 is a structural diagram of a computing device according to the present application;

[0070] FIG. 16 is a structural diagram of a computing device cluster according to the present application;

[0071] FIG. 17 is a structural diagram of another computing device cluster according to the present application;

[0072] FIG. 18 is a structural diagram of yet another computing device cluster according to the present application. DETAILED DESCRIPTION

[0073] The terms "first", "second", etc. in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0074] First, some technical terms involved in the embodiments of the present application are introduced.

[0075] Data cleaning is a process of detecting and correcting (or deleting) damaged or inaccurate records from a record set, a database table, or a database. Specifically, data cleaning can include identifying incomplete, incorrect, inaccurate, or irrelevant parts of data, and then performing replacement, modification, or deletion operations.

[0076] A data cleaning operator is a set of tools or functions used in a data cleaning process to remove or repair abnormal data in a data set. Abnormal data can include, but is not limited to, missing data, duplicate data, or erroneous data. The tools used in the data cleaning process can include custom expressions or model operators, which are computational units in a model for implementing specific functions. Data cleaning operators can obtain high-quality data by performing specific cleaning tasks, such as identification and correction of erroneous data. Data cleaning operators can be divided according to cleaning granularity. For example, data cleaning operators for a database table can include cell-level data cleaning operators and column-level data cleaning operators.

[0077] For ease of understanding, the present application provides relevant examples for illustration.

[0078] A cell-level data cleaning operator can include the following operators: Replace“British”with‘English’ Replace“film-maker.”with‘filmmaker’

[0079] A column-level data cleaning operator can include the following operators: FORMAT_DATE(date,'yyyy / MM / DD') UNIX_TIMESTAMP(unix,'yyyy / MM / DD')

[0080] A data cleaning pipeline is a pipeline for implementing data cleaning and is used to represent the data cleaning process. Among them, the pipeline is used to orchestrate multiple data cleaning operators so that the multiple data cleaning operators are executed according to the orchestration result. The data cleaning process can consist of a series of cleaning nodes, and each cleaning node includes a series of fine-grained cleaning operators (which can also be simply referred to as operators). There is a sequence for the operators within a cleaning node. There is also a sequence for the cleaning nodes when they are executed. This application provides an example of a data cleaning pipeline. As shown in FIG. 1, Date, Gender, FnameToGender, and AreacodeToState are cleaning nodes. Among them, the operators in the Date cleaning node are used to modify the date to a specified format. Similarly, the operators in the Gender cleaning node are used to modify the gender to a specified format. For example, the data cleaning operator Replace “male” with ‘M’ in the Gender cleaning node is used to replace “male” in the cell with “M”, the data cleaning operator Replace “female” with ‘F’ is used to replace “female” in the cell with “F”, and the data cleaning operator Replace “Female” with ‘F’ is used to replace “Female” in the cell with “F”.

[0081] Data in the real world often has diverse defects. In the absence of comprehensive prior data labels and a unified standardization process, data cleaning is often a complex and time-consuming task. For this reason, related technologies provide a repair system for multiple data error types. This repair system supports allowing users to add existing data cleaning methods through a unified interface, and the data cleaning methods form a parameterized data cleaning library. This repair system can generate a series of intermediate results, which can be operators and parameters under a series of established goals. Then, the above intermediate results are converted into an intermediate language, and then tree search and pruning heuristic search are used to search for the intermediate results to obtain a data cleaning pipeline. This data cleaning pipeline can be a sequence of fine-grained cleaning operators. FIG. 2 shows a flowchart for generating a sequence of fine-grained cleaning operators. In order to evaluate the generated data cleaning pipeline, quality assessment metric definition can be performed first, such as defining quality assessment indicators or a quality assessment model. Among them, the quality assessment indicators can include repair accuracy. Then, a candidate set of cleaning operations can be generated. The candidate set of cleaning operations can include a series of cleaning operation operators (data cleaning operators) to be followed. A sequence of cleaning operators is generated through a search algorithm, and the sequence of cleaning operators is evaluated based on the defined quality assessment indicators. If the quality assessment indicators do not meet the requirements, continue to search to generate a new sequence of cleaning operators and perform quality assessment until the quality assessment indicators meet the requirements. Finally, a sequence of fine-grained cleaning operators that meets the quality assessment indicators is obtained. This sequence of cleaning operators can be used to perform data cleaning on large-scale data and output the cleaned data.

[0082] The repair system for multiple data error types provides significant flexibility and adaptability, but the repair system may face the main challenge of the combinatorial explosion problem that may occur during the data repair process and the efficiency problem of the heuristic search phase.

[0083] Therefore, the present application provides a pipeline generation method. The method can be executed by a pipeline generation system. The pipeline generation system is a tool for generating a pipeline, for example, generating a data cleaning pipeline. The pipeline generation system can be a software system, which can be a standalone software system or a plug-in or functional module integrated into other software systems. The software system can be provided to users in the form of a software package, which can be deployed by users on a computing device, such as a local computing device or a private cloud. The software system can also be provided to users in the form of a cloud service, such as an application programming interface (API) with pipeline generation capabilities provided by a cloud vendor of a public cloud for users to use. In some examples, the pipeline generation system can be a hardware system, such as a computing device cluster with pipeline generation capabilities, which runs and executes the pipeline generation method of the present application.

[0084] Specifically, the pipeline generation system can obtain a plurality of data cleaning operators for generating a pipeline, and obtain an attribute association relationship of data. The pipeline is used to arrange the plurality of data cleaning operators so that the plurality of data cleaning operators execute according to the arrangement result. The data cleaning operator is used to remove or repair abnormal data in a data set. The pipeline generation system can extract an attribute set of the plurality of data cleaning operators, which includes attributes of input data of the plurality of data cleaning operators and attributes of output data of the plurality of operators. Then the pipeline generation system can group the attributes in the attribute set according to the attribute association relationship, obtain a plurality of attribute groups supporting parallel computing and an order of the plurality of attribute groups. Next, the pipeline generation system can determine the intra-group order of an operator group corresponding to at least one attribute group according to the topological order result of the at least one attribute group. The pipeline generation system arranges the plurality of data cleaning operators according to the order of the plurality of attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, and obtains the pipeline.

[0085] The method realizes coarse-grained pipeline recommendation by up-sampling fine-grained data cleaning operators to obtain multiple attribute groups supporting parallel computing and the ordering of the multiple attribute groups, and determines the intra-group ordering of the operator groups corresponding to the attribute groups according to the topological ordering result of the attribute groups, so as to obtain more accurate operator execution order and improve the repair accuracy. Since the method first realizes coarse-grained pipeline recommendation by attribute grouping, and then intelligently arranges the intra-group operator execution order, the number of operator combinations that need to be searched is greatly reduced, the combination explosion problem and the efficiency problem in the heuristic search stage in the related art are solved, the pipeline generation efficiency is improved, and the data cleaning pipeline generated by the method takes into account the dependency and sequence between operators, has high accuracy, shortens the execution time and improves the execution efficiency on the basis of ensuring accuracy.

[0086] In order to make the technical solutions of the present application clearer and easier to understand, the architecture of the pipeline generation system of the present application will be introduced below in combination with the drawings.

[0087] Referring to the architecture schematic diagram of a pipeline generation system shown in FIG. 3, the pipeline generation system 10 includes an attribute grouping subsystem 100 and an operator arrangement subsystem 200. Further, the pipeline generation system 10 can also include an explanation generation subsystem 300. Similar to the pipeline generation system 10, the above-mentioned attribute grouping subsystem 100, operator arrangement subsystem 200 and explanation generation subsystem 300 can be implemented by software or by hardware. The attribute grouping subsystem 100, operator arrangement subsystem 200 and explanation generation subsystem 300 will be introduced below.

[0088] The attribute grouping subsystem 100 is configured to obtain multiple data cleaning operators for generating a pipeline, and obtain attribute association relationships of data, and then extract attribute sets of the multiple data cleaning operators. The pipeline is configured to arrange the multiple data cleaning operators to make the multiple data cleaning operators execute according to the arrangement result, the data cleaning operator is configured to remove or repair abnormal data in a data set, and the attribute set includes attributes of input data of the multiple data cleaning operators and attributes of output data of the multiple operators. For example, the attributes of the data cleaning operator can include "zip", "areacode", "state" and the like. The attribute grouping subsystem 100 is further configured to group the attributes in the attribute set according to the attribute association relationships, to obtain multiple attribute groups supporting parallel computing and the ordering of the multiple attribute groups. The multiple attribute groups supporting parallel computing can also be referred to as parallel attribute blocks, simply referred to as parallel blocks, and coarse-grained pipeline recommendation can be realized by extracting parallel blocks of operators.

[0089] The operator arrangement subsystem 200 is configured to determine the intra-group order of the operator group corresponding to at least one attribute group according to the topological sorting result of the at least one attribute group in the plurality of attribute groups. The operator arrangement subsystem 200 is further configured to arrange the plurality of data cleaning operators according to the order of the plurality of attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, and obtain a pipeline. In this way, the execution order of the operators can be intelligently arranged based on the topological sorting result of the attribute groups, thereby improving the accuracy of execution. Moreover, the method can implement distributed parallel execution, thereby improving the execution efficiency.

[0090] The explanation generation subsystem 300 is configured to generate an explanation of each data cleaning operator in the pipeline according to the semantics of the plurality of data cleaning operators and the pipeline. The semantics of the plurality of data cleaning operators can be identified by the attribute grouping subsystem 100, for example, the attribute grouping subsystem 100 identifies the semantics of the plurality of data cleaning operators when extracting the attribute set of the plurality of data cleaning operators. The attribute grouping subsystem 100 provides the semantics of the plurality of data cleaning operators to the explanation generation subsystem 300, and the operator arrangement subsystem 200 provides the generated pipeline to the explanation generation subsystem 300. Accordingly, the explanation generation subsystem 300 can generate an explanation for each data cleaning operator in the pipeline in combination with the semantics and the pipeline explanation template.

[0091] It should be noted that the input of the pipeline generation system 10 can include the plurality of data cleaning operators and the attribute association relationship. The plurality of data cleaning operators can be fine-grained cleaning operators, which can be determined based on the scheme of FIG. 1. The present application can recommend a parallel block based on the scheme of FIG. 1, and determine the range of operators that can be executed in parallel. Then, the pipeline level recommendation can be performed, for example, the determined range of operators that can be executed in parallel is internally sorted and blocked, thereby generating a final data cleaning pipeline. For details, refer to FIG. 3. The cleaning parameters of the large-scale data set are generated, and then the cleaning parameters are dynamically optimized and evaluated to obtain cleaning parameters that meet the requirements. Based on the cleaning parameters that meet the requirements, the fine-grained data cleaning operators can be generated. The plurality of generated data cleaning operators can be used to generate a pipeline. In addition, based on the attribute association analysis of the large-scale data set, the attribute association relationship can be generated, which can be input into the pipeline generation system 10 to generate a pipeline. Further, the input of the pipeline generation system 10 can also include a knowledge base, which can store the historically generated attribute association relationship. After the pipeline is generated, the user can confirm the execution to output the cleaned data.

[0092] Based on the foregoing pipeline generation system 10, the present application further provides a pipeline generation method. Referring to the flowchart of a pipeline generation method shown in FIG. 4, the method includes the following steps:

[0093] S402, the pipeline generation system 10 acquires a plurality of data cleaning operators for generating a pipeline.

[0094] The data cleaning operator can be an operator with data cleaning capability, used to remove or repair abnormal data in a data set. The abnormal data can be missing data, duplicate data, or error data. The missing data can be data with a null value, or a missing attribute column in a database table. The duplicate data can be data with repeated values, and the error data refers to data that does not meet the requirements in format or value.

[0095] The data cleaning operator can include multiple types, such as custom expressions, functions, or model operators. The data cleaning operator can be a fine-grained data cleaning operator, such as a function with data cleaning capability, which is used to perform specific data cleaning tasks, including but not limited to error data identification and correction. Further, the data cleaning operator can be divided into cell-level data cleaning operators and column-level data cleaning operators according to the cleaning granularity. The cell-level data cleaning operator can be used to repair data for a single cell, and the column-level data cleaning operator can be used to repair data for an attribute column.

[0096] The pipeline is used to arrange a plurality of data cleaning operators so that the plurality of data cleaning operators are executed according to the arrangement result. The pipeline that arranges the data cleaning operators is also called a data cleaning pipeline, which is used for automatic cleaning of data. In specific implementation, the pipeline generation system 10 can acquire a user-defined operator list, which includes a plurality of data cleaning operators. The pipeline generation system 10 can also acquire a recommended operator list according to the data to be cleaned (such as a data set). The operator list includes a series of fine-grained data cleaning operators recommended based on related technologies.

[0097] The recommended data cleaning operator can be generated in different ways. One way can be seen in FIG. 2, which is based on a user-given quality evaluation index to dynamically search the cleaning operation candidate set to determine the recommended data cleaning operator (sequence). Another way can be seen in FIG. 5, which is based on a user-given representative sample of abnormal data to generate a data cleaning operator. As shown in FIG. 5, the user can label abnormal data, and then fine-tune the model according to the labeled abnormal data, generate a possible correct data candidate set based on the fine-tuned model, extract features based on the possible correct data candidate set and perform clustering, so that the possible correct data candidate set can be extracted according to the category, and the final cleaning data can be predicted by the fine-tuned model for the extracted data. The user can decide whether to use the fine-tuned model (i.e., model operator) as a data cleaning operator according to the effect of the final cleaning data, such as accuracy or repair rate. When the user confirms to use the data cleaning operator to perform data cleaning, the data cleaning operator can be used to clean the large-scale data set, and the cleaned data can be obtained.

[0098] In S404, the pipeline generation system 10 acquires the attribute association relationship of the data.

[0099] The attribute association relationship can include the dependency relationship between attributes, such as functional dependency (FD) and multivalued dependency (MD). Further, based on the functional dependency, a new dependency relationship is extended, such as conditional functional dependency (CFD).

[0100] In database theory, functional dependency refers to a specific relationship in which the value of one attribute set uniquely determines the value of another attribute set. Let R(U) be a relation schema on attribute U, X and Y are both subsets of U = {A1, A2, …, An}, and r is any relation of R. If for any two tuples u, v in r, as long as u[X] = v[X], then u[Y] = v[Y], it is called that X functionally determines Y, or Y functionally depends on X, denoted as X→Y. For example, if in a database table, student ID (denoted as sno, as an attribute set) uniquely determines student name (denoted as sname, another attribute set), it is said that student ID has a functional dependency relationship on student name, sno→sname.

[0101] Among them, the functional dependency can be divided into complete functional dependency and partial functional dependency. In R(U), if X→Y, and for any true subset X' of X, X'→>Y, then Y completely depends on X; otherwise, if X→Y, and there is a true subset X' in X, such that X'→Y is established, then Y partially depends on X.

[0102] Conditional functional dependency is a rule for data consistency detection, which extends the traditional concept of functional dependency and allows the definition of dependency relationship under certain conditions. Conditional functional dependency helps identify inconsistencies and errors in data by specifying how attribute columns in a dataset behave based on the values of other attribute columns. For example, student ID (sno) determines student name (sname), but only when course ID (cno) is a certain value, student name can determine course name (cname), which indicates that conditional functional dependency is established, denoted as (sno, cno)→sname, cname.

[0103] Multivalued dependency is another dependency relationship in database theory. In a relational table, if the values of a set of attributes A determine the values of another set of attributes B, regardless of other attributes, then B has a multivalued dependency on A. This means that if there are two rows in the database table with the same value on A, their values on B can be interchanged without affecting other data in the table.

[0104] Let R(U) be a relation schema over a set of attributes U, X, Y, and Z be subsets of U, and Z = U-X-Y. If for any relation r of R, each value of r on (X, Z) corresponds to a set of Y values, which are determined only by the X value and are independent of the Z value, then the multivalued dependency is established, denoted as X→→Y. It should be noted that if a set in the multivalued dependency includes a value, the multivalued dependency becomes a functional dependency, in other words, the functional dependency can be regarded as a special case of the multivalued dependency.

[0105] The pipeline generation system 10 can perform attribute association analysis on the data to be cleaned, such as a large-scale data set, to obtain attribute association relationships.

[0106] S406, the pipeline generation system 10 extracts the attribute set of the plurality of data cleaning operators.

[0107] The attribute set includes the attributes of the input data of the plurality of data cleaning operators and the attributes of the output data of the plurality of operators. Further, during the execution of the data cleaning operator, if intermediate data is involved, the attribute set can also include the attributes of the intermediate data involved by the data cleaning operator.

[0108] The pipeline generation system 10 can extract the attribute set of the plurality of data cleaning operators through an information extraction template or an information extraction model (e.g., a large model). The following will be described respectively.

[0109] In some possible implementation manners, the pipeline generation system 10 can obtain a system definition, construct an information extraction template according to the system definition, and then match the data cleaning operator and the information extraction template to extract the attributes of the data cleaning operator. For example, the information extraction template can define a keyword, and tokens after the keyword are attributes of input data or output data of the data cleaning operator. Taking the data cleaning operator Replace“British”with‘English’as an example, tokens after the keyword replace can be attributes of the input data, and tokens after the keyword with can be attributes of the output data.

[0110] In some possible implementation manners, the pipeline generation system 10 can obtain a system definition, construct an information extraction template according to the system definition, and then match the data cleaning operator and the information extraction template to extract the attributes of the data cleaning operator. For example, the information extraction template can define a keyword, and tokens after the keyword are attributes of input data or output data of the data cleaning operator. Taking the data cleaning operator Replace“British”with‘English’as an example, tokens after the keyword replace can be attributes of the input data, and tokens after the keyword with can be attributes of the output data.

[0111] S408. The pipeline generation system 10 groups the attributes in the attribute set according to the attribute association relationship, and obtains a plurality of attribute groups supporting parallel computing and an order of the plurality of attribute groups.

[0112] Specifically, the pipeline generation system 10 can construct a dependency directed graph corresponding to the attribute set according to the attribute association relationship and the attribute set. The dependency directed graph is a directed graph representing the dependency relationship of the attributes in the attribute set. The vertices in the dependency directed graph represent the attributes in the attribute set, and the edges in the dependency directed graph represent the dependency relationship of the attributes in the attribute set. The pipeline generation system 10 can perform connectivity analysis on the dependency directed graph to obtain a plurality of spanning subgraphs. The spanning subgraph is a subset of the dependency directed graph, and the set of the plurality of spanning subgraphs includes all vertices in the dependency directed graph (also referred to as the original graph), but retains part of the edges of the dependency directed graph, and the retained part of the edges constitutes the spanning subgraph. For example, the set of the spanning subgraph G’ generated from the original graph G includes the same number of vertices as the original graph G, but the edge set E’ can be any subset of the original graph G. The pipeline generation system 10 can group the attributes in the attribute set according to the plurality of spanning subgraphs, and obtain a plurality of attribute groups supporting parallel computing.

[0113] As shown in FIG. 6, after constructing the dependency directed graph, performing connectivity analysis on the dependency directed graph to obtain the attribute connectivity subgraphs, the pipeline generation system 10 can group the attributes in the attribute set based on the parallelism or the operator magnitude. The connectivity subgraph is equivalent to the blocking operation on the attributes. The pipeline generation system 10 can first initialize the grouping of the attribute set based on the connectivity subgraph to obtain a plurality of attribute groups. Then, the pipeline generation system 10 can dynamically adjust the plurality of attribute groups in combination with the parallelism and the operator magnitude, thereby obtaining a plurality of attribute groups supporting parallel computing.

[0114] In some possible implementation manners, the pipeline generation system 10 can divide the attributes belonging to the same connectivity subgraph in the plurality of connectivity subgraphs into the same attribute group to obtain a plurality of attribute groups. Then, the pipeline generation system 10 can adjust at least one attribute group in the plurality of attribute groups according to the parallelism and the number of operators corresponding to the plurality of connectivity subgraphs.

[0115] For ease of understanding, an example is further provided for illustration. In the example, the pipeline generation system 10 performs connectivity analysis on the dependency directed graph to obtain two connectivity subgraphs, one connectivity subgraph includes vertices A and B, and one connectivity subgraph includes vertices C, D, E, and F, each vertex represents an attribute. The pipeline generation system 10 can obtain the attribute groups corresponding to the connectivity subgraphs according to the connectivity subgraphs, which can be specifically represented as {A, B} and {C, D, E, F}. The number of operators corresponding to the connectivity subgraphs is 47 and 103, respectively. If the corresponding operators are directly executed according to the attribute groups, some operators need to wait for the execution of other operators to be completed. Assuming that the user-defined parallelism is 2, the pipeline generation system 10 can further divide the attribute group {C, D, E, F} into attribute groups that can be executed in parallel, for example, {C, D} and {E, F}, according to the parallelism and the number of operators corresponding to the connectivity subgraphs.

[0116] S410, the pipeline generation system 10 determines the intra-group ordering of the operator group corresponding to the at least one attribute group according to the topological ordering result of the at least one attribute group.

[0117] Topological sorting is a process of sorting all nodes in a directed acyclic graph (DAG), and is usually used to describe the execution order of a series of tasks or events with dependency relationships. For example, when a university schedules courses, there is a precedence relationship between courses, and topological sorting can help determine the order of learning courses.

[0118] The topological sorting result satisfies the following condition: if there is a directed edge from node A to node B in the DAG, then in the sorting result, node A must appear before node B. In particular to the present application, the pipeline generation system 10 can perform topological sorting on the connected subgraph corresponding to the attribute group, thereby obtaining a topological sorting result of the at least one attribute group. The pipeline generation system 10 can determine the intra-group ordering of the operator group corresponding to the at least one attribute group according to the topological sorting result. In which, the operator corresponding to the attribute with higher ordering in the topological sorting result can have higher ordering in the operator group.

[0119] As shown in FIG. 6, considering the case that some connected subgraphs have cyclic dependencies (also referred to as dependency loops), the pipeline generation system 10 can perform dependency detection on the connected subgraph corresponding to at least one attribute group in the plurality of attribute groups. When there is a cyclic dependency, the pipeline generation system 10 disconnects the incoming edges of the target attribute column in the connected subgraph to obtain an acyclic attribute graph. In which, the target attribute column can be an attribute column with quality meeting the requirements in the connected subgraph, for example, the target attribute column can be the attribute column with the highest quality. Then the pipeline generation system 10 can perform topological sorting on the acyclic attribute graph to obtain a topological sorting result of the at least one attribute group.

[0120] In the above embodiment, the quality of the attribute column can be obtained from the knowledge base or determined according to the primary key attribute set. For example, the pipeline generation system 10 can determine the target attribute column from the connected subgraph according to the quality of the attribute column recorded in the knowledge base, and the target attribute column includes the attribute column with the highest quality. Alternatively, the pipeline generation system 10 can obtain the target attribute column from the connected subgraph according to the primary key attribute set, and the target attribute column is the attribute column included in the primary data in the primary key attribute set. In which, the primary data can be data shared among various systems, for example, data to be shared between operation / transaction type application systems and analytical systems. The primary key attribute set includes at least one primary key. The pipeline generation system 10 can mine a field or field combination with uniqueness and meaning as a candidate key from the data set (a set of data to be cleaned) based on a natural key mining algorithm, and then determine the primary key from the candidate key.

[0121] The pipeline generation system 10 can map the attribute groups to the operator groups according to the attribute set of the data cleaning operators, and then the pipeline generation system can determine the intra-group ordering of the operator groups corresponding to the attribute groups based on the topological ordering result of the attribute groups. Further, referring to FIG. 6, the pipeline generation system can also order the data cleaning operators in the operator groups in combination with the contribution degrees of the data cleaning operators. Specifically, the pipeline generation system 10 can obtain the contribution degrees of the plurality of data cleaning operators, the contribution degrees being used to indicate the data repair capabilities of the plurality of data cleaning operators. Accordingly, the pipeline generation system 10 determines the intra-group ordering of the operator groups corresponding to the at least one attribute group according to the topological ordering result of the at least one attribute group and the contribution degrees of the plurality of data cleaning operators.

[0122] S412, the pipeline generation system 10 arranges the plurality of data cleaning operators according to the ordering of the plurality of attribute groups and the intra-group ordering of the operator groups corresponding to the at least one attribute group, and obtains a pipeline.

[0123] Specifically, the pipeline generation system 10 can arrange the plurality of data cleaning operators according to the ordering of the attribute groups and the intra-group ordering of the operators corresponding to the attribute groups, guarantee that the attributes of the arranged operators satisfy the inter-group ordering of the attribute groups and the intra-group ordering of the attribute groups, and thus obtain a final data cleaning pipeline. When executing the above data cleaning pipeline, each node in the data cleaning pipeline can be arranged to generate a parallel cleaning plan to achieve higher parallelism, thereby helping users quickly obtain cleaned data.

[0124] Further, the pipeline generation system 10 can also identify the semantics of the plurality of data cleaning operators. For example, when the pipeline generation system 10 extracts the attributes of the data cleaning operators through the information extraction template or the information extraction model, the above information extraction template or information extraction model is used to extract the semantics of the data cleaning operators. Then, the pipeline generation system 10 can generate an explanation of the data cleaning operators in the pipeline according to the semantics of the plurality of data cleaning operators and the pipeline through a pipeline explanation template.

[0125] Based on the above content description, the pipeline generation method of the present application can obtain a plurality of attribute groups supporting parallel computing and the ordering of the plurality of attribute groups by up-sampling the fine-grained data cleaning operators, thereby realizing coarse-grained pipeline recommendation. According to the topological ordering result of the attribute groups, the group-in-order of the attribute group corresponding operator group is determined, and a more accurate operator execution order can be obtained, and the repair accuracy is improved. The method first realizes coarse-grained pipeline recommendation by grouping attributes, and then intelligently arranges the intra-group operator execution order, greatly reducing the number of operator combinations that need to be searched, solving the combination explosion problem and the efficiency problem in the heuristic search stage in related technologies, improving the pipeline generation efficiency. Moreover, the data cleaning pipeline generated by the method considers the dependency and order between operators, has high accuracy, and on the basis of ensuring accuracy, shortens the execution time and improves the execution efficiency through parallelism.

[0126] In order to make the technical solutions of the present application clearer and easier to understand, the pipeline generation method of the present application will be introduced below in combination with a specific application scenario.

[0127] In this scenario, the input of the pipeline generation system 10 can include fine-grained data cleaning operators, attribute association relationships and a knowledge base. The fine-grained data cleaning operators can include custom expressions, structured query (Structured Query Language, SQL) functions or model operators.

[0128] The custom expression can be an expression defined by a keyword, for example, Replace "British" with "English", Replace "film-maker." with 'filmmaker'. SQL is a standard programming language for managing and operating relational databases. SQL allows users to perform various data operations, including querying, inserting, updating, deleting, and creating and modifying database structures. The main feature of SQL is to provide a declarative way to express data operations. Users specify the operations they want to perform without describing how to perform the operations. For example, SQL functions can include FORMAT_DATE(date,'yyyy / MM / DD'), UNIX_TIMESTAMP(unix,'yyyy / MM / DD'). Model operators can include operators in semantic repair models, where the semantic repair model can be a Bidirectional Encoder Representations from Transformers (BERT) model based on a transformer.

[0129] The attribute association relationship can include a functional dependency (denoted as X→Y), a conditional functional dependency (denoted as (X, Z)→(Y)), or a multivalued dependency X→→Y. The knowledge base includes the knowledge of the dependency relationship, which can be an extension of the attribute association relationship.

[0130] Before performing attribute group recommendation, the fine-grained data cleaning operator can be parsed to obtain the semantics and attributes of the data cleaning operator. Specifically, the pipeline generation system 10 can extract the semantics and attributes of the input data cleaning operator through an information extraction template or an information extraction model.

[0131] Based on the attributes of the data cleaning operator and the input attribute association relationship, attribute group recommendation can be performed. The attribute group recommendation process is described in detail below.

[0132] Referring to a flowchart of attribute group recommendation shown in FIG. 7, the process includes the following steps:

[0133] S702, the pipeline generation system 10 parses the attributes of the data cleaning operator according to the attribute association relationship, and obtains the attribute association relationship of the data cleaning operator.

[0134] S704, the pipeline generation system 10 constructs a dependency directed graph according to the attribute association relationship of the data cleaning operator.

[0135] In this scenario, the attribute association relationship of the data cleaning operator can be:

[0136] The dependency directed graph constructed according to the attribute association relationship of the data cleaning operator is shown in FIG. 8. The vertices in FIG. 8 are attributes, including but not limited to "zip", "lname", "fname", "area code", "state", "gender", and "city". The directed edges in FIG. 8 represent the attribute association relationship. For example, the directed edge from "zip" to "city" represents ["zip"]→["city"].

[0137] S706, the pipeline generation system 10 performs connectivity analysis on the dependency directed graph to obtain a connected subgraph.

[0138] Specifically, the pipeline generation system 10 can perform connectivity analysis on the dependency directed graph by Tarjan algorithm or Depth-First-Search (DFS) to obtain connected subgraphs. Tarjan algorithm is an algorithm for finding strongly connected components in a graph. Tarjan algorithm takes a directed graph as input and gives a partition of its vertex set according to the strongly connected components it lies in. Each node in the directed graph only appears in one strongly connected component. The specific implementation of Tarjan algorithm can include starting a depth-first search from an optional node, not visiting the nodes that have been visited in the search process, and constructing a search tree whose subtrees constitute the strongly connected components of the directed graph. Each strongly connected component corresponds to a connected subgraph.

[0139] S708, the pipeline generation system 10 obtains a plurality of attribute groups according to the connected subgraphs and the attribute sets formed by the attributes of the data cleaning operators.

[0140] Specifically, referring to FIG. 7, the pipeline generation system 10 can generate parallel classification sets mi according to the connected subgraphs. In this example, the parallel classification sets can include m1 = ["Zip", "City", "State", "areacode"], m2 = ["fname", "lname", "gender"]. Then the pipeline generation system 10 can determine whether the attribute set Si belongs to a certain mj subset. If yes, Si is classified into mj. If not, Si is subdivided until the subdivided attribute set belongs to a certain mj subset. When each attribute of the attribute set has been classified, a plurality of attribute groups are generated according to the classified mj subsets. Each attribute group includes attributes in the attribute set that are classified into the same mj subset.

[0141] S710, the pipeline generation system 10 adjusts at least one attribute group in the plurality of attribute groups according to the parallelism and the number of operators corresponding to the connected subgraph to obtain a final attribute group.

[0142] The pipeline generation system 10 can adjust the plurality of attribute groups by a dynamic optimization algorithm based on the parallelism and the number of fine-grained data cleaning operators in each connected subgraph. As shown in FIG. 9, the pipeline generation system 10 can split attribute group 1 into two attribute groups, for example, attribute group 1-1 and attribute group 1-2. The data cleaning operators corresponding to attribute group 1-1 and attribute group 1-2 can support parallel execution. Further, the pipeline generation system 10 can also generate an order of the plurality of attribute groups.

[0143] Next, the pipeline stage recommendation is described in detail. Referring to the flowchart of one pipeline stage recommendation shown in FIG. 10, the specific steps include:

[0144] S1002, the pipeline generation system 10 extracts a representative sample from the data to be cleaned.

[0145] The purpose of extracting the representative sample (representative sample sampling) is to provide a basis for calculating the contribution of the data cleaning operator. In actual implementation, the pipeline generation system 10 can group the data using a clustering algorithm. As shown in FIG. 11, different colors or gray levels of points represent different clustering results. The analysis results of the above clustering analysis can help the user understand the distribution characteristics within the data block and ensure that samples are uniformly extracted from each cluster to form a representative sample set.

[0146] S1004, the pipeline generation system 10 calculates the quality of the attribute column based on the representative sample.

[0147] S1006, the pipeline generation system 10 calculates the contribution of the data cleaning operator based on the representative sample.

[0148] In this application, the quality of the attribute column can be represented by a quality score, where the quality score can be the ratio of the number of abnormal data in each column to the total amount of column data, and the contribution of the data cleaning operator can be the ratio of the number of repairable data of the data cleaning operator to the total amount of column data.

[0149] The pipeline generation system 10 can estimate the number of abnormal data of each attribute column / total amount of column data*100% according to the existing operator and the representative sample (sample data) to obtain the quality score. Similarly, the pipeline generation system 10 can estimate the number of repairable data of the data cleaning operator / total amount of column data*100% according to the existing operator and the representative sample to obtain the contribution of the data cleaning operator.

[0150] The quality score of the attribute column and the contribution of the data cleaning operator can be used to remove loops in the presence of loops in the dependent directed graph or connected subgraph, and to effectively sort the data cleaning operators corresponding to the attributes in the attribute group.

[0151] S1008, the pipeline generation system 10 determines whether the connected subgraph has a circular dependency. If so, perform S1010.

[0152] Specifically, the pipeline generation system 10 can determine whether the connected subgraph has a circular dependency based on DFS or Kosaraju algorithm. The Kosaraju algorithm is also known as Kosaraju-Sharir algorithm, which is used to find the strongly connected components of a directed graph in linear time. When there is a circular dependency, the pipeline generation system 10 can perform S1010 to cut off the circular dependency (remove the loop) to solve the topological sorting.

[0153] S1010, the pipeline generation system 10 cuts off the cyclic dependency, and obtains a loop-free attribute graph.

[0154] Specifically, the pipeline generation system 10 can identify the attribute column of the connected subgraph with cyclic dependency, determine whether the highest quality attribute column can be obtained from the knowledge base. If yes, the pipeline generation system 10 directly obtains the highest quality attribute column from the knowledge base, and the highest quality attribute column is the target attribute column. The pipeline generation system 10 can disconnect all incoming edges of the target attribute column to obtain a loop-free attribute graph. If not, the pipeline generation system 10 performs indirect data mode acquisition, analyzes the primary key attribute set, and determines the attribute column included in the main data as the highest quality attribute column, i.e., the target attribute column. The pipeline generation system 10 can disconnect all incoming edges of the target attribute column to obtain a loop-free attribute graph.

[0155] S1012, the pipeline generation system 10 can perform topological sorting according to the loop-free attribute graph, and based on the topological sorting result and the contribution degree of the data cleaning operator, solve the in-group sorting of the operator group corresponding to the attribute group.

[0156] The topological sorting can be implemented by DFS algorithm. When performing in-group sorting of the operator group, the pipeline generation system 10 can perform coarse sorting based on the topological sorting result, and the dependent data cleaning operator can be sorted in front. Then, the pipeline generation system 10 can perform fine sorting according to the contribution degree of the data cleaning operator, for example, the data cleaning operator with high contribution degree is sorted in front based on the topological sorting. As shown in FIG. 12, the pipeline generation system can sort OP1 in front of OP2 and OP3 in front of OP4 based on the topological sorting result and the contribution degree of the data cleaning operator.

[0157] Further, referring to FIG. 13, the pipeline generation system 10 can generate a pipeline with explainability through a pipeline explanation template according to the extracted semantic of the data cleaning operator and the generated pipeline. In this scenario, the pipeline with explainability is shown in FIG. 14, and each data cleaning operator in FIG. 14 corresponds to an explanation, and the dashed line represents the parallel relationship.

[0158] Based on the foregoing description, the pipeline generation method of the present application extracts fine-grained data cleaning operators upward to form a data cleaning pipeline, which can effectively express a data cleaning process, and each node contains a series of fine-grained cleaning operators with the same / similar functions, thereby helping users quickly locate an abnormality repaired by an operator and processing the abnormality. Moreover, the pipeline generation method of the present application can recommend an execution order of parallel operations based on parallel capabilities (such as parallelism), effectively improving execution efficiency. Based on attribute association relationships, a more accurate operator execution order can be obtained, improving repair accuracy. In addition, the pipeline generation method of the present application can generate an explanation of a data cleaning operator based on the semantics of the data cleaning operator, helping users understand the role of each node in the data cleaning pipeline.

[0159] It should be noted that the pipeline generation method of the present application is exemplarily described based on data cleaning, and in other possible implementation manners of the present application, the pipeline generation method can also be applied to the field of data development, the field of data preparation, for example, operators in the field of data development can be extracted upward to obtain attribute groups, thereby realizing coarse-grained pipeline generation.

[0160] Based on the foregoing pipeline generation method, the present application provides a pipeline generation system.

[0161] Referring to a structural schematic diagram of a pipeline generation system shown in FIG. 3, the pipeline generation system 10 comprises:

[0162] An attribute grouping subsystem 100 is configured to obtain a plurality of data cleaning operators for generating a pipeline and obtain attribute association relationships of data, wherein the pipeline is used to arrange the plurality of data cleaning operators so that the plurality of data cleaning operators are executed according to the arrangement result, and the data cleaning operator is used to remove or repair abnormal data in a data set;

[0163] The attribute grouping subsystem 100 is further configured to extract an attribute set of the plurality of data cleaning operators, and the attribute set comprises attributes of input data of the plurality of data cleaning operators and attributes of output data of the plurality of operators;

[0164] The attribute grouping subsystem 100 is further configured to group the attributes in the attribute set according to the attribute association relationships, obtain a plurality of attribute groups supporting parallel computing and an order of the plurality of attribute groups;

[0165] An operator arrangement subsystem 200 is configured to determine an intra-group order of an operator group corresponding to at least one attribute group in the plurality of attribute groups according to a topological order result of the at least one attribute group;

[0166] The operator arrangement subsystem 200 is further configured to arrange the plurality of data cleaning operators according to the order of the plurality of attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, and obtain the pipeline.

[0167] Exemplarily, the attribute grouping subsystem 100 and the operator arrangement subsystem 200 can be implemented by hardware, or can be implemented by software.

[0168] When implemented by software, the attribute grouping subsystem 100 and the operator arrangement subsystem 200 can be an application program running on a computing device. Taking the attribute grouping subsystem 100 as an example, the application program can be a computing engine. The application program can also be provided to users in the form of a virtualization service. The virtualization service can include a virtual machine (VM) service, a bare metal server (BMS) service, or a container service. The VM service can be a service that provides VMs for users to use on demand by virtualizing a VM resource pool on multiple physical hosts through a virtualization technology. The BMS service is a service that provides BMSs for users to use on demand by virtualizing a BMS resource pool on multiple physical hosts. The container service is a service that provides containers for users to use on demand by virtualizing a container resource pool on multiple physical hosts. A VM is a virtual computer, i.e., a logical computer. A BMS is a high-performance computing service that can be scaled elastically, and has the same computing performance as a traditional physical machine and the feature of secure physical isolation. A container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes, and resources. It should be understood that the VM service, the BMS service, and the container service in the above virtualization service are only specific examples. In actual applications, the virtualization service can also be other lightweight or heavyweight virtualization services, which are not limited here.

[0169] When implemented by hardware, the attribute grouping subsystem 100 and the operator arrangement subsystem 200 can include at least one computing device, such as a server or the like. Alternatively, the attribute grouping subsystem 100 and the operator arrangement subsystem 200 can also be devices implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0170] In some possible implementation manners, the operator arrangement subsystem 200 is further used for:

[0171] obtaining contribution degrees of the plurality of data cleaning operators, the contribution degree being used for indicating data repair capabilities of the plurality of data cleaning operators;

[0172] The operator arrangement subsystem 200 is specifically used for:

[0173] determining in-group ordering of an operator group corresponding to at least one attribute group according to the topological ordering result of the at least one attribute group and the contribution degrees of the plurality of data cleaning operators.

[0174] In some possible implementation manners, the operator arrangement subsystem 200 is further used for:

[0175] performing dependency detection on a connected subgraph corresponding to at least one attribute group in the plurality of attribute groups;

[0176] when there is a circular dependency, disconnecting an incoming edge of a target attribute column in the connected subgraph to obtain a loop-free attribute graph, the target attribute column being an attribute column in the connected subgraph whose quality meets a requirement;

[0177] performing topological ordering according to the loop-free attribute graph to obtain a topological ordering result of the at least one attribute group.

[0178] In some possible implementation manners, the operator arrangement subsystem is further used for:

[0179] determining the target attribute column from the connected subgraph according to qualities of attribute columns recorded in the knowledge base, the target attribute column including an attribute column with the highest quality; or

[0180] obtaining the target attribute column from the connected subgraph according to a primary key attribute set, the target attribute column being an attribute column included in primary data of the primary key attribute set.

[0181] In some possible implementation manners, the attribute grouping subsystem 100 is further used for:

[0182] recognizing semantics of the plurality of data cleaning operators;

[0183] The pipeline generation system 10 further includes:

[0184] an explanation generation subsystem 300, configured to generate an explanation of a data cleaning operator in the pipeline according to semantics of the plurality of data cleaning operators and the pipeline by using a pipeline explanation template.

[0185] Similar to the attribute grouping subsystem 100 and the operator arrangement subsystem 200, the explanation generation subsystem 300 can be implemented by using hardware, or can be implemented by using software.

[0186] When implemented by software, the explanation generation subsystem 300 can be an application program running on a computing device. The application program can also be provided to users in the form of a virtualization service, such as a VM service, a BMS service, or a container service. When implemented by hardware, the attribute grouping subsystem 100 and the operator orchestration subsystem 200 can include at least one computing device, such as a server or the like. Alternatively, the attribute grouping subsystem 100 and the operator orchestration subsystem 200 can also be devices implemented by ASICs or PLDs, and the like.

[0187] In some possible implementation manners, the attribute grouping subsystem 100 is specifically configured to:

[0188] construct a dependency directed graph corresponding to the attribute set according to the attribute association relationship and the attribute set, wherein a vertex in the dependency directed graph represents an attribute in the attribute set, and an edge in the dependency directed graph represents a dependency relationship between attributes in the attribute set;

[0189] perform connectivity analysis on the dependency directed graph to obtain a plurality of connected subgraphs;

[0190] group the attributes in the attribute set according to the plurality of connected subgraphs to obtain a plurality of attribute groups supporting parallel computing.

[0191] In some possible implementation manners, the attribute grouping subsystem 100 is specifically configured to:

[0192] divide attributes belonging to the same connected subgraph in the plurality of connected subgraphs into the same attribute group to obtain a plurality of attribute groups;

[0193] adjust at least one attribute group in the plurality of attribute groups according to the parallel degree and the number of operators corresponding to the plurality of connected subgraphs.

[0194] It should be noted that FIG. 3 is only a schematic division manner of the pipeline generation system 10, and the pipeline generation system 10 can also be divided into other modules or subsystems according to functions. For example, the above attribute grouping subsystem 100 can include an interaction module, an analysis module, and a grouping module. The interaction module is configured to obtain a plurality of data cleaning operators and an attribute association relationship, the analysis module is configured to extract attributes of the plurality of data cleaning operators to obtain an attribute set, and the grouping module is configured to group the attribute set according to the attribute association relationship to obtain a plurality of attribute groups supporting parallel computing and an order of the plurality of attribute groups.

[0195] The present application also provides a computing device 1500. As shown in FIG. 15, the computing device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate with each other through the bus 1502. The computing device 1500 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1500 is not limited.

[0196] The bus 1502 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 15, but it does not mean that there is only one bus or only one type of bus. The bus 1502 can include a path for transmitting information between various components (e.g., the memory 1506, the processor 1504, the communication interface 1508) of the computing device 1500.

[0197] The processor 1504 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0198] The memory 1506 can include a volatile memory (e.g., a random access memory (RAM)) and can also include a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0199] The memory 1506 stores executable program code which is executed by the processor 1504 to implement the aforementioned pipeline generation method. Specifically, the memory 1506 stores instructions for implementing the functions of the attribute grouping subsystem 100 and the operator arrangement subsystem 200 in the pipeline generation system 10. Further, the memory 1506 can also store instructions for implementing the functions of the explanation generation subsystem 100 in the pipeline generation system 10.

[0200] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card and a transceiver, to implement communication between the computing device 1500 and other devices or communication networks.

[0201] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0202] As shown in FIG. 16, the computing device cluster includes multiple computing devices 1500. The memory 1506 in the computing devices 1500 in the computing device cluster can store the same instructions for implementing the pipeline generation system 10 to execute the pipeline generation method.

[0203] In some possible implementations, one or more computing devices 1500 in the computing device cluster can also be used to execute part of the instructions for implementing the pipeline generation system 10 to execute the pipeline generation method. In other words, the combination of one or more computing devices 1500 can collectively execute the instructions for implementing the pipeline generation system 10 to execute the pipeline generation method.

[0204] In the case where the computing device cluster includes multiple computing devices 1500, the memories 1506 in different computing devices 1500 can store different instructions for implementing part of the functions of the pipeline generation system 10.

[0205] Figure 17 illustrates one possible implementation. As shown in Figure 17, two computing devices 1500A and 1500B are connected through communication interfaces 1508. The memory in computing device 1500A has instructions stored thereon for performing the functions of attribute grouping subsystem 100. The memory in computing device 1500B has instructions stored thereon for performing the functions of operator orchestration subsystem 200. In other words, the memories 1506 of computing devices 1500A and 1500B collectively store instructions for pipeline generation system 10 to perform the pipeline generation method. Further, computing device 1500B can also store instructions for the functions of explanation generation subsystem 300.

[0206] The manner in which the computing device cluster shown in Figure 17 is connected can be such that more computing power is needed to perform the operator ordering in view of the pipeline generation method provided herein. Therefore, it is contemplated that the functions implemented by operator orchestration subsystem 200 are performed by different computing devices.

[0207] It should be appreciated that the functions of computing device 1500A shown in Figure 17 can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0208] In some possible implementations, one or more of the computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. Figure 18 illustrates one possible implementation. As shown in Figure 18, two computing devices 1500C and 1500D are connected through a network. Specifically, the computing devices are connected to the network through communication interfaces in the respective computing devices. In this class of possible implementations, the memory 1506 in computing device 1500C has instructions stored thereon for performing the functions of attribute grouping subsystem 100. Meanwhile, the memory 1506 in computing device 1500D has instructions stored thereon for performing the functions of operator orchestration subsystem 200. Further, computing device 1500B can also store instructions for the functions of explanation generation subsystem 300.

[0209] It should be appreciated that the functions of computing device 1500C shown in Figure 18 can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500D can also be performed by multiple computing devices 1500.

[0210] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store instructions that can be executed by a computing device. The computer readable storage medium can be a magnetic-based medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical-based medium, (e.g., a compact disc, a DVD, etc.), or a semiconductor-based medium, (e.g., a solid state hard drive), etc. The computer readable storage medium includes instructions that are executable by a computing device to perform the above-described pipeline generation method applied to the pipeline generation system 10. The embodiments of the present application also provide another computer readable storage medium. The computer readable storage medium includes instructions that are executable by a computing device to perform the above-described pipeline generation method.

[0211] The embodiments of the present application also provide a computer program product including instructions. The computer program product can be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the above-described pipeline generation method. The embodiments of the present application also provide a computer program product including instructions. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the above-described pipeline generation method.

[0212] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of pipeline generation, the method comprising: The method comprises: obtaining a plurality of data cleaning operators for generating a pipeline, and obtaining an attribute association relationship of data, the pipeline being used for scheduling the plurality of data cleaning operators so that the plurality of data cleaning operators perform according to a scheduling result, the data cleaning operator being used for removing or repairing abnormal data in a data set; extracting an attribute set of the plurality of data cleaning operators, the attribute set comprising attributes of input data of the plurality of data cleaning operators and attributes of output data of the plurality of operators; grouping attributes in the attribute set according to the attribute association relationship, obtaining a plurality of attribute groups supporting parallel computing and an order of the plurality of attribute groups; determining an intra-group order of an operator group corresponding to at least one attribute group in the plurality of attribute groups according to a topological sorting result of the at least one attribute group; scheduling the plurality of data cleaning operators according to the order of the plurality of attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, and obtaining a pipeline.

2. The method of claim 1, wherein, The method further comprises: obtaining a contribution degree of the plurality of data cleaning operators, the contribution degree being used for indicating a data repairing capability of the plurality of data cleaning operators; The method further comprises: determining the intra-group order of the operator group corresponding to the at least one attribute group according to the topological sorting result of the at least one attribute group and the contribution degree of the plurality of data cleaning operators.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: performing dependency detection on a connected subgraph corresponding to at least one attribute group in the plurality of attribute groups; when there is a cyclic dependency, disconnecting an incoming edge of a target attribute column in the connected subgraph, obtaining an acyclic attribute graph, the target attribute column being an attribute column in the connected subgraph whose quality meets a requirement; performing topological sorting according to the acyclic attribute graph, and obtaining a topological sorting result of the at least one attribute group.

4. The method of claim 3, wherein, The method further comprises: determining the target attribute column from the connected subgraph according to qualities of attribute columns recorded in a knowledge base, the target attribute column comprising an attribute column with the highest quality; or obtaining the target attribute column from the connected subgraph according to a primary key attribute set, the target attribute column being an attribute column included in primary data in the primary key attribute set.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: identifying semantics of the plurality of data cleaning operators; generating an explanation of a data cleaning operator in the pipeline through a pipeline explanation template according to the semantics of the plurality of data cleaning operators and the pipeline.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: constructing a dependency directed graph corresponding to the attribute set according to the attribute association relationship and the attribute set, a vertex in the dependency directed graph representing an attribute in the attribute set, and an edge in the dependency directed graph representing a dependency relationship between attributes in the attribute set; performing connectivity analysis on the dependency directed graph, and obtaining a plurality of connected subgraphs; Grouping attributes in the attribute set according to the multiple connected sub-graphs, to obtain multiple attribute groups supporting parallel computing.

7. The method of claim 6, wherein, The grouping attributes in the attribute set according to the multiple connected sub-graphs, to obtain multiple attribute groups supporting parallel computing, comprises: Dividing attributes belonging to the same connected sub-graph in the multiple connected sub-graphs into the same attribute group, to obtain multiple attribute groups; According to the parallel degree and the number of operators corresponding to the multiple connected sub-graphs, adjusting at least one attribute group in the multiple attribute groups.

8. A pipeline generation system, characterized by, The system comprises: An attribute grouping subsystem is configured to obtain multiple data cleaning operators used for generating a pipeline, and obtain an attribute association relationship of data, the pipeline is configured to arrange the multiple data cleaning operators so that the multiple data cleaning operators perform according to the arrangement result, and the data cleaning operators are configured to remove or repair abnormal data in a data set; The attribute grouping subsystem is further configured to extract an attribute set of the multiple data cleaning operators, the attribute set comprising attributes of input data of the multiple data cleaning operators and attributes of output data of the multiple operators; The attribute grouping subsystem is further configured to group attributes in the attribute set according to the attribute association relationship, to obtain multiple attribute groups supporting parallel computing and an order of the multiple attribute groups; An operator arrangement subsystem is configured to determine an intra-group order of an operator group corresponding to at least one attribute group in the multiple attribute groups according to a topological order result of the at least one attribute group; The operator arrangement subsystem is further configured to arrange the multiple data cleaning operators according to the order of the multiple attribute groups and the intra-group order of the operator group corresponding to the at least one attribute group, to obtain the pipeline.

9. The system of claim 8, wherein, The operator arrangement subsystem is further configured to: Obtain contribution degrees of the multiple data cleaning operators, the contribution degrees being used to indicate data repair capabilities of the multiple data cleaning operators; The operator arrangement subsystem is specifically configured to: Determine the intra-group order of the operator group corresponding to the at least one attribute group according to the topological order result of the at least one attribute group in the multiple attribute groups and the contribution degrees of the multiple data cleaning operators.

10. The system of claim 8 or 9, characterized in that, The operator arrangement subsystem is further configured to: Perform dependency detection on a connected sub-graph corresponding to at least one attribute group in the multiple attribute groups; When there is a cyclic dependency, disconnect an incoming edge of a target attribute column in the connected sub-graph, to obtain an acyclic attribute graph, the target attribute column being an attribute column in the connected sub-graph whose quality meets a requirement; Perform topological ordering according to the acyclic attribute graph, to obtain a topological order result of the at least one attribute group.

11. The system of claim 10, wherein, The operator arrangement subsystem is further configured to: Determine the target attribute column from the connected sub-graph according to qualities of attribute columns recorded in a knowledge base, the target attribute column comprising an attribute column with the highest quality; or Obtain the target attribute column from the connected sub-graph according to a primary key attribute set, the target attribute column being an attribute column included in primary data in the primary key attribute set.

12. The system according to any one of claims 8 to 10, characterized in that, The attribute grouping subsystem is further configured to: Identify semantics of the multiple data cleaning operators; The system further comprises: An explanation generation subsystem configured to generate, based on semantics of the plurality of data cleaning operators and the pipeline, an explanation of the data cleaning operators in the pipeline by a pipeline explanation template.

13. The system of any one of claims 8 to 12, wherein, The attribute grouping subsystem is specifically configured to: construct a dependency directed graph corresponding to the attribute set according to the attribute association relationship and the attribute set, wherein a vertex in the dependency directed graph represents an attribute in the attribute set, and an edge in the dependency directed graph represents a dependency relationship between attributes in the attribute set; perform connectivity analysis on the dependency directed graph to obtain a plurality of connected subgraphs; group attributes in the attribute set according to the plurality of connected subgraphs to obtain a plurality of attribute groups that support parallel computing.

14. The system of claim 13, wherein, The attribute grouping subsystem is specifically configured to: divide attributes belonging to the same connected subgraph in the plurality of connected subgraphs into the same attribute group to obtain a plurality of attribute groups; adjust at least one attribute group in the plurality of attribute groups according to a parallel degree and a number of operators corresponding to the plurality of connected subgraphs.

15. A cluster of computing devices, characterized in that, The computing device cluster includes at least one computing device, the at least one computing device including at least one processor and at least one memory, the at least one memory storing computer readable instructions; the at least one processor executes the computer readable instructions, so that the computing device cluster executes the method in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, including computer readable instructions; the computer readable instructions are used to implement the method of any one of claims 1 to 7.

17. A computer program product, characterised in that, including computer readable instructions; the computer readable instructions are used to implement the method of any one of claims 1 to 7.

Images