Data pipeline construction method and system based on operator combination and medium
By constructing a data pipeline based on operator combinations, task and domain information is obtained, suitable operators are selected, the data processing flow is planned, and evaluation operators are inserted. This solves the problem of migrating the data processing system between different tasks and domains, and realizes dynamic optimization of data quality and improvement of model performance.
Patent Information
- Application Number
- CN202511425659.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-01-13
AI Technical Summary
Existing data preparation and processing systems lack versatility and are difficult to migrate between different tasks and domains. This results in a lack of effective feedback in the data generation process, making dynamic optimization difficult and severely restricting the effectiveness and scalability of models in specific domains.
By using a data pipeline construction method based on operator combination, task information is obtained, target information and domain-specific information are parsed, candidate operators are selected, data processing flow is planned by combination, evaluation operators are inserted in key steps, data pipeline is generated, and dynamic optimization is performed using an intelligent agent module.
It improved the adaptability and quality of data processing, enhanced the scalability of data pipelines, reduced the cost of manual intervention, and improved model performance and data preparation efficiency.
Smart Images
Figure CN121326342A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data pipeline technology, and for example to a data pipeline construction method, system and medium based on operator combination. Background Technology
[0002] With the rapid development of Large Language Models (LLM), their applications in fields such as healthcare, finance, and law are becoming increasingly widespread. However, model performance is highly dependent on the quality and suitability of the data. Existing data preparation and processing systems typically rely on fixed, manually designed processes, offering limited functionality and lacking versatility, making them difficult to migrate across different tasks and domains. Furthermore, data processing and evaluation are often conducted in isolation, resulting in a lack of effective feedback during data generation and hindering dynamic optimization. These issues severely restrict the effectiveness and scalability of models in specific domains.
[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.
[0005] This disclosure provides a method, system, and medium for constructing data pipelines based on operator combinations. It can generate corresponding data pipelines based on different tasks and domains, facilitating migration between different tasks and domains. The obtained data pipelines can also be evaluated during data processing to effectively provide feedback on the data processing status, thereby enabling dynamic optimization of the data pipelines.
[0006] In some embodiments, a data pipeline construction method based on operator combination includes: acquiring task information; parsing the task information to obtain task target information and specific domain information; selecting candidate operators from an operator library based on the target information and specific domain information; combining and planning the order of the candidate operators to determine the target path from input to output and generate an initial data processing flow for the data pipeline; inserting evaluation operators at one or more evaluation positions in the initial data processing flow to obtain a final data processing flow; and generating a data pipeline based on the final data processing flow.
[0007] In some embodiments, a data pipeline construction system based on operator combination includes an intelligent agent module configured to: acquire task information; parse the task information to obtain target information and domain-specific information of the task; select candidate operators from an operator library based on the target information and domain-specific information; perform combination planning on the order of the candidate operators to determine the target path from input to output and generate an initial data processing flow for the data pipeline; insert evaluation operators at one or more evaluation positions in the initial data processing flow to obtain a final data processing flow; and generate a data pipeline based on the final data processing flow.
[0008] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by a processor, perform the steps of a data pipeline construction method as described in any embodiment of this application.
[0009] The data pipeline construction method, system, and medium based on operator combination provided in this disclosure, based on task information parsing, obtain the task execution target and the domain involved in the task. This allows for better selection of operators suitable for the current task and domain, ensuring data processing aligns with the task and domain, and improving the scalability of the data pipeline. By planning the combination order of candidate operators and the path planning, the logical correctness of the data processing flow can be guaranteed. By inserting evaluation operators into key steps in the data processing flow, the results of key steps or the final result can be evaluated to obtain the status of the data processing process in real time, enabling dynamic optimization of the data pipeline. This is beneficial for improving the quality and adaptability of the generated data, thereby enhancing model performance. The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description
[0010] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 A flowchart illustrating a data pipeline construction method based on operator combination according to an embodiment of the present disclosure is shown; Figure 2 A structural diagram of a data pipeline construction system based on operator combination according to an embodiment of the present disclosure is shown. Detailed Implementation
[0011] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures, steps, and apparatus may be simplified to simplify the illustrations.
[0012] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.
[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The term "multiple" means two or more. In embodiments of this disclosure, the character " / " indicates an "or" relationship between the preceding and following objects. For example, A / B means: A or B. The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.
[0014] It should be noted that, unless otherwise specified, the embodiments and features described in the present disclosure can be combined with each other.
[0015] Those skilled in the art will understand that in the methods described in this application and other parts thereof, for example, in the methods of various embodiments, examples, or claims, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The detailed execution order of each step should be determined by its function and possible internal logic. Unless otherwise specified, all steps of this application may be performed sequentially or randomly, preferably sequentially. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C can be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.
[0016] In this application, open-ended technical features or solutions described using terms such as "containing," "including," or "comprising" do not exclude additional members beyond those listed unless otherwise specified. They can be considered as providing both closed-ended features or solutions comprised of the listed members and open-ended features or solutions that include additional members beyond the listed members. For example, if A includes a1, a2, and a3, it may also include other members or exclude additional members unless otherwise specified. This can be considered as providing both the feature or solution that "A consists of a1, a2, and a3" and the feature or solution that "A includes not only a1, a2, and a3, but also other members."
[0017] The "range" disclosed in this application can be defined in the form of a lower limit and an upper limit. A given range is defined by selecting a lower limit and an upper limit, which define the boundaries of the specific range. Ranges defined in this way can include or exclude endpoints. Any endpoint can be independently included or excluded, and they can be arbitrarily combined; that is, any lower limit can be combined with any upper limit to form a range. For example, if ranges of 2000–10000 rpm and 3000–10000 rpm are listed for a specific parameter, it is also expected that ranges of 2000–10000 rpm and 3000–10000 rpm are also included. Furthermore, if minimum range values 1 and 2 are listed, and if maximum range values 3, 4, and 5 are also listed, then the following ranges are all expected: 1–3, 1–4, 1–5, 2–3, 2–4, and 2–5. In this application, unless otherwise stated, the numerical range "a~b" represents a shortened representation of any combination of real numbers between a and b, where a and b are both real numbers. For example, the numerical range "0~5" means that all real numbers between "0~5" have been listed in this document, and "0~5" is simply a shortened representation of these numerical combinations. Furthermore, when a parameter is expressed as an integer ≥2, it is equivalent to listing the parameter as, for example, integers 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, etc. For instance, when a parameter is expressed as an integer selected from "2~10", it is equivalent to listing integers 2, 3, 4, 5, 6, 7, 8, 9, and 10.
[0018] In this application, unless otherwise specified, A (e.g., B) means that B is a non-limiting example of A, and it is understood that A is not limited to B.
[0019] This disclosure provides a data pipeline construction method based on operator combinations. The data pipeline construction method includes: in step 101, obtaining task information. The task information can be a task input by a customer, such as "building a question-and-answer dataset for a financial knowledge base," "building a medical question-and-answer dataset," or "cleaning financial knowledge base data." The task information can be text information or audio information, etc.
[0020] In step 102, the task information is parsed to obtain the task's target information and specific domain information; candidate operators are selected from the operator library based on the target information and specific domain information.
[0021] The target information of a task can be extracted from a natural language processing model. For example, in the task of "building a question-and-answer dataset for a financial knowledge base," the target could be generating a question-and-answer dataset within the financial domain. Similarly, in the task of "building a medical question-and-answer dataset," the target could be generating a question-and-answer dataset within the medical domain.
[0022] Semantic vectors can be used to represent relevant information about processing operators, and then suitable candidate operators can be selected through comparison. Taking the task of "building a medical question-answering dataset" as an example, the first step is to identify the target as "generating question-answer pairs," within the medical field. Document parsing operators specific to the medical field are selected from the operator library, along with question generation and answer generation operators. Document parsing operators are used to extract key information from case studies or medical literature, question generation operators generate questions in natural language, and answer generation operators generate corresponding answers. This approach selects operators that align with the task's objective and meet specific domain requirements. For example, in the task of knowledge base cleaning, combinations of entity recognition, redundancy detection, compliance verification, and integrity assessment might be chosen.
[0023] In step 103, the order of candidate operators is combined and planned to determine the target path from input to output, generating the initial data processing flow of the data pipeline. Operators are selected and combined to form the nodes of the data pipeline, and the order of multiple candidate operators is planned to find the target path from input to output, thereby ensuring the correctness of the pipeline logic and the completeness of functional coverage. Taking the task of "building a medical question-answering dataset" as an example, the order of operators includes selecting document parsing operators - question generation operators - answer generation operators.
[0024] In step 104, evaluation operators are inserted at one or more evaluation locations in the initial data processing flow to obtain the final data processing flow. Evaluation locations can be between two adjacent processing nodes or after the last processing node, with each node corresponding to one operator position. This allows for the evaluation of the output results of key nodes and / or the final result, ensuring that data quality assessment and data processing are completed within the same framework. The evaluation results can be used for subsequent data pipeline optimization, etc. Taking the task of "building a medical question-and-answer dataset" as an example, accuracy and consistency evaluation operators can be inserted after the final answer generation operator to perform quality checks on the generated results. This can assess whether the answer conforms to medical common sense or contains logical inconsistencies.
[0025] In step 105, a data pipeline is generated based on the final data processing flow.
[0026] This application's modular operator system provides a rich set of functional building blocks, while the intelligent agent-driven mechanism ensures efficient mapping from task objectives to pipeline execution. The embedded evaluation mechanism enables dynamic detection and feedback optimization of result quality. Compared to existing manual configuration or fixed-process systems, this application significantly reduces the cost of manual intervention, improves the efficiency and quality of data preparation, and provides strong support for the practical application of large-scale language models in various fields.
[0027] The data pipeline construction method of this application can be executed by the intelligent agent module.
[0028] In some embodiments, the annotation information of the operators in the operator library includes at least functional description information and scope of application information, and may also include input and output format and / or dependency information. Based on target information and specific domain information, corresponding operators can be selected through functional information comparison, scope of application comparison, etc. This enables the selection of operators that are more suitable for the task and domain.
[0029] Optionally, operators are divided into three main categories, and can also be further categorized based on specific functions. The first three categories are: general-purpose operators, covering the most common data processing needs, such as text cleaning, word segmentation and tagging, information extraction, format conversion, and text generation. These general-purpose operators often do not rely on specific domain knowledge and can be reused in most tasks. The second category is domain-specific operators, geared towards professional fields such as healthcare, finance, and law. These operators can incorporate industry-specific rules or call dedicated models, such as medical entity recognition, financial compliance checks, and legal clause parsing, thereby ensuring the professionalism and compliance of data processing. The third category is evaluation operators, used to check data quality at intermediate or final stages of the data pipeline. These operators support multi-dimensional evaluation, including metrics such as accuracy, consistency, completeness, coverage, and fluency, and can be implemented by rule engines or machine learning models. Any new functionality can be seamlessly integrated into the system by encapsulating it as a new operator, making the operator library highly scalable.
[0030] Optionally, the various operators in the operator library can have a unified interface standard. This allows for diverse implementations of rule-based, statistical model, deep neural network, or large language model APIs, ensuring system scalability.
[0031] In some embodiments, task nodes to be evaluated are selected based on task information, and then corresponding evaluation operators are selected based on the information of the task nodes. Task nodes to be evaluated can be selected based on importance, etc. Evaluation operators also include functional description information and scope of application information. Evaluation operators are first matched, and then the intelligent agent module inserts the evaluation operators into the corresponding task node positions. For example, a fluency and consistency detection module is inserted after the text generation step, and an integrity and redundancy detection module is inserted after the data cleaning step. Inserting evaluation operators in key steps enables real-time detection of intermediate results and measures quality across different dimensions.
[0032] In some embodiments, the process of combining candidate operators in sequence to determine the target path from input to output includes: treating candidate operators as graph nodes and input-output relationships as edges, using a graph search method to combine candidate operators to obtain at least one feasible path from input to output, and determining the target path from input to output based on at least one feasible path. Optionally, the search algorithm can be A* search, Beam Search, or Monte Carlo tree search, etc. The search algorithm can find logically feasible paths, thus completing the entire data processing process of the data pipeline. There may be multiple feasible paths.
[0033] Graph search methods can include heuristic graph search methods, which can specifically include: 1) Candidate retrieval first filters out the set of operators related to the task through semantic matching, forming a sparse search graph.
[0034] 2) Heuristic Search Setting status: The currently constructed sequence of operators and their output format.
[0035] The heuristic function h(n) estimates the minimum remaining cost from the current node to the target node, which can be obtained based on statistics such as differences in output format, functional gaps, and historical execution time.
[0036] Use A* or Beam Search to expand nodes in the state space, prioritizing the exploration of branches that minimize the estimated total cost g(n) + h(n).
[0037] Optionally, the generated data pipeline can be stored using a graph structure, where each node corresponds to an operator instance and edges represent the transfer of data between operators.
[0038] In some embodiments, determining the target path from input to output based on at least one feasible path includes: parsing the task information to obtain the global objective of the task; evaluating at least one feasible path from input to output based on the global objective to obtain path evaluation results; and selecting the target path from input to output based on the path evaluation results.
[0039] When building a data processing pipeline, the intelligent agent not only finds a feasible sequence of operators that can smoothly transform the input data into the target output, but also finds the optimal path among all feasible paths. The "optimal" is defined in relation to the overall task objective (such as the highest data quality, the lowest operating cost, the shortest latency, or a weighted combination of multiple objectives).
[0040] The path evaluation results are obtained by evaluating at least one feasible path from input to output based on the global objective. This includes setting a global objective function for each feasible path, with the weights determined by the global objective described in the task description (or given by the strategy configuration). There can be multiple global objectives, which can support linear or nonlinear combinations of multiple objectives. The path evaluation results are obtained by scoring each feasible path through the global objective function.
[0041] A higher evaluation score for a feasible path indicates that it better meets the overall requirements of the task. The global objectives for setting weights include the following: Data quality: the accuracy, completeness, and noise level of the results. Higher quality results in a higher score.
[0042] Operating costs: Costs of computing resources, storage, bandwidth, etc. Lower costs result in higher scores.
[0043] Execution speed: How long does it take from start to finish? The faster the better.
[0044] Stability and reliability: failure rate, fault tolerance. The more reliable, the better.
[0045] Weights are assigned to these aspects in advance. When scoring a candidate feasible path, the system first calculates the score for each aspect separately, and then adds them up according to their weights to form a total score.
[0046] The process of evaluating feasible paths and selecting the optimal target path results in a data pipeline with better performance, such as lower power consumption and faster execution speed.
[0047] In some embodiments, the data pipeline construction method further includes: obtaining the execution evaluation results of the data pipeline; feeding back the execution evaluation results to the learning module, training the intelligent agent module, and optimizing the operator selection strategy of the intelligent agent module.
[0048] For example, a fluency and consistency detection module can be inserted after the text generation step to evaluate fluency and consistency metrics. Fluency can be evaluated using quantitative indicators such as grammatical accuracy and semantic coherence, resulting in an evaluation score. Similarly, an integrity and redundancy detection module can be inserted after the data cleaning step to evaluate integrity and redundancy.
[0049] Each data processing step can be evaluated. After multiple rounds of iterative training, the intelligent agent module can gradually form a stable and efficient pipeline generation strategy. The intelligent agent module can record the training results, operator selection strategies, operator combination strategies, path planning strategies, etc., corresponding to various task types, so as to achieve rapid response to different task types and optimal combination of operators.
[0050] Optionally, the evaluation scores can also be fed back to the client, allowing users to understand the processing status of the data pipeline in a timely manner.
[0051] Training ensures that the intelligent agent module has the ability to learn and adapt, enabling the data pipeline execution performance to continuously improve over long-term operation.
[0052] Optionally, the path evaluation results are fed back to the learning module to train the intelligent agent module and optimize its operator path planning strategy. Through reinforcement learning or feedback from historical tasks, the weights of the global objective function and the heuristic parameters of the heuristic function are dynamically adjusted to make the search more closely approximate the optimal solution of the real task.
[0053] Optionally, if the execution evaluation result of the first node of the target path does not meet the requirements, the output result of the first node can be corrected.
[0054] If the evaluation result is expressed as an evaluation score, it can be that the evaluation score is below a threshold to indicate that the requirements are not met. The first node can be one or more nodes in the target path, that is, if the output result after running one or more operators does not meet the requirements, the output result is corrected.
[0055] Optionally, the correction method includes: the error classification module diagnosing the error type based on the quality problem or low confidence output of the evaluation operator; the decision module selecting an appropriate correction strategy based on the error type, context, cost budget, and risk level; and the correction module executing specific correction operations according to the correction strategy.
[0056] The basic correction process is as follows: The evaluation operator detects quality issues or low-confidence outputs; the error classification module diagnoses the problem type (e.g., format errors, missing / incomplete information, semantic contradictions, factual errors, compliance violations, duplicate data, etc.); the decision-making module selects an appropriate correction strategy (automatic correction, reprocessing, multi-path optimization, or manual intervention) based on the error type, context, cost budget, and risk level; and the specific correction operation is executed (replacement, parameter adjustment, data supplementation, backtracking and rerunning, etc.).
[0057] Optionally, the evaluation operator performs an evaluation operation based on the corrected result to obtain a corrected evaluation score; this corrected evaluation score is then sent to the intelligent agent module. The evaluation module scores again and feeds the result back to the intelligent agent as a reward signal for subsequent policy updates, until the result meets a preset threshold or triggers a manual review condition. This optimization can improve the accuracy of the data results generated after the corresponding node runs.
[0058] Example of medical question-and-answer correction: After answer generation, the accuracy evaluator finds several answers that conflict with medical common sense (low confidence). The agent triggers the retrieval module to pull authoritative literature and clinical guidelines, using the evidence and the original question as context to regenerate the answer; multiple candidates are generated in parallel, and the fact checker compares the consistency of the evidence, finally selecting the candidate that best matches the evidence. If the answer is still below the threshold, the output is entered into the human physician review queue and marked as "requires verification".
[0059] If the data results deviate from the expected range, corrections can be made to improve their accuracy. Ensuring the overall accuracy of the generated results improves the dataset's accuracy rate and overall quality.
[0060] In some embodiments, the execution evaluation result output by the evaluation operator during the data pipeline execution is obtained; if the execution evaluation result indicates that the current execution result does not meet the requirements, the data pipeline is re-executed and / or the target path of input and output is adjusted. This could be the execution evaluation result of a critical step or the final execution result. If the deviation between the execution result and the correct result is large, it indicates that there may be a problem with the path. Timely adjustment of the path can update the data results, making them more aligned with actual task requirements and the domain.
[0061] Optionally, triggering the data pipeline to adjust the target path of input and output includes: sorting at least one feasible path according to the path evaluation result of at least one feasible path, and selecting feasible paths as the adjusted target paths in descending order of path evaluation results.
[0062] Path evaluation results can be presented as evaluation scores, which can be used to rank feasible paths in the data pipeline. These scores can be logged. If the evaluation score falls below a threshold (potentially sent as an error report to the intelligent agent module), the data pipeline is triggered to re-process. This allows for dynamic adjustments to the data processing based on real-time evaluation data, creating a dynamically adjustable operating mechanism. If the evaluation score at a critical step falls below the threshold, the data pipeline execution can be paused, the path adjusted, and then resumed, reducing energy consumption and allowing for timely path adjustments to achieve more optimized results.
[0063] Taking the task of "building a medical question-and-answer dataset" as an example, if the accuracy and consistency evaluation operator performs quality checks on the generated results, and finds that some answers do not conform to medical common sense or have logical conflicts, an error report can be issued.
[0064] Taking "cleaning up financial knowledge base data" as an example, the intelligent agent prioritizes calling compliance checks, redundancy detection, and data standardization operators in the financial field, while inserting integrity assessment modules in key steps to ensure that the output results meet regulatory requirements and have a clear structure.
[0065] This application can integrate an evaluation and optimization framework, enabling closed-loop operation of task parsing, pipeline generation, data processing, and result evaluation without human intervention.
[0066] In some embodiments, a unified representation format for intermediate data during the execution of the data pipeline is predetermined. This ensures interoperability at different operator locations. For example, JSON structured data is used, containing fields such as content, tags, and metadata. This design allows operators implemented with different algorithms to seamlessly connect within the same framework, ensuring high system flexibility and compatibility.
[0067] In some embodiments, the data pipeline construction method further includes: parsing the task information to obtain the input data type and the expected output format; and selecting candidate operators from an operator library based on the target information, specific domain information, the input data type, and the expected output. The input data type (such as documents, structured data, dialogue corpora, etc.) and the expected output format (such as question-answer pairs, summary text, structured tables, etc.) can also be used as the basis for operator selection.
[0068] This disclosure provides a data pipeline construction system that integrates operator combinations. For example... Figure 2As shown, the data pipeline construction system 200 includes an intelligent agent module 201 and an operator library 202. The intelligent agent module 201 is configured to: acquire task information; parse the task information to obtain the task's target information and specific domain information; select candidate operators from the operator library 202 based on the target information and specific domain information; perform combination planning on the order of the candidate operators to determine the target path from input to output and generate the initial data processing flow of the data pipeline; insert evaluation operators at one or more evaluation positions in the initial data processing flow to obtain the final data processing flow; and generate the data pipeline based on the final data processing flow.
[0069] This disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by a processor, perform the steps of a data pipeline construction method as described in any embodiment of this application.
[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0071] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for constructing a data pipeline based on operator combinations, characterized in that, The data pipeline construction method includes: Obtain task information; Based on the task information, the target information and specific domain information of the task are parsed to obtain the target information and specific domain information of the task; candidate operators are selected from the operator library based on the target information and specific domain information. The order of candidate operators is combined and planned to determine the target path from input to output, and the initial data processing flow of the data pipeline is generated. An evaluation operator is inserted at one or more evaluation locations in the initial data processing flow to obtain the final data processing flow; A data pipeline is generated based on the final data processing flow.
2. The data pipeline construction method according to claim 1, characterized in that, Combining candidate operators in order to determine the target path from input to output, including: Candidate operators are treated as graph nodes, and input-output relationships are treated as edges. A graph search method is used to combine and plan the candidate operators to obtain at least one feasible path from input to output. Based on at least one feasible path, the target path from input to output is determined.
3. The data pipeline construction method according to claim 2, characterized in that, Determining the target path from input to output based on at least one feasible path includes: parsing the task information to obtain the global objective of the task; evaluating at least one feasible path from input to output based on the global objective to obtain path evaluation results; and selecting the target path from input to output based on the path evaluation results.
4. The data pipeline construction method according to claim 1, characterized in that, Obtain the execution evaluation result output by the evaluation operator during the execution of the data pipeline; if the execution evaluation result indicates that the current execution result does not meet the requirements, trigger the re-execution of the data pipeline and / or adjust the target path of the input and output.
5. The data pipeline construction method according to claim 4, characterized in that, If the execution evaluation result of the first node in the target path does not meet the requirements, the output result of the first node is corrected.
6. The data pipeline construction method according to claim 1, characterized in that, Obtain the execution evaluation results of the data pipeline; feed the execution evaluation results back to the learning module to train the intelligent agent module and optimize the operator selection strategy of the intelligent agent module.
7. The data pipeline construction method according to claim 1, characterized in that, Predetermine the representation format of intermediate data during the execution of a unified data pipeline.
8. The data pipeline construction method according to claim 1, characterized in that, Select the task nodes that need to be evaluated based on the task information, and then select the corresponding evaluation operators based on the information of the task nodes.
9. A data pipeline construction system that integrates operator combinations, characterized in that, The data pipeline construction system includes an intelligent agent module, which is configured as follows: Obtain task information; Based on the task information, the target information and specific domain information of the task are parsed to obtain the target information and specific domain information of the task; candidate operators are selected from the operator library based on the target information and specific domain information. The order of candidate operators is combined and planned to determine the target path from input to output, and the initial data processing flow of the data pipeline is generated. An evaluation operator is inserted at one or more evaluation locations in the initial data processing flow to obtain the final data processing flow; A data pipeline is generated based on the final data processing flow.
10. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, perform the steps of the data pipeline construction method as claimed in any one of claims 1-8.