Automated interactive large-scale language model pipeline orchestration method, system, device and storage medium for complex queries

Abstract

Disclosed are a method, system, device, and storage medium for orchestrating an automated, interactive, large-scale language model pipeline for complex queries. These methods are designed to address the difficulty of existing technologies in efficiently and accurately processing complex queries in data lakes that require multi-hop semantic retrieval, multi-step logical reasoning, and multi-stage semantic analysis. Based on a predefined set of operators, the method extracts relevant operators from queries using a large language model, generates an execution pipeline, rewrites the chained pipeline into a DAG structure to support parallel execution, uses a cost model to select the optimal execution path, and monitors intermediate results in real time during execution, dynamically adjusting the execution pipeline. The solution proposed in this disclosure significantly improves the accuracy and efficiency of complex query processing in data lakes.

Description

Technical Field

[0001] The present disclosure relates to the field of information retrieval technology, and in particular to a method, system, device, and storage medium for automated interactive large-scale language model pipeline orchestration for complex queries. Background Art

[0002] With the advent of the big data era, data lakes, as a solution for centrally storing large amounts of raw data, have become a core component of enterprise data management. Capable of storing structured, semi-structured, and unstructured data, data lakes provide a rich data resource for data analysis and business intelligence. However, despite their advantages in data storage, data lakes face a number of challenges in data analysis.

[0003] First, one of the data analysis challenges in data lakes is insufficient processing capabilities for complex queries. Existing data lake technologies primarily provide basic access operations for unstructured data and analytical queries for structured data. However, they perform poorly when handling complex queries that require multi-hop semantic retrieval and linking across data types, multi-step logical reasoning, and multi-stage semantic analysis. These queries often require a deep understanding of data content, complex logical reasoning, and the ability to convert and integrate data between different formats. These requirements far exceed the capabilities of traditional structured query languages ​​(such as SQL). Therefore, even advanced natural language to SQL (NL2SQL) methods cannot meet these requirements.

[0004] Secondly, the emergence of large language models (LLMs) has brought new opportunities for data search and analysis. With their powerful semantic understanding and reasoning capabilities, they can theoretically handle complex data analysis tasks. However, large language models have also encountered challenges in practical applications, especially when processing complex queries that require complex task decomposition, pipeline orchestration, pipeline optimization, interactive execution, and self-reflection. These tasks are beyond the current reasoning capabilities of large language models because they require complex multi-step reasoning or "thinking chaining" processes and in-depth understanding of the data in the data lake, which large language models themselves lack.

[0005] Furthermore, as described in the official documentation for current large-scale language model execution frameworks, such as Llamaindex and LangChain, existing large-scale language model application methods rely on static, manually orchestrated execution pipelines when processing complex queries. While these pipelines can decompose queries into subtasks and combine pre-programmed steps, retrieval steps, and prompt-based subtasks to obtain accurate answers, they have significant limitations. Manually orchestrated pipelines rely on user expertise and are often complex, involving hundreds of steps, which increases labor costs. Furthermore, these pipelines are static and cannot dynamically adjust to address failures in intermediate operations. For example, if a retrieval step produces irrelevant results, subsequent steps need to be adjusted accordingly; otherwise, the final answer may be incorrect based on irrelevant information. Furthermore, while traditional retrieval-augmented generation (RAG) methods can improve information retrieval accuracy, they still suffer from insufficient accuracy when processing queries that require multi-step reasoning and complex logical analysis. This is because these methods still rely on pre-set retrieval paths and fixed execution processes, making them unable to adapt to the changes and uncertainties that may arise during the query process. Furthermore, these methods cannot effectively support aggregate analysis of large amounts of data, beyond point queries.

[0006] Existing approaches are also limited in their functionality. Human-designed pipelines are often tailored to specific queries and cannot adapt to more complex queries. As a result, many queries are not covered by existing pipelines, requiring only relatively basic pipelines to be used, resulting in reduced accuracy.

[0007] To sum up, there is an urgent need for a technical solution to efficiently process complex queries in data lakes. Summary of the Invention

[0008] The purpose of this disclosure is to provide an automated interactive large-scale language model pipeline orchestration solution for complex queries. By combining a large language model, a predefined set of operators, and a cost model, the optimal execution pipeline is automatically generated and dynamically optimized and adjusted based on the actual execution situation, enabling efficient processing of complex queries in data lakes.

[0009] According to one embodiment of the present disclosure, an automated interactive large-scale language model pipeline orchestration method for complex queries is proposed, including:

[0010] receiving a query input by a user in natural language;

[0011] Extracting operators from a predefined set of operators using a large language model to construct multiple chained candidate pipelines with different reasoning paths corresponding to the query;

[0012] rewriting each chain candidate pipeline into a directed acyclic graph (DAG) structure, and identifying parallel opportunities during the rewriting process to write the corresponding operators into a parallel structure;

[0013] Use the cost model to estimate the cost of each operator in the DAG structure and select at least one DAG candidate pipeline with the lowest overall cost as the optimal execution pipeline;

[0014] executing the selected at least one optimal execution pipeline, and monitoring intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline;

[0015] The outputs of each execution pipeline are integrated and the integrated result is returned to the user as the final query result.

[0016] In some embodiments, the predefined set of operators includes some or all of the following operators: search, scan, filter, sort, summarize, generate, refine, classify, translate, transform, evaluate, interpret, integrate, conceptualize, extract, plan, link, aggregate, validate, group, compare, and cluster.

[0017] In some embodiments, the cost model is used to estimate the cost of each operator based on the size of the input data and the cost function of the operator, wherein, when estimating the cost of the operator, the cardinality estimate is used as the size of the input data, and the cardinality estimate approximates the selectivity through a random sampling method.

[0018] In some embodiments, the method further comprises:

[0019] The parameters of the cost functions of the operators in the cost model are adjusted according to the sampled workload.

[0020] In some implementations, real-time monitoring of intermediate results and execution status to dynamically and adaptively adjust the execution pipeline includes:

[0021] Real-time monitoring of intermediate results and current execution status generated during the execution process;

[0022] evaluating the execution quality of the currently executed pipeline based on the intermediate results and the execution status;

[0023] If the execution quality does not meet the expected quality requirement, stopping the execution of the currently executed pipeline, regenerating at least one reference execution pipeline based on the collected intermediate results and using the large language model, and determining an adjusted execution pipeline from the at least one reference execution pipeline using the cost model.

[0024] In some implementations, during execution, a layer-by-layer execution strategy is adopted to execute the pipeline of the DAG structure.

[0025] In some embodiments, the method further comprises employing the following prefetching technique during execution:

[0026] When the corresponding search operator is executed, a primary search and multiple backup searches are generated, where different backup searches correspond to different search scopes that are directly or indirectly related to the query intent;

[0027] The plurality of backup searches are performed in parallel while processing the results returned in response to the primary search, and the results returned in response to the backup searches are received and stored.

[0028] In some embodiments, integrating the outputs of various execution pipelines includes:

[0029] After each execution pipeline is completed, the output of each execution pipeline is explained using a large language model;

[0030] Analyze and merge the interpretation results of the outputs of each execution pipeline, and use the merged results as the integrated results.

[0031] In some embodiments, the method further comprises performing feedback optimization using a reward model:

[0032] Evaluate the relevance and impact of each operator based on its output and execution status during execution;

[0033] Rewards are distributed based on the relevance and impact of the evaluation, and the reward information is used to fine-tune a large language model that extracts relevant operators from the query to enhance the chain thinking ability of the large language model.

[0034] According to one embodiment of the present disclosure, an automated interactive large-scale language model pipeline orchestration system for complex queries is proposed, characterized in that the system is used to implement the above-mentioned method, and the system includes:

[0035] A query interface, used to receive queries entered by users in natural language and identify the user's query intent;

[0036] Operator set module, used to store predefined operator sets;

[0037] A pipeline builder, configured to extract operators from an operator set using a large language model and construct multiple chained candidate pipelines with different reasoning paths corresponding to the query;

[0038] a pipeline rewriter for rewriting the chain candidate pipeline into a directed acyclic graph (DAG) structure and identifying parallel opportunities during the rewriting process to write corresponding operators into a parallel structure;

[0039] An optimizer, which uses a cost model to estimate the cost of each operator in the DAG structure and selects at least one DAG candidate pipeline with the lowest overall cost as the optimal execution pipeline;

[0040] A pipeline executor is used to execute at least one selected optimal execution pipeline according to a layer-by-layer execution strategy, and monitor intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline;

[0041] Pipeline integrator, which evaluates and integrates the output of each executed pipeline;

[0042] A context manager to manage intermediate results in execution to keep the information coherent and avoid exceeding the context length limit for large language models;

[0043] Indexing and storage modules for storing and indexing structured, semi-structured, and unstructured data in the data lake; and

[0044] A reward model that distributes rewards based on the relevance and impact of operators in a given state, fine-tuning large language models to enhance chain thinking capabilities.

[0045] According to one embodiment of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement any of the above methods when executing the computer instructions.

[0046] According to one embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the above items is implemented.

[0047] The automated interactive large-scale language model pipeline orchestration solution for complex queries proposed in this disclosure is based on an automated pipeline orchestration and optimization mechanism, which enables the system to cope with various complex query scenarios and significantly reduces the labor cost of complex query processing in the data lake. By introducing predefined operators as standardized building blocks, not only the standardization of pipeline construction is improved, but also unified optimization and management are facilitated. By rewriting the chain pipeline into a DAG structure that supports parallelism, the execution efficiency is further improved. Pipeline selection and adjustment based on the cost model, as well as dynamic and adaptive adjustment of the execution path through real-time monitoring of intermediate results and execution status, these multi-level execution and optimization mechanisms are conducive to significantly improving query efficiency and query accuracy and reducing resource consumption. The disclosure also further reduces execution delays and improves system response speed through a pre-fetch optimization mechanism, and adopts a reward model for feedback optimization to continuously enhance the chain thinking ability of large language models, so that their performance in processing complex queries continues to improve.

[0048] The application of the automated interactive large-scale language model pipeline orchestration solution for complex queries proposed in this disclosure is conducive to the automation, intelligence and efficiency of complex query processing in data lakes.

[0049] Other features and advantages of the technical solution proposed in this disclosure are described in detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0051] Figure 1 A flowchart of a method for automated interactive large-scale language model pipeline orchestration for complex queries according to an embodiment of the present disclosure is shown.

[0052] Figure 2 FIG. 4 is a schematic diagram showing rewriting the candidate pipeline 1 into a DAG structure according to an embodiment of the present disclosure.

[0053] Figure 3 FIG. 4 shows a schematic diagram of rewriting the candidate pipeline 2 into a DAG structure according to an embodiment of the present disclosure.

[0054] Figure 4 A schematic diagram of dynamic adaptive adjustment of pipelines according to an embodiment of the present disclosure is shown.

[0055] Figure 5 A schematic diagram illustrating a prefetch optimization technology according to an exemplary embodiment of the present disclosure is shown.

[0056] Figure 6 A schematic diagram of context management according to an exemplary embodiment of the present disclosure is shown.

[0057] Figure 7 A schematic diagram of an automated interactive large-scale language model pipeline orchestration system for complex queries according to an exemplary embodiment of the present disclosure is shown.

[0058] Figure 8 A schematic diagram of a processing flow of a complex query according to an exemplary embodiment of the present disclosure is shown.

[0059] Figure 9 A schematic diagram of the execution process of rewriting a chain structure execution pipeline into an R1 path in a DAG structure execution pipeline according to an embodiment of the present disclosure is shown.

[0060] Figure 10 A schematic diagram of the execution process of rewriting a chain structure execution pipeline into an R2 path in a DAG structure execution pipeline according to an embodiment of the present disclosure is shown.

[0061] Figure 11 It is a schematic structural diagram of an electronic device according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0063] The disclosed embodiments may be applied to a computer system / server that is operable with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, among others.

[0064] Computer systems / servers may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and the like, that perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computer system storage media, including storage devices.

[0065] The core concept of the automated interactive large-scale language model pipeline orchestration solution for complex queries proposed in this disclosure is: using a predefined standardized set of operators as the basic building block, using a large language model to automatically extract relevant operators from the query and build an initial chain execution pipeline; rewriting the chain pipeline into a DAG structure to support parallel execution, while using a cost model to select the optimal execution path; monitoring intermediate results in real time during execution, dynamically adjusting the pipeline based on the execution status, and improving efficiency through prefetching mechanisms and parallel execution; using a large language model to assist in result interpretation and integration, and continuously optimizing system performance through a reward model. This design realizes the automation of the entire process of complex query processing, from pipeline construction to execution to optimization, overcomes the limitations of static manual orchestration in existing technologies, and can improve the automation, intelligence, and efficiency of complex query processing in data lakes.

[0066] Figure 1 A flowchart illustrating a method for orchestrating an automated, interactive, large-scale language model pipeline for complex queries, according to one embodiment of the present disclosure, is provided. This method can be applied to complex queries of structured, semi-structured, and unstructured data in a data lake. As shown in the figure, the method includes steps 1 through 6.

[0067] Step 1: Receive a query input by a user in natural language.

[0068] Users can express their queries in everyday language, rather than structured query statements (such as SQL). Queries can involve multiple data sources, require multi-step reasoning, or analyze different types of data. For example, "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and identify abnormal expenditure items that exceed the historical average" is a typical natural language query.

[0069] In this embodiment, semantic understanding technology based on pre-trained language models such as GPT and BERT can be used to identify the intent and parse the semantics of queries. Supported query types include but are not limited to: multi-hop semantic retrieval queries, multi-step logical reasoning queries, and multi-stage semantic analysis queries.

[0070] Natural language-based querying lowers the barrier to entry for users, allowing even non-technical users to easily perform complex data analysis. Furthermore, the flexibility of natural language provides the system with more semantic information, facilitating subsequent intelligent processing.

[0071] Step 2: Using a large language model, extract operators from a predefined operator set and construct multiple chain candidate pipelines with different reasoning paths corresponding to the query.

[0072] After obtaining the user query, operators can be extracted based on the query intent and multiple chained candidate pipelines can be constructed. This embodiment achieves standardization and modularization of query processing by pre-defining operators, facilitating subsequent pipeline construction and optimization. The combination of operators can handle various complex query scenarios, while its standardized nature facilitates unified optimization and management. Chained pipelines are linear structures formed by sequentially connecting operators. Leveraging the randomness of large language models, multiple candidate pipelines with different inference paths can be generated.

[0073] In some embodiments, the predefined operator set includes some or all of the following operators:

[0074] Retrieval operators: Fetch relevant data from external sources to answer queries, enhancing the timeliness or specificity of responses from large language models;

[0075] Scan operators: load documents or tables and enumerate their elements, typically applying filtering or initial transformations while scanning;

[0076] Filter operators: remove irrelevant information from retrieved or generated data based on specific criteria, similar to the select operator in SQL;

[0077] Sorting operator: Sorts the retrieved documents or information according to a given criterion, similar to the SQL ORDER BY operator, but with semantic sorting capabilities;

[0078] Summarization operator: condenses longer text into shorter summaries to optimize context consumption and readability;

[0079] Generative operators: Generate coherent text based on input, typically used to retrieve relevant information and perform a series of analyses before a large language model generates the final response.

[0080] Refinement operator: Improve or adjust the input text to better meet requirements or improve coherence;

[0081] Classification operators: classify or label input text or entities in it, and perform simple classification tasks;

[0082] Translation operator: converts text from one language to another;

[0083] Transformation operators: convert data from one format to another, such as converting a structured table to a text description;

[0084] Evaluation operators: evaluate the quality of different inputs according to a criterion, usually through the output or internal log-probability of the LLM;

[0085] Explanation operator: provides explanations or reasoning for the model's responses or decisions, enabling the model's self-reflection;

[0086] Integration Operator: Receives information from multiple sources, judges their accuracy, and integrates them into a coherent response;

[0087] Conceptualization operators: extract and identify key concepts described in the text, simplifying complex information by locating the main ideas;

[0088] Extraction operators: Identify and extract specific information from data, similar to the projection operator in SQL;

[0089] Plan Operator: Automatically orchestrate a series of operators to execute complex queries;

[0090] Link operators: process heterogeneous data sets, identifying and linking data that contains relevant information for the current query;

[0091] Set operators: perform set operations on data sets, such as union, intersection, and complement;

[0092] Verification Operator: Verifies the accuracy and reliability of generated or retrieved information;

[0093] Grouping operators: divide data into subgroups based on specific attributes, allowing summary statistics to be calculated for each group;

[0094] Comparison: Evaluates two input values ​​and returns a Boolean result if the comparison criteria is met;

[0095] Aggregation operators: Calculate aggregate results of data, such as count, sum, maximum value, etc.

[0096] Those skilled in the art may select some or all of the operators to form a predefined operator set as needed, and may also design other operators as needed.

[0097] The implementation of operators can include large language model-based implementation and pre-programmed implementation.

[0098] Implementation based on a large language model: This is done by designing specific prompts for the large language model. For example, the execution of the overview and extraction operators involves taking the operator and its description as prompt input, guiding the large language model to perform the operation, and returning the result. For operators that require data retrieval, such as semantic retrieval or semantic clustering, the data is first retrieved, and then the large language model is instructed to perform the operation.

[0099] Pre-programmed implementations: Operators that do not involve large language models can be directly executed using pre-defined programs. For example, a search operator might perform a nearest neighbor search on document embeddings or a keyword search using the BM25 algorithm, while a filter operator might apply programmatic rules, such as filtering by attribute value.

[0100] Large language models can use semantic understanding to identify the required operations from a query. For example, for the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and identify unusual expenditure items that exceed the historical average," a large language model can identify that "the fourth quarter of 2024" requires a filter operator to filter the time range, that "per capita expenditure of each department" requires a search operator to obtain expenditure data and an aggregation operator to calculate the per capita value, and that "identify unusual expenditure items that exceed the historical average" requires a search for historical data, an aggregation operator to calculate the average, and a comparison operator to identify anomalies.

[0101] Taking the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and find abnormal expenditure items that exceed the historical average" as an example, the following two chain candidate pipelines may be generated:

[0102] Candidate Pipeline 1:

[0103] Retrieve (department expenditure data) -> Filter (2024Q4) -> Extract (per capita expenditure) -> Retrieve (historical data) -> Aggregate (calculate mean) -> Compare (identify anomalies)

[0104] Candidate Pipeline 2:

[0105] Search (2024Q4 spending data) -> Filter (by department) -> Extract (per capita spending) -> Search (historical average) -> Compare (identify anomalies)

[0106] The stochastic nature of large language models can be exploited to generate multiple candidate pipelines with different inference paths. The semantic understanding of large language models facilitates the construction of reasonable operation sequences, with each candidate pipeline containing the complete chain of operations required to solve the query. Operators in each pipeline are executed sequentially in topological order, meaning the output of the previous operator serves as the input to the next.

[0107] By generating a simple and intuitive initial chain pipeline, it provides a foundation for subsequent DAG rewriting and optimization.

[0108] In step 3, each chain candidate pipeline is rewritten into a directed acyclic graph (DAG) structure, and parallel opportunities are identified during the rewriting process to rewrite the corresponding operators into a parallel structure.

[0109] Chained candidate pipelines can be rewritten as DAG structures to support parallel execution. This rewriting process can be guided by prompts from a large language model, which identifies operations that can be executed in parallel during the conversion process. For example, retrieval operators with no dependencies can be executed in parallel, while operations such as extraction and aggregation must wait for the completion of preceding operations.

[0110] For the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and find abnormal expenditure items that exceed the historical average", the two candidate pipelines given above can be rewritten as a DAG.

[0111] Figure 2 FIG. 4 is a schematic diagram showing rewriting the candidate pipeline 1 into a DAG structure according to an embodiment of the present disclosure. Figure 3 FIG. 4 is a schematic diagram showing rewriting the candidate pipeline 1 into a DAG structure according to an embodiment of the present disclosure.

[0112] The above DAG structure supports parallel execution while maintaining operation dependencies, which can improve execution efficiency and provide a basis for subsequent cost evaluation and dynamic adjustment.

[0113] In step 4, the cost model is used to estimate the cost of each operator in the DAG structure, and at least one DAG candidate pipeline with the lowest overall cost is selected as the optimal execution pipeline.

[0114] In this embodiment, a cost model is used to evaluate candidate pipelines of the DAG structure. Cost evaluation can take into account the computational characteristics of operators, data size, and resource consumption to achieve accurate cost estimation.

[0115] In some embodiments, the cost model is used to estimate the cost of each operator based on the size of the input data and the cost function of the operator, wherein, when estimating the cost of the operator, the cardinality estimate is used as the size of the input data, and the cardinality estimate approximates the selectivity through a random sampling method.

[0116] The cost model defines a resource-specific cost function for each operator, which can be based on two core parameters: the operator's computational complexity and the input data size (cardinality). Different types of operators consume different resources. Structured data operators primarily consume I / O resources, while semi-structured and unstructured data operators primarily consume CPU / GPU resources to perform embedding and attention calculations. Those skilled in the art can design the specific cost function form as needed.

[0117] The parameterization of the cost function depends on the operator type. For pre-programmed operators, it has a fixed computational complexity coefficient; for operators based on large language models, the cost is approximately linear in the output size.

[0118] Constructing a cost model is equivalent to determining specific parameter values ​​for the cost function of each operator, including computational complexity coefficients, resource consumption weights, and so on.

[0119] In data lake analysis or unstructured data analysis, the scale of intermediate results may vary significantly, so cardinality estimation is of great significance to the cost model. According to this embodiment, a random sampling method (for example, sampling 1% of the data) can be used to approximate the selectivity and thus estimate the cardinality of the result. By executing queries on samples, the system is able to estimate the cardinality of the results with relatively little time consumption. This embodiment is particularly suitable for semantically related operators and unstructured data, because these scenarios are difficult to use histograms in traditional databases or learning-based data distribution methods. When multiple candidate pipelines are given, the cost of each pipeline can be estimated and the cardinality of the intermediate data can be predicted to select the optimal execution path.

[0120] According to some embodiments of the present application, the parameters of the cost function of the operator in the cost model can be adjusted according to the sample workload to ensure that the cost can be accurately estimated in actual execution. See the relevant introduction below for details.

[0121] The number of optimal execution pipelines finally selected can be set. For example, if K=1 is set, the candidate pipeline with the lowest overall cost can be selected as the optimal execution pipeline. If K=2 is set, the two candidate pipelines with the lowest overall cost can be selected as the optimal execution pipelines.

[0122] The cost model also allows for dynamic adjustment of the pipeline's operator execution order without compromising result accuracy, further improving execution efficiency. Intelligent data lake analytics often involve multiple optimization objectives, such as minimizing query latency, maximizing result accuracy, or minimizing the use of large language models (LLMs). To meet different optimization goals, the cost model can be adjusted to suit different optimization criteria. A composite cost model can also be employed, which takes a weighted average of costs from different sources to achieve multi-objective optimization.

[0123] With the support of the unified cost model, operators can be dynamically scheduled in combination with resource thresholds and cost estimation results, so as to achieve parallel execution of operators as much as possible without exceeding resource constraints.

[0124] In addition, to reduce operator delays caused by the materialization of intermediate results, operators located between multiple pipeline processing stages can be prioritized during execution, allowing downstream operators to start processing some outputs earlier and improve execution efficiency.

[0125] Through the above-mentioned detailed pipeline optimization strategy, this embodiment can maximize the efficiency of query execution while ensuring the accuracy of the results.

[0126] Step 5: Execute the selected at least one optimal execution pipeline, and monitor the intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline.

[0127] If the number of optimal execution pipelines finally selected in step 4 is greater than or equal to 2, these optimal execution pipelines are executed in parallel during execution.

[0128] In some implementations, a layer-by-layer execution strategy is employed to execute DAG-structured pipelines. This ensures that operators are executed in the DAG's topological order, with execution of an operator beginning only after all of its pre-dependencies have completed. This layer-by-layer execution strategy helps ensure the correctness and reliability of data flow.

[0129] In some implementations, pre-fetching technology can be used to further improve query execution efficiency. Detailed description of pre-fetching technology is provided below.

[0130] During execution, the execution results and status of each operator can be continuously tracked, and dynamic adjustments can be made when necessary to ensure the accuracy and efficiency of the query results.

[0131] In some implementations, real-time monitoring of intermediate results and execution status to dynamically and adaptively adjust the execution pipeline includes:

[0132] Real-time monitoring of intermediate results and current execution status generated during the execution process;

[0133] evaluating the execution quality of the currently executed pipeline based on the intermediate results and the execution status;

[0134] If the execution quality does not meet the expected quality requirement, stopping the execution of the currently executed pipeline, regenerating at least one reference execution pipeline based on the collected intermediate results and using the large language model, and determining an adjusted execution pipeline from the at least one reference execution pipeline using the cost model.

[0135] According to this embodiment, the intermediate results and execution status generated during the execution process can be continuously monitored. The intermediate results refer to the data content and scale output by the operator, for example, whether the data returned by the retrieval operation is relevant, and whether the amount of data after the filtering operation is reasonable; the execution status usually includes whether the operator is successfully completed, resource consumption, execution time, etc. The relevance, accuracy and execution status of the intermediate results can be comprehensively considered to evaluate whether the execution quality of the current path meets the expected quality requirements. For example, when the retrieval operator does not return the expected relevant data, the amount of data returned by the filtering operator is abnormal, the operator execution fails or times out, and / or resource consumption exceeds expectations, it can be judged that the current execution quality does not meet the expected quality requirements, then the current execution path will be stopped, and a new execution path exploration will be started based on the collected intermediate results and using a large language model.

[0136] The following DAG structure for the complex query "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and find abnormal expenditure items that exceed the historical average" is used as an example to illustrate dynamic adaptive adjustment. Figure 4 shown.

[0137] First, execute layer by layer according to the DAG structure. First, execute "Search (department expenditure data)" and "Search (historical data)" in parallel. After executing "Search (department expenditure data)", the system passes the data to the "Filter (2024Q4)" operator. During this process, the system monitors intermediate results and execution status in real time.

[0138] Through real-time monitoring, it can be monitored that "Retrieve (department expenditure data)" returns a large amount of data, and it can also be monitored that the execution speed of the "Filter (2024Q4)" operator is abnormally slow, consumes a large amount of CPU and memory resources, and is in a "waiting" state for a long time.

[0139] Based on the above monitored information, it can be considered that the data quality is low and the execution efficiency is low, and it is judged that the execution quality of the current execution path does not meet the expected quality requirements.

[0140] At this time, dynamic adjustment can be triggered to stop executing the "Filter (2024Q4)" operator and its subsequent "Extract (Per Capita Expenditure)" operator, and the collected intermediate results and execution status can be passed as input to the large language model. The input information can include the output result of "Retrieve (Department Expenditure Data)", the error information generated when executing the "Filter (2024Q4)" operator, resource consumption information, execution time information, etc.

[0141] Based on this information and the user's initial query intent, large-scale language models can perform semantic analysis and reasoning, determine data quality issues in the current execution path, and generate multiple reference execution paths. For example, they can consider using the "verify" operator to verify data before filtering, or adjust the parameters of the "retrieve" operator to specify the data fields to be retrieved to avoid retrieving invalid data. For example, they can consider using the "classify" operator to classify departments and then filter the classified results, and so on. The cost model can estimate the multiple reference execution paths generated by the large language model, estimating the overall cost of each reference execution path based on the operator type, input data size, and cost function.

[0142] Next, based on the cost model's estimated results, we can determine the adjustment targets from multiple reference execution paths. Assume that, based on the estimated results, we update the execution path to include "Search (department expenditure data)" -> "Verify" -> "Filter (2024Q4)" -> "Extract (per capita expenditure)." The "Search (historical data)" and its downstream operators in the original execution path continue to execute in parallel.

[0143] During dynamic adjustment, a large language model can analyze the problem and generate multiple reference execution pipelines based on query intent semantic analysis. The cost model can estimate the overall cost of the reference execution pipelines and determine the adjustment plan.

[0144] After adjusting the pipeline to continue execution, you can continue to monitor the intermediate results and execution status and trigger dynamic adjustments again if necessary.

[0145] Through a dynamic adjustment mechanism that combines a large language model with a cost model, execution anomalies can be discovered and handled promptly. For example, when retrieval results are irrelevant, the pipeline can be adjusted promptly, and downstream steps of the path can be pruned early to avoid continuing execution based on erroneous information, thereby saving resources and improving efficiency.

[0146] Step 6: Integrate the outputs of each execution pipeline and return the integrated result to the user as the final query result.

[0147] In some embodiments, integrating the outputs of various execution pipelines includes:

[0148] After each execution pipeline is completed, the output of each execution pipeline is explained using a large language model;

[0149] Analyze and merge the interpretation results of the outputs of each execution pipeline, and use the merged results as the integrated results.

[0150] For example, the output of each execution pipeline can be input to the interpretation operator, the output of the interpretation operator passes through the comparison, validation, summary and merger operators in sequence, and then the output of the merger operator is returned to the user as the integrated result.

[0151] For example, for the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024 and identify abnormal expenditure items that exceed the historical average", the query results can be obtained through the following two different channels:

[0152] Execution pipeline 1: Calculate abnormal expenditure items based on department expenditure data;

[0153] Execution pipeline 2: Analyze quarterly expense data to identify unusual expense items.

[0154] A large language model can be used to interpret the output of each pipeline, compare the similarities and differences between the two results, verify the rationality of the results, and integrate them into a consistent final answer. Depending on the query intent, this final result may be an aggregated dataset, a calculated value, a piece of text, or another form of answer.

[0155] Through integration, the accuracy and reliability of query results are further improved, ensuring the integrity and consistency of the final output.

[0156] The automated interactive large-scale language model pipeline orchestration method for complex queries proposed in this embodiment is based on an automated pipeline orchestration and optimization mechanism, can cope with various complex query scenarios, and greatly reduces the labor cost of complex query processing in the data lake. By introducing predefined operators as standardized building blocks, not only the standardization of pipeline construction is improved, but also unified optimization and management are facilitated. By rewriting the chained execution pipeline into a DAG structure that supports parallelism, the execution efficiency is further improved. Pipeline selection and adjustment based on the cost model, as well as dynamic and adaptive adjustment of the execution path through real-time monitoring of intermediate results and execution status, improves query efficiency and query accuracy, and reduces resource consumption.

[0157] In some embodiments, the following prefetching technology may be used during execution: when executing the operator corresponding to the retrieval, a main search and multiple backup searches are generated, and different backup searches correspond to different retrieval scopes directly or indirectly associated with the query intent; when processing the results returned in response to the main search, the multiple backup searches are executed in parallel, and the results returned in response to the backup searches are received and saved.

[0158] This embodiment introduces pre-fetch optimization technology, which helps to reduce delays caused by retrieval failures and improve overall execution efficiency.

[0159] During the execution process, the search operator may not be able to retrieve the expected results. At this time, re-planning and retrieval are required, resulting in a delay in the query processing process. In order to reduce the delay caused by retrieval failure, according to this embodiment, idle computing resources can be used to pre-execute in the background and obtain potentially useful information. When the search operator generates the main search, multiple backup searches can be generated at the same time. The backup search can cover search ranges of multiple granularities, such as exact match searches directly related to the query intent, such as "2024 Q4 department expenditure data", and indirectly related extended searches such as "2024 department expenditure data", "department quarterly expenditure summary" and "department expenditure details", providing support for possible pipeline adjustments through multi-level retrieval strategies.

[0160] While processing the results returned by the primary search, these backup searches can be executed in parallel in the background. If the initial results are unsatisfactory and the execution pipeline needs to be adjusted, the results of the backup searches can be directly retrieved, avoiding additional waiting time. This prefetching mechanism significantly reduces the latency caused by failed searches, improves query processing efficiency, and effectively utilizes computing resources that may be idle during analysis and inference of large language models.

[0161] Figure 5 Schematic diagram of a prefetch optimization technology according to an exemplary embodiment of the present disclosure is shown. In order to more clearly show the effect of the prefetch optimization technology in this embodiment, Figure 5 A comparison of the execution flow with and without prefetch optimization is shown. Figure 5 The upper middle portion corresponds to the execution process without prefetch optimization, and the lower portion corresponds to the execution process with prefetch optimization.

[0162] During the execution process without prefetch optimization, after executing the main search O4 and receiving the returned results, the system must wait for the LLM (Large Language Model) to analyze the intermediate results and execution status. If the LLM analysis indicates that the main search O4 has failed, it will guide the pipeline adjustment to execute search O3. Next, search O3 is executed and the returned results are received. The system then continues to wait for LLM analysis and guidance on subsequent pipeline adjustments, and finally, search O2 succeeds. This entire process results in multiple waiting delays due to multiple search failures.

[0163] During the execution process with prefetch optimization, after the primary search O3 executes and receives the returned results, the LLM processes the results and adjusts the pipeline while executing backup searches O3 and O2 in parallel. By the time the LLM finishes processing the results returned by the primary search O4 and guiding pipeline adjustments, the target pipelines O3 and O2 have already completed execution. Therefore, there is no need to wait for the search to execute, and the results of backup searches O3 and O2 can be directly obtained. This entire process fully utilizes computing resources and significantly reduces execution latency.

[0164] This embodiment significantly improves query execution efficiency and result accuracy through a multi-level execution monitoring and adjustment mechanism, combined with prefetch optimization. The system can quickly respond to execution anomalies and dynamically adjust execution strategies, while also reducing latency caused by these adjustments through the prefetch mechanism.

[0165] In some embodiments, parameters of the cost function of an operator in the cost model may be adjusted based on the sampled workload to ensure accurate cost estimation in actual execution.

[0166] According to this embodiment, a dynamic parameter adjustment mechanism based on sampled workload is introduced to improve the accuracy and adaptability of the cost model. Specifically, a portion of execution data can be randomly extracted from the actual operation process to obtain a sampled workload. These sampled data can reflect the performance characteristics of the system in real scenarios. By analyzing the sampled workload, deviations in the current cost model can be found, for example, the cost of certain operators is overestimated or underestimated, so that the parameters of the cost model can be adjusted in time. In one example, the actual cost can be calculated based on the collected sampled workload and compared with the cost predicted by the cost model to calculate the prediction error of the model, and further adjust the parameters of the model through machine learning. By dynamically adjusting the cost model according to the actual operation conditions, the cost model can better adapt to different data characteristics and execution environments.

[0167] In some embodiments, the method further comprises performing feedback optimization using a reward model:

[0168] Evaluate the relevance and accuracy of each operator based on its output and execution status during execution;

[0169] Rewards are distributed based on the relevance and accuracy of the evaluation, and the reward information is used to fine-tune a large language model that extracts relevant operators from the query to enhance the chain thinking ability of the large language model.

[0170] According to this embodiment, the reward model can be used to continuously optimize the performance of the system. Taking the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024, and find out the abnormal expenditure items that exceed the historical average" as an example, the execution of each operator can be evaluated. After the retrieval operator is executed, the relevance of the retrieval result to the query is evaluated. If accurate expenditure records are obtained when retrieving "department expenditure data", a higher reward is assigned; if irrelevant data is retrieved, a lower reward is assigned. After the filtering operator is executed, the accuracy of the filtering results can be evaluated. If the "fourth quarter of 2024" data is correctly filtered out, a higher reward is assigned; if relevant data is omitted or irrelevant data is included, a lower reward is assigned.

[0171] Based on these evaluation results, the large language model used to extract operators from queries can be updated. For example, if the model frequently extracts poorly performing operator combinations, feedback optimization can be used to adjust the model's operator selection strategy. This continuous optimization mechanism strengthens the large language model's chain-thinking capabilities and improves the accuracy of subsequent query processing.

[0172] According to some embodiments, context management techniques can be used to address the context length limit of large language models (e.g., 8,192 tokens) to ensure that only context information closely related to the current state is retained during execution. During execution, context management can be achieved by using the following methods: using the summary operator to compress and summarize intermediate execution information; using the interpretation operator to compress the context of each execution path; and retaining only information relevant to the current execution state.

[0173] For example, when executing the query "Analyze the per capita expenditure of each department in the fourth quarter of 2024, and find out the abnormal expenditure items that exceed the historical average", after using the retrieval operator to obtain a large amount of raw data, the overview operator can be used to generate a data summary; the intermediate results produced by multiple execution paths are compressed through the explanation operator; when integrating the results, only key information is retained to avoid exceeding the context restrictions.

[0174] The context management mechanism can effectively reduce execution delays caused by context restrictions, enabling the system to efficiently handle complex queries that require multi-step reasoning and large-scale data processing.

[0175] Figure 6A context management diagram according to an exemplary embodiment of the present disclosure is shown. During execution, multiple context units can be maintained, which are used to store intermediate results and state information generated during the execution of operators. When executing a step that requires a large language model for reasoning (for example, the "interpretation" and "integration" operators in the figure), the large language model can read the information in the current context unit and analyze, reason or perform corresponding operations based on this information. In order to avoid the input length of the large language model exceeding the limit, the system can use the "interpretation" operator to compress and summarize the context information and only pass the key information to the subsequent steps. In addition, the context unit will be passed between different operators as the execution process progresses, thereby ensuring the consistency and integrity of the information. When the execution process requires result integration, the system can read the information in multiple context units and use the integration operator to integrate information from multiple sources into a consistent response.

[0176] This embodiment uses a large language model in multiple processing steps. For different tasks, such as extracting operators from queries, generating execution pipelines, and interpreting results, the same large language model can be used, or different large language models can be used depending on the specific tasks. The solution proposed in this embodiment supports flexible configuration to meet the needs of different scenarios.

[0177] The present disclosure also proposes an automated interactive large-scale language model pipeline orchestration system for complex queries, wherein the system is used to implement the above-mentioned method, and the system includes:

[0178] A query interface, used to receive queries entered by users in natural language and identify the user's query intent;

[0179] Operator set module, used to store predefined operator sets;

[0180] A pipeline builder, configured to extract operators from an operator set using a large language model and construct multiple chained candidate pipelines with different reasoning paths corresponding to the query;

[0181] a pipeline rewriter for rewriting the chain candidate pipeline into a directed acyclic graph (DAG) structure and identifying parallel opportunities during the rewriting process to write corresponding operators into a parallel structure;

[0182] An optimizer, which uses a cost model to estimate the cost of each operator in the DAG structure and selects at least one DAG candidate pipeline with the lowest overall cost as the optimal execution pipeline;

[0183] A pipeline executor is used to execute at least one selected optimal execution pipeline according to a layer-by-layer execution strategy, and monitor intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline;

[0184] Pipeline integrator, which evaluates and integrates the output of each executed pipeline;

[0185] A context manager to manage intermediate results in execution to keep the information coherent and avoid exceeding the context length limit for large language models;

[0186] Indexing and storage modules for storing and indexing structured, semi-structured, and unstructured data in the data lake; and

[0187] A reward model that distributes rewards based on the relevance and impact of operators in a given state, fine-tuning large language models to enhance chain thinking capabilities.

[0188] Figure 7 A schematic diagram of an automated interactive large-scale language model pipeline orchestration system for complex queries, according to an exemplary embodiment of the present disclosure, is shown. As shown in the figure, the system primarily comprises modules such as an application layer, a query interface, a planner, an executor, a data index, a reward model, and a data storage.

[0189] Applications supported by the system include data analysis, data discovery, information extraction, retrieval-augmented generation (RAG) and machine learning, which can explore data value from multiple perspectives.

[0190] The query interface accepts complex queries in natural language format. The planner includes two types of operators: semantic operators and pre-programmed operators. Operators refer to the operators mentioned above, and pipeline planning is performed through the optimizer (including pipeline builder, rewriter, and integrator).

[0191] The executor consists of two parts: the operator executor and the interactive executor. The operator executor implements the physical execution of operators and improves efficiency through parallel execution. The interactive executor handles unexpected intermediate results through pipeline adjusters and context managers.

[0192] Data indexing is divided into vector indexing (supporting text and image embedding) and obtaining external data (through retrieval and scanning operators). The reward model provides feedback based on the accuracy of the execution results for fine-tuning large language models.

[0193] Data storage supports structured (tables), semi-structured (JSON, XML, etc.) and unstructured (documents, images, videos, etc.) data.

[0194] Figure 7 The system shown supports flexible complex queries, multi-step planning based on predefined operators, pipeline conversion from chain to DAG, optimization based on cost model, and the ability to process heterogeneous data.

[0195] Figure 8FIG. 1 shows a schematic diagram of a processing flow of a complex query according to an exemplary embodiment of the present disclosure. Figure 8 As shown, for the complex query in natural language form "What is the average height of New York Knicks players who attend Villanova University?", the system performs the following processing.

[0196] First, the system generates the initial pipeline - the chain structure execution pipeline:

[0197] R1 path: Retrieve -> Aggregate -> Verify -> Generate;

[0198] R2 path: Retrieval -> Extraction -> Retrieval -> Verification -> Aggregation.

[0199] The system rewrites the chain structure execution pipeline into the execution process of the R1 path in the DAG structure execution pipeline as follows Figure 9 As shown, the execution process of the R2 path is as follows Figure 10 shown.

[0200] During the layer-by-layer execution phase, the search for path R1 ("New York Knicks players who attended Villanova University") failed to yield the desired result, and the system prematurely pruned its subsequent steps. However, the search for path R2 ("New York Knicks players") successfully retrieved data, and after executing the complete process, the average height of the players was determined to be 1.91 meters.

[0201] The above examples demonstrate the system's automated pipeline orchestration and interactive execution capabilities when processing complex queries, including features such as multi-path parallel processing, dynamic adjustment, and result integration.

[0202] Figure 11 An electronic device provided for at least one embodiment of the present disclosure includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the automated interactive large-scale language model pipeline orchestration method for complex queries described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0203] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automated interactive large-scale language model pipeline orchestration method for complex queries as described in any embodiment or implementation of the present disclosure.

[0204] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the data processing device embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0206] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0208] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0209] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0210] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0211] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0212] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0213] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0214] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. An automated interactive large-scale language model pipeline orchestration method for complex queries, characterized by: include: receiving a query input by a user in natural language; Extracting operators from a predefined set of operators using a large language model to construct multiple chained candidate pipelines with different reasoning paths corresponding to the query; rewriting each chain candidate pipeline into a directed acyclic graph (DAG) structure, and identifying parallel opportunities during the rewriting process to write the corresponding operators into a parallel structure; Use the cost model to estimate the cost of each operator in the DAG structure and select at least one DAG candidate pipeline with the lowest overall cost as the optimal execution pipeline; executing the selected at least one optimal execution pipeline, and monitoring intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline; The outputs of each execution pipeline are integrated and the integrated result is returned to the user as the final query result.

2. The method according to claim 1, characterized in that The predefined set of operators includes some or all of the following operators: retrieve, scan, filter, sort, summarize, generate, refine, classify, translate, transform, evaluate, interpret, integrate, conceptualize, extract, plan, link, aggregate, validate, group, compare, and aggregate.

3. The method according to claim 1, wherein: The cost model is used to estimate the cost of each operator according to the size of input data and the cost function of the operator, wherein, when estimating the cost of the operator, the cardinality estimation value is used as the size of the input data, and the cardinality estimation is used to approximate the selectivity through a random sampling method.

4. The method according to claim 1, wherein The method further comprises: The parameters of the cost functions of the operators in the cost model are adjusted according to the sampled workload.

5. The method according to claim 1, wherein Real-time monitoring of intermediate results and execution status to dynamically and adaptively adjust the execution pipeline, including: Real-time monitoring of intermediate results and current execution status generated during the execution process; evaluating the execution quality of the currently executed pipeline based on the intermediate results and the execution status; If the execution quality does not meet the expected quality requirement, stopping the execution of the currently executed pipeline, regenerating at least one reference execution pipeline based on the collected intermediate results and using the large language model, and determining an adjusted execution pipeline from the at least one reference execution pipeline using the cost model.

6. The method according to claim 1, characterized in that During execution, a layer-by-layer execution strategy is adopted to execute the pipeline of the DAG structure.

7. The method according to claim 1, characterized in that The method further includes employing the following prefetching technique during execution: When the corresponding search operator is executed, a primary search and multiple backup searches are generated, where different backup searches correspond to different search scopes that are directly or indirectly related to the query intent; The plurality of backup searches are performed in parallel while processing the results returned in response to the primary search, and the results returned in response to the backup searches are received and stored.

8. The method according to claim 1, characterized in that Integrate the output of various execution pipelines, including: After each execution pipeline is completed, the output of each execution pipeline is explained using a large language model; Analyze and merge the interpretation results of the outputs of each execution pipeline, and use the merged results as the integrated results.

9. The method according to claim 1, characterized in that The method also includes feedback optimization using a reward model: Evaluate the relevance and impact of each operator based on its output and execution status during execution; Rewards are distributed based on the relevance and impact of the evaluation, and the reward information is used to fine-tune a large language model that extracts relevant operators from the query to enhance the chain thinking ability of the large language model.

10. An automated interactive large-scale language model pipeline orchestration system for complex queries, characterized by: The system is used to implement the method according to any one of claims 1 to 9, and the system comprises: A query interface, used to receive queries entered by users in natural language and identify the user's query intent; Operator set module, used to store predefined operator sets; A pipeline builder, configured to extract operators from an operator set using a large language model and construct multiple chained candidate pipelines with different reasoning paths corresponding to the query; a pipeline rewriter for rewriting the chain candidate pipeline into a directed acyclic graph (DAG) structure and identifying parallel opportunities during the rewriting process to write corresponding operators into a parallel structure; An optimizer, which uses a cost model to estimate the cost of each operator in the DAG structure and selects at least one DAG candidate pipeline with the lowest overall cost as the optimal execution pipeline; A pipeline executor is used to execute at least one selected optimal execution pipeline according to a layer-by-layer execution strategy, and monitor intermediate results and execution status in real time to dynamically and adaptively adjust the execution pipeline; Pipeline integrator, which evaluates and integrates the output of each executed pipeline; A context manager to manage intermediate results in execution to keep the information coherent and avoid exceeding the context length limit for large language models; Indexing and storage modules for storing and indexing structured, semi-structured, and unstructured data in the data lake; and A reward model that distributes rewards based on the relevance and impact of operators in a given state, fine-tuning large language models to enhance chain thinking capabilities.

11. An electronic device, characterized in that: The device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method according to any one of claims 1 to 9 when executing the computer instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images