Method and device for distilling private domain data from natural language to structured query language

Abstract

The invention discloses a private domain data distillation method and device from a natural language to a structured query language. The method comprises the steps that the identity of a first large model with the tool calling capacity is given to be a question generation agent, the question generation agent calls all pre-built tool functions, and a question list is generated based on a private domain database; updating the identity of the first large model as a structured query statement generation agent, calling each tool function by the structured query statement generation agent, and generating a corresponding structured query statement list based on the question list and the private domain database; and based on the problem list and the corresponding structured query statement list, training a second large model of the small-scale parameters to obtain a private domain data distillation model. By means of the mode, the natural language can be converted into the structured query language through the private domain data distillation model, the computing power is reduced, the structured query language is more accurately converted into the structured query statement suitable for the private domain database, and the accuracy and efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for private domain data distillation from natural language to structured query language. Background Technology

[0002] Currently, Natural Language to Structured Query Language (nl2sql) technology based on Large Language Models (LLM) has demonstrated certain SQL statement generation capabilities in many fields. However, in real-world engineering applications, the data structures involved in various computer applications have distinct characteristics. These private domain data differ significantly from publicly available data in terms of terminology and storage structure. This makes it difficult for models trained using publicly available data to directly play an effective role in real production environments.

[0003] Furthermore, large-scale models with extremely high parameters place extremely high demands on computing power, and deploying such models privately in various enterprise data application scenarios faces significant cost challenges. Even with deployment using techniques such as model quantization, it is often impossible to meet the business's needs for efficient and accurate data querying, thus limiting the widespread application of nl2sql technology in real-world business scenarios. Summary of the Invention

[0004] This application provides a method and apparatus for private domain data distillation from natural language to structured query language, in order to solve the problems in the prior art where large models used to implement natural language to structured query language cannot adapt to private domain databases and where large models with large parameters have high computational requirements.

[0005] To address the aforementioned technical problems, this application adopts the following technical solution: a method for private domain data distillation from natural language to structured query language. This method includes: The first major model with tool calling capabilities is given the identity of a problem-generating agent, and the problem-generating agent calls pre-built tool functions to generate a problem list based on a private domain database; The identity of the first large model is updated to a structured query statement generation agent, and the structured query statement generation agent calls each of the aforementioned tool functions to generate a corresponding structured query statement list based on the question list and the private domain database; Based on the aforementioned list of questions and the corresponding list of structured query statements, a second large model with a small number of parameters is trained to obtain the private domain data distillation model.

[0006] In one optional embodiment of this application, the problem-generating agent calls pre-built tool functions to generate a problem list based on a private domain database, including: In response to the input scene and character prompts, the question-generating agent parses the scene and character prompts and extracts scene and character information; Based on the scene and character information, the tool functions are invoked to autonomously explore the private database and generate a list of questions corresponding to the scene and character prompts.

[0007] In an optional embodiment of this application, before generating the problem list, the following steps are further included: For the problem-generating agent, several initial problems are generated in a single instance, and the first similarity between the initial problems is calculated. In response to the first similarity being higher than a first preset similarity threshold, the corresponding initial question is deleted; In response to the initial number of questions generated multiple times by the question-generating agent reaching a preset number, the question list is constructed based on the preset number of initial questions.

[0008] In an optional embodiment of this application, after generating the problem list, the method further includes: Each of the initial questions in the question list is converted into a semantic vector, and the first cluster is obtained based on the density clustering algorithm; Based on the binary search framework combined with the semantic verification of the large model, the first clustering is iteratively optimized to obtain the clustering threshold; The first cluster is updated based on the clustering threshold to obtain multiple second clusters; Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, duplicates are removed from each of the second clusters to obtain the deduplicated list of questions.

[0009] In an optional embodiment of this application, after generating the list of structured query statements, the method further includes: The identity of the first large model is updated to that of an adversary agent, and the adversary agent, based on each structured query statement in the structured query statement list, calls each of the tool functions to generate several simulation problems corresponding to the structured query statements based on the private domain database; Calculate the second similarity between each of the simulated problems and the corresponding initial problems; In response to the second similarity being higher than a second preset similarity threshold, the structured query statement and the corresponding initial question are used as the final question-answer pair; A training dataset is constructed based on all the final question-answer pairs to train the second major model.

[0010] In an optional embodiment of this application, after obtaining the private domain data distillation model, the method further includes: In response to an input natural language query question for the private domain database, the private domain data distillation model parses the natural language query question and outputs the corresponding structured query statement.

[0011] In one optional embodiment of this application, the tool function is encapsulated based on various database interface tools. The tool function includes at least one or more of the following: database schema generation tool, JSON field parsing tool, selection execution tool, database switching tool, database information acquisition tool, database name acquisition tool, and several common tools. Before assigning the identity of the primary model with tool-calling capabilities as the problem-generating agent, it also includes: The first large model declares the utility functions in its calling interface to understand and call the utility functions.

[0012] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a private domain data distillation device for converting natural language to structured query language. This device includes: The first generation module is used to assign the identity of the first large model with tool calling capabilities as a problem generation agent, and the problem generation agent calls pre-built tool functions to generate a problem list based on a private domain database; The second generation module is used to update the identity of the first large model to a structured query statement generation agent, and the structured query statement generation agent calls each of the tool functions to generate a corresponding structured query statement list based on the question list and the private domain database. The training module is used to train a second large model with a small number of parameters based on the list of questions and the corresponding list of structured query statements, so as to obtain the private domain data distillation model.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, including a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the above-mentioned private domain data distillation method from natural language to structured query language.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a storage medium on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the above-mentioned private domain data distillation method from natural language to structured query language.

[0015] The beneficial effects of this application are as follows: Unlike existing technologies, this application discloses a private domain data distillation method and apparatus for converting natural language to structured query language. This method assigns the identities of a question-generating agent and a structured query statement-generating agent to a first large-scale model with large parameters, and pre-constructs various tool functions that can be called by the first large-scale model. This allows the first large-scale model to learn and generate a list of questions and corresponding list of structured query statements in a private domain database, which is then used to train a second large-scale model with small parameters, resulting in a private domain data distillation model. This model enables online inference from natural language to structured query language. The computational resources required for inference by the large model with small parameters are significantly reduced compared to the large model with large parameters, thereby reducing hardware costs and energy consumption. Simultaneously, because the data used to train the private domain data distillation model is more closely aligned with the structure and business needs of the private domain database, the final trained private domain data distillation model can more accurately convert natural language into structured query statements suitable for that private domain database, improving the accuracy and efficiency of the natural language to structured query language conversion. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an embodiment of the private domain data distillation method from natural language to structured query language provided in this application. Figure 2 This is a schematic diagram of the data distillation process implemented by the interaction of various intelligent agents in an embodiment of the private domain data distillation method from natural language to structured query language provided in this application. Figure 3 This is a schematic diagram of an embodiment of the private domain data distillation device for converting natural language to structured query language provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the storage medium provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] This application provides a method for private domain data distillation from natural language to structured query language, see reference. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the private domain data distillation method for converting natural language to structured query language provided in this application. The private domain data distillation method for converting natural language to structured query language includes: S10: Assign the identity of the first major model with tool calling capabilities as a problem-generating agent, and the problem-generating agent calls the pre-built tool functions to generate a problem list based on the private domain database.

[0021] The first type of large-scale model with tool-calling capabilities can be an open-source model with powerful language understanding and generation capabilities, featuring a large number of parameters, such as OpenAI, Zhipu, and ChatGLM. These large-scale models have been trained on a large amount of data, possess excellent semantic understanding and text generation capabilities, can accurately understand the intent of input commands, and have the ability to call tools. By assigning these large-scale models appropriate identities, they can fully utilize their powerful language processing capabilities to autonomously call tools to achieve corresponding tasks.

[0022] In this application, the utility functions are encapsulated based on various database interface tools. The utility functions include at least one or more of the following: database schema generation tool, JSON field parsing tool, selection and execution tool, database switching tool, database information acquisition tool, database name acquisition tool, and several common tools. Before assigning the identity of the primary model with tool-calling capabilities as the problem-generating agent, it also includes: The first major model declares the utility functions in its calling interface to facilitate understanding and calling of each utility function.

[0023] In this application, a primary model with tool-calling capabilities is initialized and adapted to enable it to call pre-built tool functions. Specifically, firstly, the primary model's parameters are configured, setting its basic language processing parameters to adapt to semantic understanding needs in different scenarios. Next, the interface between the primary model and the tool functions is established, declaring each tool function in the primary model's calling interface, allowing the primary model to recognize and understand the input / output format and functional purpose of each tool function. Furthermore, simulated call tests can be conducted, such as inputting simple natural language commands to check whether the primary model can accurately call the corresponding tool functions and return correct results. If problems arise during testing, the causes are analyzed promptly, and model parameters or adaptation code are adjusted and optimized until the primary model can stably and accurately call the pre-built tool functions.

[0024] In this application, the pre-built utility functions include at least one or more of the following: a database schema generation tool, a JSON field parsing tool, a select execution tool, a database switching tool, a database information retrieval tool, a database name retrieval tool, and several public tools. Among them, the public tools include at least one or more of the following: a todo queue addition tool, a todo queue removal tool, and a user query tool.

[0025] A database scheme generation tool, built by encapsulating the core components of SQLAlchemy, is used to establish and manage connections to databases and execute specified SQL commands to obtain metadata.

[0026] When the database schema generation tool is invoked, the first major model connects to the database through the tool and executes the "SHOW TABLES" statement to retrieve a list of all table names. For each table name, the "DESCRIBE" statement is executed to obtain a structural description, including field names, field types, nullability, key names, default values, and additional information. Simultaneously, the "SHOWCREATE TABLE" statement is used to retrieve the complete table creation statement and filters out the COMMENT field, thus obtaining field and table comment information. This information is combined to form complete field information, which is then concatenated with table indexes and foreign keys to form the table's schema information, which is returned to the major language model.

[0027] This JSON field parsing tool, built by encapsulating the SQLAlchemy query interface, is used to sample and parse JSON type fields in a database and return a detailed description of the field.

[0028] When the JSON field parsing tool is invoked, it randomly samples a certain percentage of values ​​from the JSON type fields in the specified data table and removes duplicates. The deduplicated values ​​are then structured into JSON objects, and their hierarchical structure and data types are parsed. Based on the parsing results, a schema description of the JSON field is generated and updated in the database schema information. For example, given a table named "instance" with a JSON type field named "instance_locat" recording detailed device location information, the JSON field parsing tool randomly samples a certain percentage (e.g., 10%) of values ​​from this field and removes duplicates. The deduplicated values ​​are then structured into JSON objects, and the structure of the JSON object, such as the field hierarchy and data types, is analyzed to form a detailed description of the field. This description is then updated in the database schema cache, and the field description is returned to the large language model.

[0029] The Select execution tool, built by encapsulating SQLAlchemy query functionality, is used to receive query statements, execute queries in the database, and return query results and error information.

[0030] When the Select execution tool is invoked, the primary model can input an SQL query statement. The tool then executes the query in the database and returns the results and error information. This allows the primary model to autonomously query the database based on existing database schema information and other relevant data to obtain more information, thereby formulating more targeted and high-quality questions. This relevant data includes information that assists the primary model in database reading and can be stored in other databases or non-databases. Examples include IoT terminology such as common IoT device types, metrics, national (military) standards documents, industry regulations, and company rules; other timely information and historical experience, such as current dates and historical incident investigation and analysis reports. This relevant information can be retrieved using mainstream Retrieval-Augmented Generation (RAG) techniques (including vector retrieval combined with reranking models).

[0031] The database switching tool, built by encapsulating SQLAlchemy's connection management functionality, provides the ability to switch databases for the primary model.

[0032] When the primary model needs to obtain information from different databases, it can call the database switching tool, pass the information of the database to be switched (such as the database name or connection string) to the database switching tool, and realize the database switching.

[0033] The database information retrieval tool is built by encapsulating SQLAlchemy's database metadata retrieval functionality. It is used to provide the first major model with the ability to retrieve basic database information under a specified IP and port.

[0034] When the database information retrieval tool is invoked, it can obtain basic information about all databases under the specified IP and port, including the number of tables, database size, and character set, providing basic data for database switching and other operations.

[0035] The database name retrieval tool encapsulates SQLAlchemy's function of retrieving the name of the currently connected database, enabling the first major model to perform database switching operations when necessary.

[0036] The TodoQueue tool is used to receive a task description as a parameter and add the task to a task queue, recording the tasks to be completed by the agent. For example, if a question-generating agent needs to refine hypernyms during the question generation process, the hyponyms that need to be traversed and generated can be stored through this tool and then traversed and executed.

[0037] The TodoQueue removal tool receives a task identifier and marks the task as completed from the task queue. For example, after a question-generating agent generates a question and completes the traversal and generation of a hyponym, this tool can remove the corresponding task from the task queue to indicate that the task is completed. Simultaneously, this tool ensures the cleanliness of the task queue, preventing invalid or completed tasks from occupying queue space and affecting the addition and execution of subsequent tasks.

[0038] The UserQuery tool is used to send requests to users when the agent encounters unclear requirements or data anomalies, in order to obtain relevant clarification information or report anomalies. For example, when a user's input is vague and the primary model cannot accurately understand the user's intent, the UserQuery tool can automatically generate a request containing detailed questions and send it to the user, requesting further clarification of the requirements. As another example, when the primary model discovers data anomalies during data processing, such as incorrect data format or missing data, the UserQuery tool can also promptly report the anomalies to the user so that the user can address them in a timely manner, ensuring the smooth progress of the entire data distillation process.

[0039] Furthermore, other utility functions can be built and extended according to actual needs, such as data validation tools, used to rigorously validate input data to ensure its accuracy, integrity, and consistency. This application does not specify the type or number of utility functions.

[0040] By constructing the aforementioned utility functions, the efficiency and accuracy of data distillation can be effectively improved. This not only helps agents better understand user needs but also enables timely identification and resolution of problems during data processing, ensuring a smooth data distillation process. Furthermore, the flexibility and scalability of these utility functions facilitate subsequent feature upgrades and optimizations.

[0041] In this application, the declaration of the above-mentioned tool functions is completed through the calling interface of the first major model, so that the first major model can correctly understand and use the above tools, thereby enabling the agents built based on the first major model to generate user-oriented questions when observing real data.

[0042] By using specific prompts, the primary models with tool-calling capabilities are given distinctive identities, transforming them into corresponding intelligent agents. These agents can flexibly invoke various tool functions to complete tasks based on different requirements.

[0043] In this application, a primary model is assigned the identity of a question-generating agent based on preset question-generating prompts. These preset prompts can be set according to different business scenarios and needs. For example, in the financial field, prompts can be designed for questions related to financial statement analysis and risk assessment; in the medical field, prompts can be designed around disease diagnosis and treatment plan recommendations. The prompts describe the type, scope, depth, and expected output format of the question list to be generated, thereby guiding the question-generating agent to accurately generate a high-quality question list that fits the specific business scenario and meets the user's actual needs.

[0044] In this application, the problem-generating agent calls various pre-built utility functions to generate a list of problems based on a private domain database, including: S11: In response to the input scene and character prompts, the question-generating agent parses the scene and character prompts and extracts scene and character information.

[0045] Specifically, the question-generating agent first performs deep semantic analysis on the input scene and character prompts, identifying key scene elements and character characteristics. For scene elements, it further analyzes the specific environmental background and business scope; for character characteristics, it accurately extracts information such as the character's role, professional field, or skill level. Through this meticulous analysis and extraction, it can comprehensively and accurately grasp the core content conveyed by the scene and character prompts, laying a solid foundation for generating a realistic question list.

[0046] For example, in urban emergency management scenarios, by inputting relevant prompts, the question-generating agent can sequentially stand in front of the emergency management leader (such as the secretary or bureau chief), the emergency management plan editor, and the database administrator (DBA) to generate questions.

[0047] S12: Based on the scene and character information, call various tool functions to explore autonomously in the private domain database and generate a list of questions corresponding to the scene and character prompts.

[0048] A private domain database refers to a database specifically built for a particular business scenario or organization, intended for internal use only, and whose data content is closely aligned with the business needs of that scenario or organization. Unlike widely used public domain databases, it is highly targeted and specialized. The data stored within is often generated, collected, and organized within specific business processes, containing a large amount of key information closely related to the business of that scenario or organization, such as customer information, business transaction records, and operational data. The private domain database in this application can be a cluster of multiple databases, meaning it must contain at least one database.

[0049] The question-generating agent analyzes the scene character prompts, extracts scene character information, and autonomously explores the private domain database from the perspective of the scene character, calling various tool functions, including consulting relevant information to assist the first major model in database reading. The question-generating agent will generate a list of questions from the perspective of the scene character.

[0050] Autonomous exploration refers to the use of utility functions to proactively search for and extract question materials highly relevant to prompts from the vast ocean of information in a private database. This process is not simply data retrieval, but rather combines various technologies such as intelligent analysis and semantic understanding to simulate human thinking and exploration patterns in specific scenarios, thereby generating a series of accurate, valuable questions that meet the needs of the characters and roles in the scenario.

[0051] For example, if the scenario character information is "emergency management leader", the question-generating AI will consider the issues that the emergency management leader might be concerned about from the perspective of the emergency management leader, and generate a list of questions, such as "Has the risk been 'detected early'?", "Has the hidden danger been 'truly rectified'?", "Has the contingency plan been 'mobilized'?", "Has the supplies been 'deployed quickly'?", "Has public opinion been 'controlled'?", "Has the enterprise's main responsibility been 'fully implemented'?", and "Has the situation been 'maintained' at night and on holidays?"

[0052] For example, if the scenario character information is "emergency management plan editor", the question-generating AI will consider the issues that the emergency management plan editor might care about from the perspective of the editor, and generate a list of questions, such as "Is the risk profile (ledger) 'complete'?", "Is the division of responsibilities 'accurate' (are the 'three-fixed plan' and emergency responsibilities of each unit clear, who commands, who handles, and who provides support, and can an executable plan be constructed based on existing data within the specified time)?", "Are the laws and standards updated in a timely manner?", and "Are the existing plans and templates both concise and comprehensive?"

[0053] For example, if the scene character information is "database administrator", the question-generating agent will consider the issues that the database administrator might care about from the perspective of the database administrator, and generate a list of questions, such as "Given a data requirement, can it be retrieved using SQL, such as querying the access status of smoke detectors in the High-tech Zone in the past three months, the number of alarms reported on the devices, etc." or "Given several data returned from the query, can data prediction be performed?"

[0054] By equipping the primary model with a question-generating agent and inputting specific scenario-related prompts, a corresponding question list is generated. This ensures that the questions in the question list closely revolve around the challenges and needs that the person in that scenario might face in their actual work. Through the autonomous exploration of the private domain database using various utility functions, a question list closely related to the data in the private domain database is generated. This generated question list not only accurately reflects the specific requirements for data processing and analysis in a particular scenario but also guides the primary model to further understand the business logic of different roles, ensuring that the subsequently generated queries can effectively meet the diverse data acquisition needs during the private domain data distillation process.

[0055] This application also includes the following steps before generating the problem list: For a number of initial questions generated in a single instance by the problem-generating agent, calculate the first similarity between the initial questions; If the first similarity score is higher than a first preset similarity threshold, the corresponding initial question is deleted. In response to the initial number of questions generated multiple times by the question-generating agent reaching a preset number, a question list is constructed based on the preset number of initial questions.

[0056] In this application, the question-generating agent returns a number of initial questions at a time when calling the utility function to generate questions. The specific number of questions returned at a time will be determined in practice based on the context window available to the first main model, the database scheme, and the length of relevant information assisting the first main model in database reading. Alternatively, it can be set as a single return quantity, such as returning no fewer than 20 questions at a time.

[0057] For a single returned list of questions (containing several initial questions), after converting the initial questions into vectors, calculate the first similarity between every two initial question vectors, compare the first similarity with the first preset similarity, and if the first similarity is higher than the first preset similarity threshold, delete the two initial questions corresponding to that first similarity, or choose to keep one of them.

[0058] The similarity calculation method can use cosine similarity. The first preset similarity can be set according to actual needs, such as 0.6, 0.7, or 0.8. Alternatively, Euclidean distance, dot product, or Manhattan distance can also be used. Correspondingly, the threshold for determining whether the similarity between two items reaches the standard for deletion can also be flexibly set, without specific limitations here.

[0059] After similarity filtering, the returned question list yields several semantically inconsistent initial questions. The question generation agent iteratively executes the task of generating the question list. After each generation, it accumulates the number of initial questions remaining after each filtering. When a preset number (e.g., 50, 100, or 200) is reached, question list generation stops, and these filtered initial questions are grouped together to construct a question list that meets the preset number, which is then used for subsequent secondary deduplication steps.

[0060] In this application, after generating the list of issues, the following is also included: S13: Convert each initial question in the question list into a semantic vector, and obtain the first cluster based on the density clustering algorithm.

[0061] The process involves converting each initial question in the question list into a semantic vector, and then using a density-based clustering algorithm to obtain the sample access order and reachability distance, including: Based on the text embedding model, each initial question is converted into a corresponding semantic vector; Normalize each semantic vector and calculate the cosine distance between the normalized semantic vectors; Based on various cosine distance and density clustering algorithms, the sample access order and multiple first clusters and their corresponding reachability distances are obtained.

[0062] In this application, a pre-trained text embedding model can be used to convert each initial question in the list of questions to be deduplicated into a high-dimensional semantic vector, so as to ensure that semantically similar semantic vectors are closer together.

[0063] Specifically, leveraging the powerful language representation capabilities of text embedding models, the semantic features of each initial input question are accurately captured, and the initial questions are mapped into a high-dimensional space to form corresponding semantic vectors. In this process, the text embedding model fully considers various factors such as the semantics of the words in the initial questions, grammatical structure, and contextual information, ensuring that the generated semantic vectors comprehensively and accurately reflect the semantic content of the initial questions. In this way, the vectors corresponding to semantically similar initial questions are closer together in the high-dimensional space, providing a solid foundation for subsequent clustering operations.

[0064] In this application, the text embedding model can be a model with excellent semantic representation capabilities, such as the Qwen3-Embedding model, the Bidirectional Encoder Representations from Transformers (BERT) model, or the Enhanced Representation through Knowledge Integration (ERNIE) model. These models are pre-trained on large-scale corpora and can deeply understand the semantic information of the initial question, transforming the initial question into a vector representation rich in semantic features. In practical applications, an appropriate text embedding model can be selected according to specific needs and scenarios to ensure the accuracy and effectiveness of semantic vector transformation.

[0065] The high-dimensional semantic vectors obtained from the initial problem transformation can be 768-dimensional, 1024-dimensional, or higher. These high-dimensional vectors can capture the semantic features of the initial problem more comprehensively and meticulously, providing a rich and accurate data foundation for subsequent density clustering and semantic verification. Different dimension choices can be flexibly adjusted according to factors such as the actual data scale, accuracy requirements, and computing resources.

[0066] Each generated high-dimensional semantic vector is normalized. Normalization can be L2 normalization, which divides the semantic vector by its L2 norm (i.e., magnitude). Other methods include L1 normalization, maximum normalization, or minimum-maximum normalization, which can be flexibly selected based on actual needs and vector characteristics. This application uses L2 normalization as an example for explanation.

[0067] First, using the vector magnitude formula, the magnitude of each semantic vector in its corresponding dimensional space is calculated. Then, by dividing the semantic vector by its magnitude, the L2 norm of the normalized vector becomes 1, resulting in a unit vector. This eliminates the influence of vector scale differences on subsequent distance calculations. The L2 normalization formula is as follows: Formula 1; in, Represents the original semantic vector. This represents the normalized semantic vector. Semantic vectors The length of the module.

[0068] The above normalization process can eliminate the differences in length between different semantic vectors, making the subsequent cosine distance calculation more accurate and stable, reflecting only the directional differences between vectors, and not affected by the vector length.

[0069] For each normalized semantic vector, the cosine distance between every two semantic vectors is calculated. The cosine distance measures the semantic similarity between semantic vectors; the smaller the cosine distance, the more semantically similar the two vectors are. The formula for calculating the cosine distance is as follows: Formula 2; in, for and The cosine distance; and These are two normalized semantic vectors, namely Then formula 2 can be simplified to .

[0070] By calculating the cosine distance, the similarity between each pair of semantic vectors can be accurately quantified, providing a reliable basis for subsequent density clustering operations. This allows for more precise classification of semantically similar vectors into the same category during the clustering process, thereby effectively distinguishing the initial problems of different semantics.

[0071] Density clustering is an algorithm that clusters data based on density reachability between samples. It divides the data space into multiple clusters by finding the largest set of density-connected data points. This algorithm does not require pre-specifying the number of clusters and can automatically identify natural groupings in the data, making it particularly suitable for processing datasets with irregular shapes or uneven density. In text deduplication scenarios, density clustering can effectively identify semantically similar text vectors.

[0072] Density clustering algorithms include Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Ordering Points To Identify the Clustering Structure (OPTICS). OPTICS is an improvement on DBSCAN; instead of directly generating clusters, it generates a reachability order for each data point. Subsequent processing can then yield clusters of varying densities, better handling datasets with uneven density. This application employs the OPTICS clustering algorithm for clustering.

[0073] For all normalized semantic vectors, perform OPTICS clustering. The three core inputs of the OPTICS clustering algorithm include the sample dataset, the upper bound of the neighborhood radius, and the minimum number of core points. The sample dataset consists of all normalized semantic vectors, with each semantic vector representing a sample; the upper bound of the neighborhood radius... This refers to the maximum radius used to define the "neighborhood." The OPTICS clustering algorithm analyzes the distribution of data points within this neighborhood radius. The value can range from 0.5 to 0.8, and can be set flexibly according to actual needs; minimum number of core points The minimum number of samples in the neighborhood required to determine whether a point is a "core point" is set based on the sample size, such as in a medium-sample scenario. The value can be between 10 and 20.

[0074] The two core outputs of the OPTICS clustering algorithm include the sample access order ( ) and reachable distance ( Here, sample access order refers to the sample index sorted from high to low text density. Sample access order ensures that samples with the same semantics (i.e., density contiguous) are arranged continuously. Reachability distance refers to the minimum distance from each sample to its corresponding core point. The core point refers to the sample with the smallest total distance from other samples in the same cluster. Reachability distance reflects the degree of association between the sample and the core semantics.

[0075] It is important to note that although the OPTICS clustering algorithm does not directly return cluster labels, it can be understood that the result of the OPTICS clustering algorithm is a number of clusters arranged in order of semantic density. Therefore, a core point can always be found in each cluster.

[0076] For ease of understanding, in this application, these clusters arranged in semantic density order are referred to as the first cluster, the reachability distance is the reachability distance of each sample in the first cluster, and the sample access order is the semantic density sorting of all samples in all first clusters.

[0077] S14: Based on the binary search framework combined with the semantic verification of the large model, the first cluster is iteratively optimized to obtain the clustering threshold.

[0078] In this application, based on a binary search framework combined with semantic verification of a large model, the first clustering is iteratively optimized to obtain the clustering threshold, including: Based on the binary search framework, iterative search is performed within a preset neighborhood radius, and the intermediate threshold is calculated in each iteration; Update each first cluster based on the intermediate threshold, sample access order, and reachability distance; The median value of the cluster diameter corresponding to the updated first cluster is used as the median diameter, and the two samples corresponding to the median diameter are input into the large model to obtain the semantic verification result. The neighborhood radius range is optimized based on the semantic verification results until the iteration is complete, and the clustering threshold is obtained.

[0079] Binary search is an algorithm for finding a specific element in a sorted array. It quickly locates the target by repeatedly halving the search range. Initializing the binary search boundaries involves setting an initial neighborhood radius as the upper and lower bounds of the search; the upper bound is the maximum neighborhood radius. The lower bound is the minimum neighborhood radius. Preset neighborhood radius range The value is typically [0.1, 0.8], but can be set according to actual clustering needs, such as [0.2, 0.8] or [0.2, 0.9]. A precision threshold is also preset. As the termination condition for the iteration, the precision threshold can be set according to the actual requirements for the clustering effect, such as 0.01 or 0.005. A preset upper limit for the number of iterations is set to control the maximum number of times the binary search is executed, preventing infinite loops caused by setting the precision threshold too low or special data distribution. It can usually be set according to the data scale and computing resources, such as 20 or 30 times. When the number of iterations reaches the upper limit, the search is forcibly terminated even if the precision threshold condition is not met, and the current best result is returned.

[0080] In each iteration, the median value of the current neighborhood radius is calculated as an intermediate threshold. This intermediate threshold serves as a crucial criterion for determining whether samples belong to the same cluster. Specifically, the formula for calculating the intermediate threshold is as follows: Formula 3; in, The intermediate threshold; The minimum neighborhood radius; The maximum neighborhood radius.

[0081] In each iteration, after calculating the current intermediate threshold, based on the sample access order and reachability distance, consecutive samples whose reachability distance does not exceed the intermediate threshold are selected. These consecutive samples are labeled into the same cluster, and cluster labels are generated, resulting in the updated first cluster. In this way, similar samples can be effectively grouped together to form clusters with clear semantic and spatial features, providing an accurate foundation for subsequent text deduplication operations.

[0082] For each updated first cluster, its cluster diameter and pivot diameter are calculated. Specifically, the cosine distance of each sample pair within the cluster is calculated, and the largest cosine distance is selected as the cluster diameter for that cluster. All calculated cluster diameters are sorted by size, and the median value is selected as the pivot diameter, which represents the average semantic dispersion in the scenario where the current pivot threshold is used as the cluster threshold.

[0083] Extract the two samples corresponding to the middle diameter (i.e., the two samples with the largest cosine distance in the first cluster to which the middle diameter belongs, indicating the greatest semantic difference). Input these two samples into the Large Language Model (LLM) for semantic verification. Based on the semantic verification result of "true" or "false" returned by the LLM, determine whether the core semantics of the two samples are consistent. The choice of the large model can be flexibly adjusted according to actual needs and application scenarios. For example, large models such as GLM-4.6, GPT-4, or ERNIE Bot can be used. The first large model mentioned above can be used, or another pre-trained large model can be used. There is no specific limitation on this.

[0084] By leveraging the deep semantic understanding capabilities of large models, it is possible to accurately determine whether two input samples are consistent at the core semantic level, thus providing a reliable basis for subsequent text deduplication decisions. If the semantic verification result is "true," it indicates that the two samples are consistent in core semantics, although they may differ in expression or specific word choice; if the result is "false," it means that the two samples have significant differences in core semantics and should not be considered as duplicate initial problems.

[0085] In this application, the neighborhood radius range is optimized based on the semantic verification results until the iteration is completed, and the clustering threshold is obtained, including: In response to the semantic verification result being semantically consistent, the minimum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold; In response to a semantic inconsistency result, the maximum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold. The iteration continues until the neighborhood radius does not exceed the preset precision threshold or the preset number of iterations is reached. Once the iteration is complete, the minimum neighborhood radius is used as the clustering threshold.

[0086] In this application, when the semantic verification result returned by the large model is "true", meaning that the two samples are semantically consistent, it indicates that the current intermediate threshold is too fine. The intermediate threshold can be expanded to merge more similar initial questions. The way to expand the intermediate threshold is to update the minimum neighborhood radius in the current neighborhood radius range to the current intermediate threshold. For example, if the current neighborhood radius range is [0.1, 0.8], and the current intermediate threshold is (0.1+0.8) / 2=0.45, updating the minimum neighborhood radius in the neighborhood radius range to the intermediate threshold will result in an updated neighborhood radius range of [0.45, 0.8], and an updated intermediate threshold of (0.45+0.8) / 2=0.625.

[0087] When the semantic verification result returned by the large model is "false," meaning the semantics of the two samples are inconsistent, it indicates that the current intermediate threshold is too coarse. The intermediate threshold can be narrowed to separate the initial problems with different semantics. Narrowing the intermediate threshold involves updating the maximum neighborhood radius within the current neighborhood radius range to the current intermediate threshold. For example, if the current neighborhood radius range is [0.1, 0.8], and the current intermediate threshold is (0.1 + 0.8) / 2 = 0.45, updating the maximum neighborhood radius within the current neighborhood radius range to the intermediate threshold will result in an updated neighborhood radius range of [0.1, 0.45] and an updated intermediate threshold of (0.1 + 0.45) / 2 = 0.275.

[0088] After determining the neighborhood radius, calculate the intermediate threshold and repeat steps S22-S24 until the iteration is complete. The minimum neighborhood radius at the end of the iteration is then taken as the optimal clustering threshold. The criteria for determining the completion of the iteration include that the neighborhood radius does not exceed a preset precision threshold, i.e. The iteration is considered complete when either condition is met, or the number of iterations reaches the preset number (e.g., 20 times). The minimum neighborhood radius obtained at this time can more accurately reflect the semantic similarity between the initial questions, and thus serve as the optimal clustering threshold for deduplication of the initial questions in mixed Chinese and English text.

[0089] During iterative clustering, the neighborhood radius is automatically adjusted based on the semantic similarity between the initial questions, thereby obtaining a more accurate clustering threshold. This allows for better adaptation to the initial questions of mixed Chinese and English texts with different semantic densities, avoiding "over-clustering" or "under-clustering" and improving the accuracy and efficiency of deduplication.

[0090] S15: Update the first cluster based on the clustering threshold to obtain multiple second clusters.

[0091] In this application, the optimal clustering threshold is obtained through the above iteration. The cluster labels (IDs) are re-extracted based on the sample access order and reachability distance to obtain the final clustering results (Clusters, in the format of "cluster ID → sample index list", that is, each cluster uses its cluster ID as the key and the index list of its samples as the value). These clusters are recorded as the second cluster.

[0092] These second clusters, generated based on the optimal clustering threshold, can more accurately segment the initial questions in mixed Chinese and English text. The initial questions within each second cluster exhibit high semantic similarity, while the semantic differences between initial questions in different second clusters are relatively large. In this way, duplicate initial question content can be effectively distinguished, thereby achieving the goal of deduplicating initial questions in mixed Chinese and English text and improving the accuracy and efficiency of initial question processing.

[0093] S16: Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, deduplication is performed on each second cluster to obtain a deduplicated list of questions.

[0094] In this application, based on a preset semantic filtering algorithm and a preset representation filtering algorithm, duplicates are removed from each second cluster to obtain a deduplicated list of questions, including: For each second cluster, calculate the mean of all samples and normalize it to obtain the cluster center; Based on the preset semantic filtering algorithm and cluster centers, several candidate samples are selected from all the second clusters; The similarity between candidate samples is calculated based on a preset expression filtering algorithm, and several final samples are selected. The initial questions corresponding to several final samples are converted into a structured list, resulting in a deduplicated list of questions.

[0095] In this application, for each second cluster, the arithmetic mean of all normalized semantic vectors within the cluster is calculated, and then this arithmetic mean is L2 normalized to transform it back into a unit vector. Assume that there are m samples in a certain cluster, and the normalized semantic vectors are v1, v2, …, v m (each v) i ∈R d And ||v i ||2 = 1, R d (i.e., d-dimensional real space), calculate the average value: μ = 1 / m · Σ i=1 m v i μ is the average value of all samples within the cluster. Then, μ is L2 normalized, i.e., μ no m = μ / ||μ||2, μ no m This is the normalized mean. Using this normalized mean as the core semantic representation of the cluster, denoted as the cluster center, ensures that the cluster center falls within the "geometric center" of the semantic space while maintaining the comparability of cosine distance.

[0096] In this application, based on a preset semantic filtering algorithm and cluster centers, several candidate samples are selected from all second clusters, including: For each sample in each second cluster, calculate the cosine distance between the sample and the cluster centers of the other second clusters, and calculate the average cosine distance. In each second cluster, the top few samples with the largest average cosine distance are selected as candidate samples.

[0097] Specifically, for each sample in each second cluster, the cosine distance between that sample and the cluster centers of all other second clusters is calculated to accurately quantify the semantic similarity between the sample and different cluster centers. All calculated cosine distances are averaged to obtain the mean cosine distance for that sample, comprehensively reflecting the degree of deviation of the sample from the overall cluster centers. For each second cluster, samples are sorted according to the magnitude of the mean cosine distance across all samples in that second cluster, and the top few samples with the largest distances are selected as candidate samples due to their significant differences from the cluster centers, thereby improving the semantic uniqueness of deduplication.

[0098] In one embodiment, assuming there are 50 second clusters, each with 20 samples, then there are a total of 1000 samples in all the second clusters. For any sample x among these 1000 samples, calculate the cosine distance between x and the cluster centers of the other 49 second clusters, thus obtaining 49 cosine distances. Then, take the average of these 49 cosine distances. Where c(x) is the cluster ID to which x belongs. For the k-th cluster center, Let x be the mean cosine distance. For each of the 20 samples in the second cluster, calculate a corresponding mean cosine distance, resulting in 20 mean cosine distances {D(xi)}. Sort the samples according to their mean cosine distances and select the samples with the largest distances. Sample of names ( The maximum number of samples to be retained for each second cluster is usually 2-3, depending on actual needs. If the value is 3, then 2×3=6 candidate samples need to be selected as the most representative or most dispersed. These can be obtained by calculating the maximum edge or the average edge.

[0099] The formula for calculating the maximum edge of 6 candidate samples is as follows: Formula 4; in, The cosine similarity between sample x and sample y. Let S be the cosine distance between two samples; S is a subset of 6 candidate samples; X is a set of 20 samples. A subset of 6 candidate samples is selected to maximize the difference between the two most similar samples within each subset.

[0100] The formula for calculating the 6 candidate samples using the average edge method is as follows: Formula 5; in, The cosine similarity between sample x and sample y. Let S be the cosine distance between two samples; S is a subset of 6 candidate samples; X is a set of 20 samples. This represents the average difference between each pair of samples, i.e., the average cosine distance between all sample pairs within subset S. To maximize the average difference between all sample pairs within it, a subset consisting of 6 candidate samples is selected.

[0101] The two screening methods mentioned above are essentially about selecting 6 samples from 20 samples, so that the pair of samples that are semantically "most similar" among these 6 samples is also as "least similar" as possible. In other words, they aim to maximize the cosine distance between each pair of samples. That is, in all the second clusters, the top few samples with the largest average cosine distance are selected as candidate samples.

[0102] By employing the semantic uniqueness screening method described above, samples with significant semantic differences can be effectively selected from a large number of samples. This screening method not only considers the direct similarity between samples but also ensures the semantic diversity and uniqueness of the selected samples by maximizing the mean cosine distance, thereby improving the accuracy and efficiency of deduplication.

[0103] In this application, a preset expression screening algorithm is used to calculate the expression similarity between candidate samples, and a number of final samples are selected, including: Based on a character-level n-gram model, the representation similarity between candidate samples is calculated. In response to the existence of candidate samples whose expression similarity does not exceed a preset similarity threshold, several candidate samples whose expression similarity does not exceed the preset similarity threshold are used as the final samples. In response to the absence of samples whose representation similarity does not exceed a preset similarity threshold, the candidate sample with the largest average cosine distance is selected as the final sample.

[0104] Specifically, the candidate samples are screened for representational diversity. A character-level n-gram model is used to calculate the representational similarity between the candidate samples, and the candidate samples with similarity not exceeding a preset similarity threshold are selected as the final samples to ensure that the representations are not duplicated. Here, the value of n can be 2 or 3, and the preset similarity threshold can be 0.3, 0.35 or 0.4, etc., which can be flexibly set according to actual needs.

[0105] In some embodiments, there may be no candidate samples that can satisfy the diversity of expression, that is, there are no samples whose expression similarity does not exceed a preset similarity threshold. In this case, in order to avoid clustering without representative samples, at least one candidate sample with the most unique semantics is retained as the final sample. This candidate sample with the most unique semantics is the candidate sample with the largest average cosine distance.

[0106] It ensures semantic consistency within clusters through cosine distance filtering and preserves expression diversity through n-gram filtering, solving the problems of "loss of expression during deduplication" or "retention of duplicate expression". Furthermore, it uses character-level n-gram processing for mixed Chinese and English text, which can preserve special English symbols (such as "-" and "_"), thereby avoiding misjudgment of expression similarity due to differences in language structure.

[0107] For the obtained final samples, organize the initial questions corresponding to these final samples into a structured list and store it, such as storing it in JSON format. This structured list is the deduplicated question list.

[0108] The structured list stored in JSON format has good readability and scalability, which facilitates further processing and analysis of the deduplicated problem list.

[0109] In this application, after obtaining a question list containing a preset number of initial questions, full deduplication of the question list can be performed through steps S13-S16 above to further ensure the uniqueness and accuracy of the initial questions in the question list. Besides the methods described in steps S13-S16, full deduplication can also be achieved by comparing similarity using methods such as cosine similarity, Euclidean distance, dot product, or Manhattan distance to remove duplicate initial questions with high similarity. The method can be flexibly selected according to actual needs, and no specific limitation is placed on the full deduplication method.

[0110] S20: Update the identity of the first major model to a structured query statement generation agent, and the structured query statement generation agent calls various tool functions to generate a corresponding list of structured query statements based on the question list and the private domain database.

[0111] In this application, prompt words are generated based on preset structured query statements, giving the primary model a new identity and making it a structured query statement generation agent. The preset structured query statement generation prompt words can be set according to different business scenarios and needs. For example, in a financial business scenario, it can be set to "generate a structured query statement conforming to financial database query specifications based on the natural language questions input by the user"; in a medical business scenario, it can be set to "generate a structured query statement adapted to a medical database based on natural language questions related to the patient's condition". This flexible setting allows the structured query statement generation agent to better adapt to diverse business needs.

[0112] After updating the identity of the first major model to a structured query statement generation agent, the initial questions from the fully deduplicated question list obtained in S10 are input into the structured query statement generation agent. The structured query statement generation agent can also autonomously explore the private database by calling the aforementioned tool functions, including consulting relevant information to assist the first major model in database reading, and generating structured query statements corresponding to the initial questions based on the input initial questions. For example, for a question about "querying the names of users in the user table whose age is greater than 30", the structured query statement generation agent can combine database schema information and use Select execution tools to generate the corresponding structured query statement "SELECT user_name FROM users WHERE age>30".

[0113] The generated structured query statements undergo syntax validation. A pre-defined syntax rule base can be used to perform a preliminary syntax check, determining whether keywords and symbols conform to the rules. If syntax errors are found, the error locations are marked, and corresponding modification suggestions are provided based on common syntax error types. Simultaneously, considering the specific syntax requirements of the private domain database, the applicability of the structured query statements in the private domain database environment is further verified to ensure correct execution in the target database and avoid query failures due to syntax incompatibility.

[0114] Generate corresponding structured query statements for all initial questions in the question list after full deduplication, forming a list of structured query statements that are mapped one-to-one with the question list.

[0115] To ensure that the structured query statements generated for the initial question have good recall and precision (F1 score), a second validation can be performed on the structured query statements in the list.

[0116] In this application, after generating the list of structured query statements, the following is also included: The identity of the first major model is updated to that of an adversary agent. Based on each structured query statement in the list of structured query statements, the adversary agent calls various tool functions to generate several simulation problems corresponding to the structured query statements based on the private domain database. Calculate the second similarity between each simulated problem and its corresponding initial problem; In response to a second similarity score exceeding a second preset similarity threshold, the structured query statement and its corresponding initial question are used as the final question-answer pair. A training dataset was built based on all the final question-answer pairs to train the second-largest model.

[0117] In this application, based on preset opponent prompts, the first main model is reassigned a new identity, becoming an opponent agent. The preset opponent prompts include the task of assigning the opponent agent to generate several simulated questions corresponding to a structured query statement based on a database scheme and the structured query statement. These simulated questions can be answered using data retrieved from the structured query statement. These simulated questions are not only sufficient in number but also semantically and expressively close to real-world scenarios, providing a rich and accurate data foundation for subsequent similarity calculations and the final determination of question-answer pairs.

[0118] In this application, the adversary agent can also call the aforementioned tool functions to autonomously explore the private database, including consulting relevant information to assist the primary model in database reading, and generating several simulated questions corresponding to the input structured query statement. The number of simulated questions can be set in the prompt words, for example, to no less than 20, but is not limited here.

[0119] For each structured query, the simulated questions generated by the adversary agent are compared with the initial questions generated by the question-generating agent after full deduplication. A second similarity score is calculated between each simulated question and the initial question; this second similarity score is the sum of the similarities between each simulated question and the initial question. If the second similarity score is higher than a second preset similarity threshold, the corresponding structured query and its initial question are considered a final question-answer pair. If the second similarity score is not higher than the second preset similarity threshold, the corresponding structured query and its initial question are marked for manual review. After manual review, it is determined whether they can be considered a final question-answer pair.

[0120] Similarity can be calculated using cosine similarity, or other similarity methods such as Euclidean distance, dot product, or Manhattan distance. In addition, each simulated problem and its corresponding initial problem can be input into a reranking model for sorting to detect similarity. Accordingly, the second preset similarity threshold can also be flexibly set, without specific limitations here.

[0121] The resulting set of all final question-answer pairs is used to construct a training dataset for training the second-largest model. This training dataset ensures data diversity and representativeness, covering structured queries of varying complexity and their corresponding natural language questions across different domains. Furthermore, the training dataset can be preprocessed, including data cleaning and format standardization, to improve the effectiveness and efficiency of subsequent model training.

[0122] Unlike existing technologies where private domain data differs significantly from public data in terms of terminology and storage structure, making it difficult for large models trained on public data to directly perform effective nl2sql functionality in real production environments, this application provides a private domain data distillation method for converting natural language to structured query language. This method constructs separate question-generating agents and structured query statement-generating agents. These agents autonomously explore the private domain database by calling various utility functions, generating initial questions and corresponding structured query statements specific to the private domain database. This results in a training dataset that more closely matches the structure and business needs of the private domain database, effectively improving the nl2sql capabilities of the private domain data distillation model trained on the training dataset in real production environments. This method not only solves the model adaptation problem caused by the differences between public and private data but also achieves in-depth mining and efficient utilization of the private domain database through the autonomous exploration mechanism of the agents.

[0123] S30: Based on the list of questions and the corresponding list of structured query statements, train a second large model with a small number of parameters to obtain the private domain data distillation model.

[0124] In this application, the second largest model with small parameters typically refers to models with fewer than 7 billion (7B) parameters but whose capabilities approach or reach the level of "large models." These models have significant advantages in terms of lightweight design and low deployment costs, such as the Qwen-7B and Qwen-32B from the Tongyi Qianwen series and the GLM-Z1-32B-0414 from Zhipu GLM. These models have relatively small parameter sizes, making them easy to train and deploy while maintaining high performance. They possess powerful natural language understanding and generation capabilities, effectively handling natural language problems and generating corresponding structured query statements. By training with these small-parameter models, private domain data distillation models can be obtained more efficiently, thereby improving the performance and efficiency of the entire system.

[0125] The training dataset, constructed from multiple final question-answer pairs obtained from the question list and the corresponding structured query statement list, is used to train a second large model with a small number of parameters. Specifically, reinforcement learning training methods can be used, such as Group Relative Policy Optimization (GRPO) or Proximal Policy Optimization (PPO), without any specific limitations.

[0126] During training, the initial question from the final question-answer pair, the database scheme, and relevant information to assist the first model in database reading are input into the second-largest model. The second-largest model predicts the corresponding structured query statement based on the initial question in the final question-answer pair, and provides feedback and adjustments based on the difference between the predicted structured query statement and the structured query statement in the final question-answer pair. Through multiple iterations of training, the second-largest model gradually learns how to accurately convert natural language into structured query statements suitable for private domain databases, ultimately obtaining a private domain data distillation model.

[0127] In this application, after obtaining the private domain data distillation model, it also includes: In response to the input natural language query question targeting the private domain database, the private domain data distillation model parses the natural language query question and outputs the corresponding structured query statement.

[0128] In this application, the trained private domain data distillation model can understand and analyze input natural language queries and output corresponding structured query statements suitable for private domain databases, thereby retrieving the required data from the private domain database. Furthermore, when faced with natural language query expressions never encountered before, it can still accurately parse and output corresponding structured query statements based on learned rules and patterns. Moreover, this private domain data distillation model exhibits generalization ability, handling various natural language query questions with different expressions but similar semantics. It also performs exceptionally well in handling natural language queries with complex logical relationships, accurately capturing key information and generating structured query statements that meet the requirements of private domain database queries, thus efficiently retrieving the required data from the private domain database.

[0129] Unlike existing technologies where large-scale parameter models have extremely high computational requirements, deploying such models privately in various enterprise data application scenarios faces significant cost challenges. Even with deployment techniques such as model quantization, it often fails to meet the business's demand for efficient and accurate data queries, thus limiting the widespread application of nl2sql technology in real-world business scenarios. This application provides a private domain data distillation method for natural language to structured query language. It uses a large-scale parameter first model to generate a training dataset for a private database, and then trains a small-scale parameter second model, enabling the second model to achieve online inference. This significantly reduces the reliance on computational power. This allows for easy private deployment of the private domain data distillation model in various enterprise data application scenarios without facing enormous cost challenges. Furthermore, the private domain data distillation model can efficiently and accurately handle natural language query problems without relying on complex techniques such as model quantization, meeting the business's requirements for efficiency and accuracy in data queries, and greatly expanding the wide application scope of nl2sql technology in real-world business scenarios.

[0130] See Figure 2 , Figure 2 This is a schematic diagram illustrating the data distillation process implemented through the interaction of various agents in an embodiment of the private domain data distillation method from natural language to structured query language provided in this application. The question-generating agent autonomously explores the private domain database using constructed tool functions to generate a question list. The generated question list is then fully deduplicated, resulting in a deduplicated question list, which is input into the structured query statement generating agent. The structured query statement generating agent autonomously explores the private domain database using constructed tool functions, generating corresponding structured query statements based on each initial question in the question list, resulting in a structured query statement list, which is input into the adversary agent. The adversary agent autonomously explores the private domain database using constructed tool functions, generating several simulated questions corresponding to the input structured query statements. The second similarity between each simulated question and its corresponding initial question is compared to determine the final question-answer pair, thus constructing a training dataset. Finally, the training dataset is used to train a second large-scale open-source model with a small number of parameters, resulting in a private domain data distillation model for the private domain database.

[0131] This application provides a private domain data distillation apparatus for converting natural language to structured query language, see reference. Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the private domain data distillation apparatus for converting natural language to structured query language provided in this application. The private domain data distillation apparatus for converting natural language to structured query language includes: The first generation module 10 is used to assign the identity of the first large model with tool calling ability as a problem generation agent, and the problem generation agent calls the pre-built tool functions to generate a problem list based on the private domain database; The second generation module 20 is used to update the identity of the first model to a structured query statement generation agent, and the structured query statement generation agent calls each of the aforementioned tool functions to generate a corresponding structured query statement list based on the question list and the private domain database. Training module 30 is used to train a second large model with a small number of parameters based on the question list and the corresponding list of structured query statements, thus obtaining the private domain data distillation model.

[0132] The first generation module 10, the second generation module 20, and the training module 30 interact to realize the process of private domain data distillation from natural language to structured query language. You can refer to the specific description of steps S10 to S30 above. Where there are repetitions, they will not be repeated here.

[0133] See Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the storage medium provided in this application.

[0134] The storage medium 400 stores program data 410, which, when executed by the processor, implements, as follows: Figure 1 The steps of the described method for private domain data distillation from natural language to structured query language.

[0135] The program data 410 is stored in a storage medium 400 and includes several instructions for causing a network device (which may be a router, personal computer, server, or other network device) or processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0136] Optionally, the storage medium 400 can be any medium that can store program data, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), disk, or optical disc.

[0137] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application.

[0138] The device 500 includes a processor 520 and a memory 510 interconnected. The memory 510 stores a computer program. When the processor 520 executes the computer program, it implements the steps of the private domain data distillation method for converting natural language to structured query language as described above.

[0139] Unlike existing technologies, this application discloses a private domain data distillation method and apparatus for converting natural language to structured query language. This method assigns the identities of a question-generating agent and a structured query statement-generating agent to a first large-scale model with large parameters, and pre-constructs various tool functions that can be called by the first large-scale model. This allows the first large-scale model to learn and generate a list of questions and corresponding list of structured query statements in a private domain database, which is then used to train a second large-scale model with smaller parameters, resulting in a private domain data distillation model. This model enables online inference from natural language to structured query language. The computational resources required for inference by the large model with smaller parameters are significantly reduced compared to the large model with larger parameters, thereby reducing hardware costs and energy consumption. Simultaneously, because the data used to train the private domain data distillation model is more closely aligned with the structure and business needs of the private domain database, the final trained private domain data distillation model can more accurately convert natural language into structured query statements suitable for that private domain database, improving the accuracy and efficiency of the natural language to structured query language conversion.

[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the storage medium embodiments and computer device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0141] This application can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, distributed computing environments including any of the above systems or devices, etc.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for private domain data distillation from natural language to structured query language, characterized in that, include: The first major model with tool calling capabilities is given the identity of a problem-generating agent, and the problem-generating agent calls pre-built tool functions to generate a problem list based on a private domain database; The identity of the first large model is updated to a structured query statement generation agent, and the structured query statement generation agent calls each of the aforementioned tool functions to generate a corresponding structured query statement list based on the question list and the private domain database; Based on the aforementioned list of questions and the corresponding list of structured query statements, a second large model with a small number of parameters is trained to obtain the private domain data distillation model.

2. The private domain data distillation method for converting natural language to structured query language according to claim 1, characterized in that, The problem-generating agent calls various pre-built tool functions to generate a list of problems based on the private domain database, including: In response to the input scene and character prompts, the question-generating agent parses the scene and character prompts and extracts scene and character information; Based on the scene and character information, the tool functions are invoked to autonomously explore the private database and generate a list of questions corresponding to the scene and character prompts.

3. The method according to claim 1, characterized in that, Before generating the aforementioned list of issues, the following is also included: For the problem-generating agent, several initial problems are generated in a single instance, and the first similarity between the initial problems is calculated. In response to the first similarity being higher than a first preset similarity threshold, the corresponding initial question is deleted; In response to the initial number of questions generated multiple times by the question-generating agent reaching a preset number, the question list is constructed based on the preset number of initial questions.

4. The method according to claim 1, characterized in that, After generating the aforementioned list of issues, the following is also included: Each of the initial questions in the question list is converted into a semantic vector, and the first cluster is obtained based on the density clustering algorithm; Based on the binary search framework combined with the semantic verification of the large model, the first clustering is iteratively optimized to obtain the clustering threshold; The first cluster is updated based on the clustering threshold to obtain multiple second clusters; Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, duplicates are removed from each of the second clusters to obtain the deduplicated list of questions.

5. The method according to claim 3, characterized in that, After generating the list of structured query statements, the process also includes: The identity of the first large model is updated to that of an adversary agent, and the adversary agent, based on each structured query statement in the structured query statement list, calls each of the tool functions to generate several simulation problems corresponding to the structured query statements based on the private domain database; Calculate the second similarity between each of the simulated problems and the corresponding initial problems; In response to the second similarity being higher than a second preset similarity threshold, the structured query statement and the corresponding initial question are used as the final question-answer pair; A training dataset is constructed based on all the final question-answer pairs to train the second major model.

6. The method according to claim 1, characterized in that, After obtaining the private domain data distillation model, the following is also included: In response to an input natural language query question for the private domain database, the private domain data distillation model parses the natural language query question and outputs the corresponding structured query statement.

7. The method according to claim 1, characterized in that, The utility functions are encapsulated based on various database interface tools. The utility functions include at least one or more of the following: database schema generation tool, JSON field parsing tool, selection and execution tool, database switching tool, database information acquisition tool, database name acquisition tool, and several common tools. Before assigning the identity of the primary model with tool-calling capabilities as the problem-generating agent, it also includes: The first large model declares the utility functions in its calling interface to understand and call the utility functions.

8. A private domain data distillation apparatus for converting natural language to structured query language, characterized in that, include: The first generation module is used to assign the identity of the first large model with tool calling capabilities as a problem generation agent, and the problem generation agent calls pre-built tool functions to generate a problem list based on a private domain database; The second generation module is used to update the identity of the first large model to a structured query statement generation agent, and the structured query statement generation agent calls each of the tool functions to generate a corresponding structured query statement list based on the question list and the private domain database. The training module is used to train a second large model with a small number of parameters based on the list of questions and the corresponding list of structured query statements, so as to obtain the private domain data distillation model.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the private domain data distillation method for natural language to structured query language as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the private domain data distillation method for converting natural language to structured query language as described in any one of claims 1-7.

Images