Systems and methods for generating synthetic data and optimizing large language model performance

By using a semi-automated method of generating synthetic data and continuous validation, combined with the DSPy framework to optimize large language models, the problem of inaccurate output of large language models in new domains is solved, achieving efficient and low-cost performance optimization, which is suitable for financial systems.

CN122471074APending Publication Date: 2026-07-28HSBC SOFTWARE DEV (GUANGDONG) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610825550.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-08
Publication Date
2026-07-28

AI Technical Summary

Technical Problem

Large language models lack specific domain-specific prior experience in new fields, making it difficult for output results to fit the scenario and be accurate. Traditional optimization methods are costly in terms of manpower and difficult to scale, while fine-tuning LLMs is resource-intensive and slows down the development process.

Method used

This paper provides a system and method that generates synthetic data and optimizes the performance of large language models. It utilizes semi-automatic generation of synthetic data and continuous validation, combines the DSPy framework to optimize the LLM program, reduces the reliance on full fine-tuning, and employs online and offline learning for continuous optimization.

Benefits of technology

It improves the performance of large language models in specific domains, alleviates the cold start problem, reduces labor costs, and enhances the flexibility and adaptability of the system, making it suitable for complex systems in the financial field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122471074A_ABST
    Figure CN122471074A_ABST
Patent Text Reader

Abstract

Provided are systems and methods and computer-implemented systems for generating synthetic data and optimizing performance of large language models, by integrating semi-automatic synthetic data generation, prompt optimization and continual learning into a unified system, without full fine-tuning of the model, to improve performance of the large language model. The cold start problem of the large language model is solved through a multi-stage process, including: generating a diversified parameter set and natural language sentences; constructing synthetic user queries using feature descriptions, parameter sentences and constraint instructions; bootstrapping with a small amount of samples with the help of a compiler, and iteratively evaluating instruction-example combinations to optimize the large language model module; and combining real-time data and synthetic data to complete continual validation and optimization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the field of large oracle model technology. For example, it relates to systems and methods for generating synthetic data and optimizing the performance of large language models in a target domain. It also relates to computer-implemented systems for generating synthetic data and providing continuous validation and optimization for large language model programs. Background Technology

[0002] Integrating Large Language Models (LLMs) into backend systems represents a significant advancement in the field of intelligent user input processing. However, LLMs can encounter a "cold start" problem, struggling to make accurate predictions or generate compliant content when model prompts are poorly set or the model lacks prior experience in a specific domain. When LLMs are first introduced into a new domain or tasked with generating content within a specific domain, their unfamiliarity with the domain's terminology, context, and subtle semantic differences often results in outputs that are not context-appropriate or accurate.

[0003] Traditional LLM optimization methods require manual optimization of customized instructions, coupled with demonstration or few-sample examples, to ensure the model's performance. This approach is labor-intensive and difficult to scale, especially for complex systems with numerous subtasks. Furthermore, fine-tuning LLM to adapt to the high-frequency updates in dynamic fields such as finance requires continuous learning and iteration, which not only consumes significant resources but also slows down development progress.

[0004] Given that human language and trader query needs encompass various financial functions and a large number of combinations of financial parameters, model optimization requires rich and diverse datasets. As LLM deployments continue to scale up and their architectures become increasingly complex, and as manual optimization becomes unsuitable and unsustainable, there is a further demand for automated optimization of large language model programs. Summary of the Invention

[0005] The following summary is provided to illustrate the technical concept in a simplified form; the relevant details will be further described in the specific embodiments below. This summary is not intended to limit the key or essential features of the claimed subject matter, nor is it intended to restrict the scope of protection of the claimed subject matter.

[0006] The purpose of some embodiments of this disclosure is to provide a system and method for generating synthetic data and optimizing the performance of large language models (LLMs) in a target domain, as well as a computer-implemented system for generating synthetic data and providing continuous validation and optimization for large language model programs. This can be used to generate synthetic data, optimize large language models (LLMs), and conduct continuous validation in financial applications. Some embodiments of this disclosure integrate semi-automatic synthetic data generation, prompt word optimization, and continuous learning into a unified system, improving the performance of LLMs without requiring comprehensive fine-tuning of the model.

[0007] In a first aspect, some embodiments of this disclosure provide a system for generating synthetic data and optimizing the performance of a large language model in a target domain. The system includes: a preparation module configured to generate multiple parameter sets for target domain features, wherein each parameter set includes selected values ​​for parameters related to the target domain features; for each parameter set, generating multiple natural language phrases representing the parameters; and receiving constraint instructions; a generation module configured to combine feature descriptions, the multiple natural language phrases, and the constraint instructions and input them into the large language model to generate a synthetic user query statement; and a training module configured to use the synthetic user query statement as training data to optimize a self-optimizing language program module, wherein optimization includes extracting bootstrap samples from the training data and iteratively evaluating instruction-example combinations to determine the optimal prompt configuration.

[0008] Optionally, the preparation module is further configured to extract parameter descriptions for the target domain features, wherein the parameter descriptions include explanations of the parameters and their value ranges.

[0009] Optionally, the generation module employs a similarity metric, which includes at least one of ROUGE score and embedding similarity, wherein the ROUGE score measures the lexical overlap between texts, and the embedding similarity captures semantic relevance by comparing the vector representations of the query statements.

[0010] Optionally, the training module optimizes the self-optimizing language program module, which includes a parameter extraction module configured to parse user query statements to identify and extract parameters to be transmitted to the application programming interface.

[0011] Optionally, the training module uses the ReAct framework to optimize the self-optimizing language program module, which enables the large language model to perform inference and execution actions in a collaborative manner.

[0012] Optionally, the training module is further configured to perform continuous validation using online learning and offline learning, wherein the online learning incrementally processes data arriving in real time, and the offline learning processes data collected in batches in advance.

[0013] Optionally, the continuous verification includes: using a feature recognition module to classify user queries into corresponding domain categories; when the classification is incorrect, the verification module determines the error type; using a parameter extraction module to extract parameters from the correctly classified user queries; and

[0014] Improve feature and parameter descriptions based on validation results.

[0015] Secondly, some embodiments of this disclosure provide a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The method includes: in a preparation phase, generating multiple parameter sets for target domain features, wherein each parameter set includes selected values ​​for parameters related to the target domain features; for each parameter set, generating multiple natural language phrases representing the parameters, and receiving constraint instructions; in a generation phase, the one or more processors combine feature descriptions, the multiple natural language phrases, and the constraint instructions and input them into a large language model to generate a synthetic user query statement; and in a training phase, the one or more processors use the synthetic user query statement as training data to optimize a self-optimizing language program module, wherein optimization includes extracting bootstrap samples from the training data and iteratively evaluating instruction-example combinations to determine the optimal prompt configuration.

[0016] Optionally, the preparation phase may further include extracting parameter descriptions of the target domain features from the feature management system, wherein the parameter descriptions include explanations of the parameters, data types, and value ranges.

[0017] Optionally, the preparation phase generates the plurality of parameter sets by selecting a subset of parameters from each parameter set based on a probability distribution.

[0018] Optionally, the preparation phase receives constraint instructions, which include at least one of the following: instructions to simplify query statements, instructions to mimic the query style of a specific user type, and instructions to use declarative sentences.

[0019] Optionally, the training phase further includes: providing candidate instructions by a prompting optimizer, the candidate instructions describing the program task and differing in structure and style; iteratively combining the candidate instructions with candidate example samples; and evaluating all combinations based on a test dataset to determine the optimal configuration.

[0020] Optionally, the training phase further includes: saving the optimized self-optimizing language program module to a local file; and integrating the optimized self-optimizing language program module into the production system.

[0021] Optionally, the training phase further includes: using a feature recognition module to classify the input user query statement into the corresponding domain category; using a parameter extraction module to extract parameters from the input user query statement; verifying the accuracy of the extracted parameters; and based on the verification results, using an optimization program based on a large language model to automatically improve the feature description and parameter description.

[0022] Thirdly, some embodiments of this disclosure provide a computer-implemented system for generating synthetic data and providing continuous verification and performance optimization for a large language model program. The system includes: a preparation unit configured to generate parametric phrase examples and constraint instruction examples for target domain features; and to extract feature descriptions and parametric descriptions from a feature management system; a generation unit configured to receive input signals from the preparation unit via a first data path, iteratively generate synthetic user query statements by combining the feature descriptions with the parametric phrase examples and the constraint instruction examples, and add different synthetic user query statements to a query pool based on similarity comparison; a training unit configured to receive input signals from the generation unit via a second data path; optimize a self-optimizing language program module using the synthetic user query statements as training data, wherein the optimization compiler is prompted to bootstrap few-sample examples and iteratively evaluate instruction-example combinations; and perform continuous verification using online and offline learning to identify errors and automatically improve the feature descriptions and parametric descriptions; and a feedback path configured to feed verification signals back from the training unit to the preparation unit to achieve iterative improvement of the feature descriptions and parametric descriptions.

[0023] Optionally, the performance of the large language model can be evaluated by using a customized synthetic dataset as training data and by generating few-shot examples as demonstration examples. The generation unit and the training unit work together to solve the cold start problem of the large language model.

[0024] Optionally, the training unit optimization includes a modular architecture of self-optimizing language program modules, wherein each system component is constructed as a separate sub-process capable of independent optimization, including a feature recognition sub-process and a parameter extraction sub-process.

[0025] Optionally, the training unit further includes: a feature recognition subprocess configured to process user query statements and classify the user query statements into corresponding domain categories; an error analysis subprocess configured to analyze the error when the classification is incorrect to determine whether the error is caused by inappropriate feature descriptions or ambiguity in the query statement; and a parameter extraction subprocess configured to extract parameters for input to the application programming interface when the classification is correct.

[0026] Optionally, the target domain features include financial features, and the workflow is configured to process financial query statements, which identify the correct financial features to locate the corresponding application programming interface (API), extract parameters related to the user query statement, obtain calculation results from the corresponding API, and generate a summary response.

[0027] Optionally, the preparation unit uses a feature management system to maintain feature descriptions and parameter descriptions. The feature management system serves as a knowledge base, wherein each feature description describes the function and usage instructions of the corresponding application programming interface, and wherein the parameter description specifies the input required for each application programming interface, the input including parameter name, data type, and an explanation of the purpose of each parameter.

[0028] The technical features and other technical features of the present invention, as well as their corresponding advantages, will become apparent from the following detailed description and accompanying drawings. It should be understood that the above-described invention, the following detailed embodiments, and the accompanying drawings are merely illustrative and do not limit the claimed aspects. Attached Figure Description

[0029] Figure 1 A flowchart of a phase 1 (preparation phase) for generating synthetic data is shown, in which two types of sample examples are generated and selected: parameter (phrase) sample examples and constraint prompt instruction sample examples.

[0030] Figure 2 A flowchart of a phase 2 (generation phase) is shown, in which each generation takes two types of exemplary samples selected from phase 1 and static feature descriptions as context input to create a new user query, and the process is executed iteratively to populate the query pool.

[0031] Figure 3 A schematic diagram of an architecture for a parameter extractor as a DSPy module is shown, where the DSPy module is an LLM program that can be optimized by the DSPy compiler when training data is provided.

[0032] Figure 4 The diagram illustrates a phase 3 (training phase) process, in which training data is selected from the query pool and used to train data from... Figure 3The initialization DSPy LLM program.

[0033] Figure 5 A flowchart of a phase 4 (online / offline learning phase) is shown, in which generated queries and real data are used to validate queries and API descriptions.

[0034] Figure 6 This is a schematic diagram of a computer-implemented logical method for a synthetic data framework. Detailed Implementation

[0035] The following detailed description, in conjunction with the accompanying drawings, is for illustrative purposes only and is not intended to limit the scope of this embodiment to only this particular construction and implementation. This description explains the functions of each embodiment, as well as the steps for constructing and operating that embodiment. However, the same or equivalent functions and steps can also be achieved through other embodiments.

[0036] The following describes exemplary embodiments of integrated systems, devices, and methods, which are related to the precise adaptation of synthetic data generation to financial business needs and system optimization. This invention solves the challenge of optimizing large language models in specific financial systems, enabling the generation of structured output without requiring comprehensive model fine-tuning.

[0037] The first aspect of this invention relates to the semi-automation of synthetic data generation and validation. LLM (Labeled Data Modeling) is used to semi-automatically generate large amounts of synthetic data to address the data scarcity problem when real-world data and data labels are insufficient. A multi-stage generation process is developed to ensure diversity in query statements in both language structure and parameter specification. This data is tailored to specific financial themes, system requirements, and language variability, making it ideal for specific modular optimizations.

[0038] The second aspect of this invention involves data augmentation using a declarative self-improving language program (DSP y) within the Python optimization tool. A key step in this optimization is to use a custom synthetic dataset as training data to evaluate the performance of the LLM program and generate few-shot examples as demonstrations. This approach alleviates the "cold start" problem of LLMs and significantly improves module performance without requiring extensive fine-tuning.

[0039] A third aspect of this invention focuses on continuous online and offline validation and optimization. Online learning utilizes real-time user data, while offline learning leverages synthetic or offline data. Synthetic and real query statements help identify omissions or ambiguities in the current feature and parameter descriptions. By analyzing error classification and extracting errors, the optimization process can improve and enhance these descriptions, making them more comprehensive and detailed.

[0040] When the text refers to phrases such as "an implementation plan," "a specific implementation plan," "an example implementation plan," "an implementation method," "a specific implementation method," "an example," or "a specific example," it indicates that the implementation plan, implementation method, or example may include specific features, structures, or characteristics. However, not every implementation plan, implementation method, or example must include that specific feature, structure, or characteristic. Furthermore, the above expressions do not necessarily refer to the same implementation plan, implementation method, or example. Moreover, when describing a specific feature, structure, or characteristic in conjunction with an implementation plan, implementation method, or example, it should be understood that such a feature, structure, or characteristic can be implemented by combining other implementation plans, implementation methods, or examples, whether explicitly described or not.

[0041] The terminology used herein is for the purpose of describing specific examples only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” may also be intended to include the plural forms unless the context clearly specifies otherwise. For example, “a” vent may include multiple vents, etc.

[0042] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Similarly, “or” as used herein is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise stated or clearly inferred from the context, “X adopts A or B.” In other words, “X adopts A or B” is satisfied whether X adopts only A, only B, or both A and B.

[0043] Words such as “then” and “next” are not intended to restrict the order of steps; these words are only used to guide the reader through the description of the method.

[0044] The terms "comprising," "including," "having," and "characterized by" are all open-ended expressions, meaning that the stated features, elements, components, steps, values, operations, and / or parts are present, but do not exclude the addition of one or more other features, values, steps, operations, elements, parts, or combinations thereof. When used to describe and limit the various technical solutions described herein, the above open-ended terms are generally understood as non-restrictive; however, in some cases, they can also be interpreted restrictively, for example, as "composed of" or "essentially composed of." Accordingly, for any embodiment described herein involving components, materials, parts, elements, features, values, operations, and / or process steps, embodiments composed of or substantially composed of the stated contents are also covered. Embodiments "composed of" do not include any additional components, materials, parts, elements, features, values, operations, and / or process steps; embodiments "essentially composed of" exclude any additional components, materials, parts, elements, features, values, operations, and / or process steps that have a substantial impact on the basic and innovative features of the solution, but may include other contents that do not have a substantial impact.

[0045] Unless the execution order is explicitly specified, all methods, steps, processes, and operations described in this document should not be construed as requiring execution in a fixed order as described or illustrated herein. Unless otherwise stated, some steps may be added or substituted.

[0046] The technical features described for specific exemplary embodiments can be applied to other exemplary embodiments in any arrangement and combination. Different aspects or components of the exemplary embodiments disclosed herein can also be combined in the manner described above. The terms "combination," "combination form," and similar expressions used herein refer to all permutations and combinations of the foregoing items. For example, "A, B, C, or combinations thereof" is intended to include at least one of A, B, C, AB, AC, BC, or ABC; if the context requires an order, it also includes forms such as BA, CA, CB, CBA, BCA, ACB, BAC, CAB, etc. By way of example, this expression also explicitly covers combinations containing repeating items, such as BB, AAA, AB, BBC, AAABCCCC, CBBAAA, CABABB, etc. Those skilled in the art should understand that, unless the context otherwise specifies, the number of items in a combination is generally not limited.

[0047] The foregoing has described some of the technical solutions of the present invention, but the present invention can be embodied in many different forms and should not be limited to the embodiments disclosed herein. These embodiments are provided to make the content of the present invention complete and comprehensive, and to fully convey the various concepts of the present invention to those skilled in the art.

[0048] Unless otherwise stated, all numerical values ​​described herein are approximate. Therefore, this meaning should be implied even without explicit statement. The numerical values ​​disclosed herein should not be strictly limited to exact values; unless otherwise stated, each value includes the value itself and a range of values ​​with equivalent function. At least, without limiting the application of the equivalence principle to the claims, the interpretation of each value should be based on the significant figures recorded and follow conventional rounding rules. Typically, the numerical error range can be 20%, 10%, or 5% of a given value or range of values. Another interpretation is that "approximately" indicates that the values ​​are on the same order of magnitude, ranging from two to five times the given value. Although the numerical values ​​in this document are approximate, the measured values ​​obtained in specific embodiments will be recorded as accurately as possible. All numerical values ​​themselves have a certain degree of error, which stems from the standard deviation of the corresponding detection method.

[0049] All numerical intervals described in this article include all subintervals within that interval. For example, the interval "1 to 10" includes all subintervals between the minimum value 1 and the maximum value 10, because the numerical intervals disclosed in this article are continuous intervals, covering any value between the maximum and minimum values. Any upper limit of a numerical value described in this article includes all smaller upper limits; any lower limit of a numerical value includes all larger lower limits.

[0050] Features or functions described for specific exemplary embodiments can be applied, either wholly or in parts, to other exemplary embodiments. Similarly, different aspects and components of the exemplary embodiments disclosed herein can also be combined, either wholly or in parts. Some exemplary embodiments can serve as modules of a larger system in which other processes can be prioritized or their application adjusted. Furthermore, several other steps may need to be performed concurrently with, before, or after the exemplary embodiments described herein. It should be noted that all methods and processes described herein can be performed, at least partially, by any entity in any manner.

[0051] The foregoing has illustrated and described some technical solutions; however, those skilled in the art should understand that various modifications and variations can be made without departing from the spirit and scope of this invention. Those skilled in the art can determine numerous equivalent solutions, including alternatives, variations, additions, deletions, modifications, and substitutions, through conventional experiments. Therefore, this application (including the appended claims) aims to cover all modifications and variations falling within the scope of protection of this application.

[0052] Unless otherwise defined, all technical and scientific terms used herein shall have the same meaning as commonly understood by one of ordinary skill in the art.

[0053] The document lists numerous specific details to provide a full understanding of the technical solutions involved in one or more embodiments. However, it should be understood that the embodiments can still be implemented normally even without some of these specific details.

[0054] This framework addresses a fundamental challenge encountered when deploying LLMs to specialized domains: the cold start problem. This problem manifests as the language model's inability to output accurate predictions and relevant content due to a lack of domain knowledge or the absence of context-specific prompts. The financial industry, with its numerous technical terms, high accuracy requirements, and complex contexts, makes the cold start problem particularly pronounced. Traditional methods to mitigate this issue include manually designing prompts, which requires developers to spend significant time tweaking instructions, or performing comprehensive fine-tuning of the underlying model. Both methods suffer from significant drawbacks in terms of labor intensity, scalability, and the required speed of adaptation in dynamic financial environments.

[0055] This framework is built upon Large Language Models (LLMs), which refer to techniques that utilize LLMs to process inputs and generate structured outputs. When this procedure is first introduced into a new domain, it often produces suboptimal results because it is unfamiliar with the specialized vocabulary, regulatory environment, and operational details specific to financial services. By generating synthetic training data that provides hints and exemplary examples for optimizing LLM procedures, the proposed framework overcomes this limitation, thereby eliminating the reliance on scarce, real-world labeled datasets traditionally used for model improvement.

[0056] Financial query processing systems exemplify the complexity that drives this technology. Such systems must perform multiple tasks sequentially: identifying the correct financial features to locate the corresponding application programming interface (API), extracting parameters associated with the user's query, retrieving calculation results from the relevant API, and generating a summary response. Even minor inefficiencies in any single module can exacerbate and degrade overall system performance, highlighting the need for the modular optimization capabilities offered by this invention.

[0057] The open-source DSPy (Declarative Self-Improving Language Programming in Python) framework is a next-generation AI development tool that enables engineers to quickly and easily develop LLM programs. This framework proposes a novel solution utilizing the DSPy programming model, which uses a compiler to iteratively search for optimal example examples and instructions. However, the effectiveness of using DSPy alone is often limited by the availability of labeled training datasets, which are scarce and difficult to obtain. To overcome this limitation, this framework integrates synthetic data generation with financial system optimization and validation. This approach allows for the creation of high-quality labeled datasets needed to optimize LLM subroutines, thereby enhancing the intelligence and performance of each module within the financial system. This framework effectively solves the cold start problem, achieving a significant breakthrough in the automated design and optimization of large language model programs and improving overall system performance.

[0058] This framework proposes a comprehensive approach that precisely aligns synthetic data generation with financial business requirements and system optimization.

[0059] In one example, this framework includes semi-automated synthetic data generation and validation. This framework can be configured to semi-automatically generate large amounts of synthetic data using LLM to overcome data scarcity issues when real-world data and data labels are insufficient. To ensure diversity in query statements in both language structure and parameter specification, a multi-stage generation process was developed. This data is tailored to specific financial themes, system requirements, and language variability, making it ideal for specialized modular optimization.

[0060] In one example, this framework includes data augmentation using the DSPy optimization tool. A key step in this optimization is to evaluate the performance of the LLM program using a custom synthetic dataset as training data and to generate few-shot examples as demonstration cases. This approach alleviates the cold-start problem of LLM and significantly improves module performance without requiring extensive fine-tuning. Those skilled in the art can apply the design approach of this framework to other areas where LLM integration may be necessary.

[0061] In one example, this framework includes continuous online / offline validation and optimization. Online learning uses real-time user data, while offline learning utilizes synthetic or offline data. In large financial systems or similar environments, countless APIs and API descriptions need to be maintained; this invention aims to automate this process using synthetic and real data. In this system, synthetic and real query statements help identify omissions or ambiguities in the current feature and parameter descriptions. By analyzing error classification and extracting errors, the optimization process can improve and enhance these descriptions, making them more comprehensive and detailed. As the system scales and encounters more diverse user queries, the enhanced feature and parameter descriptions ensure its robustness and ability to effectively handle diverse and complex queries.

[0062] The first phase of the framework, called the preparation phase, involves generating two types of sample examples to guide subsequent synthetic query creation. The first type is a parameter phrase demonstration, which determines which parameter values ​​the new synthetic query will include and how to specify these parameters in natural language. Financial features typically accept a variety of parameters, such as currency pairs or reference dates, and the system must generate diverse representations of these inputs.

[0063] To ensure the complexity and variability of the parameter distribution, the framework employs a systematic approach. The process begins by extracting parameter descriptions of the target financial features, containing detailed explanations of all parameters and their value ranges. LLM is then used to generate multiple parameter sets, each including the parameter values ​​for all parameters mentioned in the description, while ensuring that the parameter values ​​are distinct between sets. Since most financial queries typically specify only one or two parameters rather than the entire set, a subset is selected based on a probability distribution and stored in a parameter pool. For each instance in this parameter pool, LLM generates multiple natural language phrases representing the same parameter values ​​but with different linguistic styles, such as expressing a currency pair as "Euro and Dollar" or alternatively as "Eurocoin and Dollar Note."

[0064] The second type of example is the constraint directive, which controls the sentence structure and tone of the generated query. These constraints can be manually defined as customized examples to introduce linguistic diversity. Examples include directives such as "Simplify your query," "Mimic the query style of a seasoned trader," or "Use declarative sentences." By combining parameter phrases with constraint directives, the system can generate synthetic queries that exhibit both parameter diversity and stylistic diversity, more accurately reflecting the range of content that a real user might input in a production system.

[0065] The second phase of this architecture is the generation phase, which uses prepared sample queries to iteratively create new synthetic user queries. This process begins by extracting feature descriptions of the target financial features, providing contextual information about the financial topic. The LLM then randomly selects combinations of parameter phrases and constraint instructions from those generated in the preparation phase. These selected sample queries, along with the static feature descriptions, are provided to the LLM, which generates new user queries that reflect the specified parameters in a specified language style.

[0066] The key innovation in this phase is the deduplication mechanism, which ensures diversity in the resulting query pool. Each newly generated query is compared to all existing queries in the pool using multiple similarity metrics. In one example, similarity metrics might include ROUGE score and embedding similarity. ROUGE score measures lexical overlap between texts, while embedding similarity captures semantic relevance by comparing the vector representations of the queries. Only queries sufficiently different from existing entries are added to the pool. This process is iteratively repeated to expand the query pool while maintaining diversity, resulting in a dataset of synthetic queries that is both broad and representative of real-world financial scenarios.

[0067] In one example, this framework is configured with a volatility validation function to calculate and retrieve implied volatility data for a specified currency pair. Given the parameter phrase "EUR and USD" and the constraint "use declarative statements," LLM can generate the query "show volatility of Euro vs. US Dollar." This synthesized query captures the expected financial business function, includes the specified parameter values, and conforms to the required language structure.

[0068] This framework includes the aDSPy module, which is a basic LLM application designed for processing structured input and output, where each processing operation involves making LLM function calls to retrieve results. A key advantage of DSPy is its ability to optimize prompts through a compiler that iteratively searches for optimal example examples and instructions, eliminating the need for developers to manually write and tune prompts through trial and error.

[0069] This invention utilizes the DSPy module to implement various components of a financial system, with a parameter extraction service serving as a primary example. The parameter extractor is responsible for parsing user queries to identify and extract financial parameters for transmission to the corresponding API. For example, given the query "Get volatility analysis results for the EUR / USD currency pair," and the parameter description indicating that "pair" refers to the currency pair being analyzed, the extractor will output "Currency pair = EUR / USD".

[0070] The initialization of the DSPy module includes defining and specifying inputs and outputs, as well as selecting an inference framework. The framework described in this invention can employ the ReAct method, provided by the standard DSPy framework and involved in logical inference or problem-solving, while also performing actions such as user interaction or environmental manipulation. DSPy uses this framework in conjunction with a structured cue initialization module to guide model behavior. Although these initialized cue words can extract functional parameters in the initial stage, further optimization by the DSPy compiler is still required to achieve commercial-grade accuracy.

[0071] This framework includes a training phase, which uses pre-generated synthetic training data to optimize the initialization module. The compiler needs to annotate the training data to create inference trajectories that serve as example examples for the program. A large query pool containing synthetic user queries, each labeled with real parameters, provides the necessary training dataset.

[0072] The DSPy compiler performs self-optimization through a structured process. First, the compiler is guided by few-shot examples generated from the training set, which serve as candidate inference examples to be included in the prompt words of the module. Second, based on the initialized DSPy program, the compiler analyzes the task and provides multiple candidate instructions that describe the same program task but differ in structure and style. Third, the compiler iteratively combines the candidate instructions with the candidate examples, evaluating all combinations against the test dataset to identify the optimal configuration. The prompt word configuration that achieves the best performance is stored in the optimization module, which can then be saved to a local file and integrated into commercial financial systems.

[0073] This optimization method offers significant advantages over traditional approaches. This invention eliminates the need for fine-tuning the underlying LLM weights, an expensive and data-intensive operation. The improved DSPy compiler optimizes only the prompts and examples provided to the model. This technique significantly improves module performance while avoiding the resource requirements associated with training a full-scale model. The framework is also highly scalable, as the optimization process can be applied to all DSPy modules within the system, allowing each component to be individually tuned for its specific task.

[0074] In one example, this framework includes phases of continuous validation and optimization, implemented through a combination of online and offline learning methods. Online learning may involve incremental processing of data arriving in real time, enabling the system to immediately adapt to new patterns in user behavior. Offline learning may involve processing large volumes of pre-collected data, which may originate from synthetic datasets or historical user queries. This dual approach ensures that the framework can obtain immediate feedback while simultaneously enabling systematic analysis of accumulated data.

[0075] During the continuous validation and optimization phase, queries are selected from synthetic datasets or from real-time user interactions. Each query is first processed by a feature recognition module, which categorizes the query into a financial category corresponding to a specific API feature. The accuracy of this classification affects subsequent processing. If the classification is incorrect, the validation module analyzes the error to determine its root cause. Errors may stem from inappropriate feature descriptions that fail to properly guide the model, or from vaguely worded queries that confuse the user's financial intent.

[0076] Once the feature recognition module correctly classifies the query, the query and its associated feature labels are forwarded to the parameter extraction module. This module extracts the necessary parameters as input to the target feature API. The validation process then checks the accuracy of the extracted parameters. Incorrect parameter extraction may stem from incomplete parameter descriptions, failure to provide effective guidance for the model, or unclear parameter descriptions in the original query.

[0077] Improved automation description and system adaptation

[0078] Validation results generated through online and offline learning are used as feedback for improvements to the automated system. When errors are identified, an LLM-based optimization procedure analyzes the nature of the errors and proposes improvements to the descriptions of relevant features or parameters. This improvement process makes the descriptions more comprehensive and detailed, thereby reducing the likelihood of similar errors in subsequent processing.

[0079] As the system expands and serves a wider user base, it receives increasingly diverse queries, potentially exposing omissions or ambiguities in existing descriptions. This framework addresses this challenge by continuously incorporating new data and feedback into the optimization process. Regular updates ensure the system remains up-to-date with evolving user demands and new types of queries that emerge over time. This continuous validation and optimization approach maintains system accuracy and relevance without requiring manual intervention from developers for each incremental improvement.

[0080] In large financial systems, numerous APIs and their associated descriptions require continuous maintenance, making the automation of maintenance processes highly valuable. Given the scale and complexity involved, manually maintaining these descriptions is impractical. By delegating this task to an LLM-driven optimization process guided by real-world operational data, this framework enables sustainable system iteration, adapting to the dynamic changes in financial services.

[0081] This framework relies on essential infrastructure for managing domain-specific knowledge. Feature and parameter descriptions are maintained by a feature management system, which serves as a knowledge base accessible to the LLM for contextual understanding. Each feature description outlines the functionality and usage guidelines of a specific API, enabling the model to understand the financial context of each subtask. For example, the description of the volatility check feature explains how the feature calculates and retrieves implied volatility data for a given currency pair, while displaying volatility levels across different maturities and strike prices for reference analysis.

[0082] Parameter descriptions complement feature descriptions by specifying the inputs required for each API. These descriptions include a comprehensive explanation of the parameter name, data type, and purpose of each parameter. For the volatility check feature, parameters might include "pairs" to specify a currency pair in six-digit format, "value_time" to indicate the reference date, and "spot" as a Boolean indicator of whether spot price data should be included. This structured approach ensures that the LLM provides clear guidance on how to configure each API based on user queries.

[0083] The relationship between feature descriptions, parameter descriptions, and LLM procedures is crucial for system operation. Descriptions, used for domain-specific knowledge, enable the model to correctly interpret queries and map them to appropriate API calls. The quality and understanding of these descriptions directly impact system accuracy, which is why online / offline learning processes have mechanisms for automatically optimizing descriptions based on observed performance.

[0084] Modular architecture and scalability

[0085] This invention describes a modular architecture in which each system component is built as a separate DSPy module capable of independent optimization. A feature recognition module categorizes input queries into corresponding financial categories. A parameter extraction module parses the query to extract relevant parameters. Additional modules can handle other aspects of the query, such as API result retrieval and response summarization.

[0086] This modular design offers significant scalability advantages. Each module can be optimized using task-specific training data, allowing improvements to the target without affecting other components. As the system evolves, new modules can be added to handle additional financial features or processing steps. By matching corresponding feature and parameter descriptions, the synthetic data generation process can be appropriately applied to any module. Therefore, the framework is scalable to accommodate the ever-increasing complexity of financial query processing systems.

[0087] Optimization modules can be saved to local files and integrated into production systems, further enhancing operational flexibility. Modules can be versioned, tested independently, and deployed incrementally. Performance improvements achieved through DSPy compiler optimizations can be released systematically, and rollback to previous versions is possible if issues arise. This deployment model aligns with modern software engineering practices while adapting to the specific requirements of LLM-based applications.

[0088] The processes supported by this framework can be categorized into multiple stages, which will be discussed in the appendix. Figure 1-5 Detailed description. References Figure 1Phase 1 includes a preparation phase for generating synthetic user queries. In this phase, two sample queries can be generated and selected: a parameter (phrase) sample and a constraint directive sample. Parameters and parameter phrases determine what parameter values ​​the new query includes and how these parameters are specified in natural language.

[0089] To ensure the complexity and variability of parameter distribution in the synthetic data, the following methods can be used to generate parameters: (1) For the target financial feature, extract its parameter description, which contains a detailed explanation of all parameters and their value ranges; (2) Use LLM to generate multiple parameter sets, where each parameter set includes selected values ​​of all parameters mentioned in the parameter description, ensuring that the parameter values ​​are different between parameter sets; (3) For each parameter set, select a subset based on the probability distribution, since most financial query statements typically specify only one or two parameters, and store these simplified subsets in a parameter pool for later use; (4) For each instance in the parameter pool, use LLM to generate multiple natural language phrases for each included parameter, where these phrases specify the same parameters but have different language styles.

[0090] Additionally, constraint directives are used to control the sentence structure and tone of the generated query statements. Constraint directives can be manually defined, customized examples, such as "simplify your query statement," "mimic the query style of experienced traders," or "use declarative sentences."

[0091] refer to Figure 2 Phase 2 includes the generation phase. Using feature descriptions as context, the LLM randomly selects a combination of two sample queries from Phase 1 to create a new user query statement.

[0092] The method includes: (1) extracting feature descriptions for the same target financial features as in the previous stage, which contain detailed explanations of the corresponding financial topics, as contextual input for generating synthetic query statements; (2) randomly selecting a set of parameter phrases and constraint instructions from stage 1 as information input; (3) feeding the financial feature descriptions, parameter phrases, and constraint instructions back to the LLM as input to generate new query statements; (4) comparing the new query statements with all existing query statements in the query pool, and adding the new query statement to the query pool if it differs from any query in the query pool; wherein multiple indicators are used to complete the above comparison, including ROUGE score and embedding similarity; (5) repeating steps 2-4 to expand the query pool. This method ensures that the generated synthetic data is both diverse and reflects actual financial scenarios.

[0093] refer to Figure 3The architecture of the parameter extractor is illustrated as a DSPy module. The DSPy module is designed for basic LLM applications that handle structured input and output. Each time input is processed programmatically, it makes an LLM function call to retrieve the results. Additionally, the module leverages the ReAct framework to enhance its capabilities. In the ReAct framework, LLM is designed to perform reasoning (such as logical deduction or problem solving) and actions (such as performing operations in an environment or interacting with a user) in a collaborative manner. This allows DSPy to initialize its modules using generated structured prompts.

[0094] The first step in initializing the parameter extraction program is to define the inputs and outputs. Next, an inference framework is chosen, such as ReAct provided by the DSPy framework. After initialization, the parameter extractor can extract the parameters. However, although DSPy initializes the cue words internally within the model, they still need to be optimized by the DSPy compiler to improve accuracy.

[0095] refer to Figure 4 Phase 3 includes the training phase. Within the DSPy programming framework, each LLM program (e.g., a parameter extractor) is built as a DSPy module, which can be optimized by the DSPy compiler to improve accuracy. The compiler typically requires a labeled training dataset to create inference trajectories as example examples of the corresponding programs.

[0096] The DSPy compiler performs self-optimization of the DSPy module through the following steps: (1) It generates a small number of examples from the training set and completes the bootstrapping construction, where these examples are used as candidate inference examples for module prompt words; (2) Based on the initialized DSPy program, the DSPy compiler understands the task and provides candidate instructions, where these instructions describe the same program task but differ in structure and style; (3) The compiler iteratively combines candidate instructions and examples, evaluates all combinations on the test dataset to identify the best combination, where the best prompt is then stored in the optimization module.

[0097] The optimized parameter extractor can be saved to a local file and integrated into the financial system. Therefore, this framework is highly scalable and can be extended to other financial features and LLM programs within the system.

[0098] refer to Figure 5 The online / offline learning phase focuses on continuous system validation and optimization, particularly in the context of large financial systems or similarly complex environments. The goal is to automate this maintenance process by combining query data and real-world query data.

[0099] Online learning typically involves incremental processing as data arrives in real time, while offline learning usually processes large volumes of pre-collected data, which can be obtained from synthetic datasets or user historical data. During the online / offline learning phase, queries are selected from synthetic datasets or real-time data and fed into a feature recognition module, which categorizes user queries into financial categories corresponding to specific features.

[0100] There are two possible outcomes: (1) If the module output is incorrect, the verification module determines the error type, which may be caused by an inappropriate feature description that needs improvement; or by an unclear financial intent from the query statement that requires improvement of the query statement or query statement generation; (2) If the module output is correct, the user query statement along with the feature labels is sent to the parameter extraction module, which extracts the corresponding parameters as input data for the features.

[0101] The validation module checks the accuracy of the extracted parameters. If incorrect, the error may stem from: (a) inappropriate feature parameter descriptions that require improvement; or (b) inappropriate parameter descriptions that require improvement of query parameters or parameter generation.

[0102] As the system expands and serves a wider user base, it will encounter more diverse query types. The LLM optimizer can use this feedback to automatically improve and enhance API features and parameter descriptions. This process is ongoing, with new data and feedback regularly incorporated into the system to help it adapt to changing requirements and new types of queries. Continuous validation and optimization ensure the system remains up-to-date and efficiently meets the needs of its users.

[0103] refer to Figure 6 This framework can be implemented as a data processing logic designed to generate synthetic data, optimize the performance of Large Language Models (LLMs), and enable continuous validation in specialized domains, particularly financial systems. This process addresses the "cold start" problem, where LMMs struggle to output accurate results due to a lack of domain-specific knowledge or appropriate cue words.

[0104] The preparation processing unit, acting as the input processing unit, receives the raw signals and generates preprocessed output signals. This unit comprises two main sub-processes that operate collaboratively to generate the fundamental components used for synthesizing query statements.

[0105] The first subprocess can be configured as a parametric phrase generator. This subprocess extracts parameter descriptions of the target features, generates multiple parameter sets with different values, selects a subset based on a probability distribution, and generates multiple natural language phrases with different linguistic styles for each parameter set. The output signal of this subprocess is a parametric phrase example, such as different expressions like "euro and dollar" or "Eurocoin and US dollar bill" referring to the same set of base currency parameters.

[0106] The second subprocess can be a constraint instruction receiver. This subprocess receives manually defined constraint instructions that control the sentence structure and tone of the generated query. The output signal of this subprocess is a sample constraint instruction, which may include instructions such as "Simplify your query," "Mimic the query style of experienced traders," or "Use declarative sentences."

[0107] The first data path connects the preparation processing unit and the generation process unit. This path transmits the parameter phrase examples and constraint instruction examples output by the preparation processing unit as input for the next processing stage.

[0108] The generation process unit receives input signals through the first data path and generates synthesized user query statements. This unit also includes two main sub-processes that collaborate to produce diverse, high-quality synthesized queries.

[0109] The first sub-process of the generation process unit can be a query synthesizer. This sub-process extracts feature descriptions for the context input, randomly selects parameter phrases and constraint instructions from stage 1, and feeds these inputs into the LLM to generate a new user query. The output signal is a synthesized user query statement, which reflects the specified parameters according to a prescribed language style.

[0110] The second subprocess can be a similarity comparator. This subprocess can be configured to compare each new query with existing queries in the query pool using ROUGE scores, which measure lexical overlap, and embedding similarity, which captures semantic relevance. Only new queries that significantly differ from each existing query in the query pool are added to the query library. The output is a diverse and deduplicated query library that comprehensively covers various potential user inputs.

[0111] The second data path connects the generation process unit and the training process unit. This path transmits the synthesized user query statements output by the generation process unit as training data for the optimization process.

[0112] The training unit receives the synthesized query statement through the second data path and optimizes the self-optimizing language program module. The main sub-process within this unit is the suggestion optimizer, which in one example can be implemented using the DSPy compiler.

[0113] The optimizer subprocess bootstraps a few samples from the training set as candidate inference examples. Then, it proposes candidate instructions describing the program task in various structures and styles, iteratively combines these instructions with the examples, and evaluates all combinations on a trial dataset to identify the optimal configuration. The output is an optimized module stored in a local file that can be integrated into a production system.

[0114] The feedback path forms a closed loop by sending the validation signals from the training units back to the preparation processing unit. This path includes several sub-processes that enable continuous system optimization.

[0115] The feature recognition subprocess categorizes user queries into business domains corresponding to various API features. Its output includes classification signals that route queries to the appropriate downstream processing.

[0116] When an error is incorrectly categorized, the error determination subprocess is activated. It determines whether the error is caused by an inappropriate feature description or an ambiguous query statement. Its output includes an error type signal to inform process improvements.

[0117] The parameter extraction subprocess handles correctly categorized query statements, extracting parameters from the query statements to be used as API input and generating extracted parameter signals for use by downstream processes.

[0118] The verification subprocess checks the accuracy of the extracted parameters and identifies errors caused by incomplete or unclear parameter descriptions. It generates verification feedback signals through the feedback path.

[0119] The above-mentioned sub-processes work together to enable the optimization program based on the large language model to automatically improve and enhance the feature and parameter descriptions based on the verification results, forming a continuous iterative optimization closed loop and gradually improving the system's operating performance.

[0120] This embodiment provides a system for generating synthetic data and optimizing the performance of a large language model in a target domain, comprising: a preparation module configured to generate multiple parameter sets for target domain features, wherein each parameter set includes selected values ​​of parameters related to the target domain features; for each parameter set, generating multiple natural language phrases representing the parameters; and receiving constraint instructions; a generation module configured to combine feature descriptions, multiple natural language phrases, and constraint instructions and input them into the large language model to generate synthetic user query statements; and a training module configured to use the synthetic user query statements as training data to optimize a self-optimizing language program module, wherein the optimization operation includes extracting bootstrap samples from the training data and iteratively evaluating instruction-demonstration sample combinations to determine the optimal prompt configuration.

[0121] This embodiment provides a system for generating synthetic data and optimizing the performance of a large language model in a target domain. The preparation module is further configured to extract parameter descriptions of target domain features, wherein the parameter descriptions include explanations of the parameters and their value ranges.

[0122] This embodiment provides a system for generating synthetic data and optimizing the performance of large language models in a target domain. The generation module utilizes similarity metrics, including at least one of ROUGE score and embedding similarity. The ROUGE score measures the lexical overlap between texts, and the embedding similarity captures semantic relevance by comparing the vector representations of query statements.

[0123] This embodiment provides a system for generating synthetic data and optimizing the performance of large language models in a target domain. The training module optimizes a self-optimizing language program module, which includes a parameter extraction module configured to parse user query statements to identify and extract parameters to be transmitted to the application programming interface.

[0124] This embodiment provides a system for generating synthetic data and optimizing the performance of large language models in a target domain. The training module optimizes the self-optimizing language program module, which utilizes the ReAct framework to enable the large language model to perform reasoning and execution actions in a collaborative manner.

[0125] This embodiment provides a system for generating synthetic data and optimizing the performance of large language models in a target domain. The training module is further configured to perform continuous validation by combining online learning and offline learning. The online learning incrementally processes data arriving in real time, while the offline learning processes batches of pre-collected data.

[0126] This embodiment provides a system for generating synthetic data and optimizing the performance of a large language model in a target domain. Continuous verification includes: using a feature recognition module to classify user query statements into corresponding domain categories; when the classification is incorrect, the verification module determines the error type; and using a parameter extraction module to extract parameters from correctly classified query statements and improve feature descriptions and parameter descriptions based on the verification results.

[0127] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The method includes: in a preparation phase, generating multiple parameter sets for target domain features, wherein each parameter set includes selected values ​​of parameters related to the target domain features; for each parameter set, generating multiple natural language phrases representing the parameters, and receiving constraint instructions; in a generation phase, generating synthetic user query statements by one or more processors by inputting feature descriptions, multiple natural language phrases, and constraint instructions into a large language model; and in a training phase, optimizing a self-optimizing language program module by one or more processors using the synthetic user query statements as training data, wherein optimization includes extracting bootstrap samples from the training data and iteratively evaluating instruction-demonstration sample combinations to determine the optimal prompt configuration.

[0128] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The preparation stage further includes extracting parameter descriptions of target domain features from a feature management system. The parameter descriptions include explanations of the parameters, data types, and value ranges.

[0129] This embodiment provides a method for generating synthetic data and optimizing the performance of large language models in a target domain, wherein the preparation phase generates multiple parameter sets based on a subset of parameters selected from each parameter set according to a probability distribution.

[0130] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain, wherein the preparation phase receives constraint instructions, which include at least one of the following: instructions to simplify query statements, instructions to mimic the query style of a specific user type, and instructions to use declarative sentences.

[0131] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The training phase further includes: providing candidate instructions by a prompting optimizer, the candidate instructions describing program tasks and differing in structure and style; iteratively combining the candidate instructions with candidate example samples; and evaluating all combinations on a test dataset to determine the optimal configuration.

[0132] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The training phase further includes saving the optimized self-optimizing language program module to a local file and integrating the optimized self-optimizing language program module into the production system.

[0133] This embodiment provides a method for generating synthetic data and optimizing the performance of a large language model in a target domain. The training phase further includes: using a feature recognition module to classify the input user query into the corresponding domain category; using a parameter extraction module to extract parameters from the input user query; verifying the accuracy of the extracted parameters; and automatically improving the feature description and parameter description based on the verification results using an optimization program based on a large language model.

[0134] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous verification and optimization for a large language model program. The system includes a preparation unit configured to generate parametric phrase examples and constraint instruction examples for target domain features; and to extract feature descriptions and parametric descriptions from a feature management system; a generation process unit configured to receive input signals from a preparation processing unit via a first data path, iteratively generate synthetic user query statements by combining feature descriptions with parametric phrase examples and constraint instruction examples, and add different synthetic user query statements to a query pool based on similarity comparison; a training unit configured to receive input signals from the generation unit via a second data path; an optimization self-optimizing language program module using synthetic user query statements as training data, wherein the optimization compiler is prompted to extract bootstrap sample few-shot examples and iteratively evaluate instruction-example combinations; and to perform continuous verification using online and offline learning to identify errors and automatically improve feature descriptions and parametric descriptions; and a feedback path configured to feed verification signals back from the training unit to the preparation processing unit to achieve iterative improvement of feature descriptions and parametric descriptions.

[0135] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous validation and optimization for large language model programs. The system uses a customized synthetic dataset as training data to evaluate the performance of the large language model and generates few-shot examples as demonstration examples. The generation unit and the training unit work together to solve the cold start problem of the large language model.

[0136] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous validation and optimization for large language model programs. The training unit is used to optimize the self-optimizing language program module, which includes a modular architecture. Each system component is constructed as a separate sub-process that can be optimized independently, including a feature recognition sub-process and a parameter extraction sub-process.

[0137] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous validation and optimization for large language model programs. The training process unit further includes: a feature recognition subprocess configured to process user query statements and classify them into corresponding domain categories; an error analysis subprocess configured to analyze errors when the classification is incorrect to determine whether the errors are caused by inappropriate feature descriptions or ambiguity in the query statements; and a parameter extraction subprocess configured to extract parameters for input to the application programming interface when the classification is correct.

[0138] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous verification and optimization for large language model programs. The target domain features include financial features, and the workflow is configured to process financial query statements, identify the correct financial features to locate the corresponding application programming interface (API), extract parameters associated with the user query statement, obtain calculation results from the corresponding API, and generate summary results.

[0139] This embodiment provides a computer-implemented system for generating synthetic data and providing continuous verification and optimization for large language model programs. The preparation processing unit uses a feature management system to maintain feature descriptions and parameter descriptions. The feature management system serves as a knowledge base. Each feature description describes the function and usage instructions of the corresponding application programming interface, and the parameter description specifies the input required for each application programming interface, including parameter name, data type, and purpose explanation of each parameter.

[0140] The detailed description above, taken in conjunction with the accompanying drawings, is only for illustrating exemplary embodiments and is not intended to limit the only ways in which this embodiment can be constructed and used.

[0141] It should be understood that the architecture and solutions described in this article are exemplary, and the implementation methods, forms and examples of the invention should not be construed as limiting. This solution may have various modifications.

[0142] The specific processes or methods described in this document can employ various processing strategies. Therefore, the operations illustrated and described in text can be executed in the order they are recorded, in a different order, or in parallel, or may be omitted as appropriate. The execution order of the above processes can also be changed.

[0143] While this technical content is described in conjunction with structural features and method steps, the scope of protection defined by the appended claims is not limited to the specific features and steps described above. The specific features and steps described above are merely exemplary embodiments for implementing the claims.

Claims

1. A system for generating synthetic data and optimizing the performance of large language models in a target domain, the system comprising: The preparation module is configured to generate multiple parameter sets for target domain features, wherein each parameter set includes selected values ​​of parameters related to the target domain features; for each parameter set, generate multiple natural language phrases representing the parameters; and receive constraint instructions. The generation module is configured to combine the feature description, the multiple natural language phrases and the constraint instructions and input them into the large language model to generate a synthetic user query statement. as well as The training module is configured to use the synthesized user query as training data to optimize the self-optimizing language program module, wherein optimization includes extracting bootstrap samples from the training data and iteratively evaluating instruction-demonstration sample combinations to determine the optimal prompt configuration.

2. The system according to claim 1, wherein, The preparation module is further configured to extract parameter descriptions for the target domain features, wherein the parameter descriptions include explanations of the parameters and their value ranges.

3. The system according to claim 1, wherein, The generation module employs a similarity metric, which includes at least one of ROUGE score and embedding similarity. The ROUGE score measures the lexical overlap between texts, while the embedding similarity captures semantic relevance by comparing the vector representations of the query statements.

4. The system according to claim 1, wherein, The training module optimizes the self-optimizing language program module, which includes a parameter extraction module configured to parse user query statements to identify and extract parameters to be transmitted to the application programming interface.

5. The system according to claim 1, wherein, The training module uses the ReAct framework to optimize the self-optimizing language program module. The ReAct framework enables the large language model to complete reasoning and execution actions in a collaborative manner.

6. The system according to claim 1, wherein, The training module is also configured to perform continuous validation using online learning and offline learning, wherein online learning incrementally processes data arriving in real time and offline learning processes data collected in batches in advance.

7. The system according to claim 6, wherein, The continuous verification includes: The feature recognition module is used to classify user queries into corresponding domain categories; When the classification is incorrect, the verification module determines the error type; Use the parameter extraction module to extract parameters from correctly categorized user queries; and Improve feature and parameter descriptions based on validation results.

8. A method for generating synthetic data and optimizing the performance of a large language model in a target domain, the method comprising: During the preparation phase, multiple parameter sets are generated for the target domain features, wherein each parameter set includes selected values ​​of parameters related to the target domain features; For each parameter set, generate multiple natural language phrases representing the parameters, and receive constraint instructions; During the generation phase, one or more processors combine the feature descriptions, the multiple natural language phrases, and the constraint instructions and input them into the large language model to generate a synthetic user query statement. as well as During the training phase, the one or more processors use the synthesized user query as training data to optimize the self-optimizing language program module, wherein optimization includes extracting bootstrap samples from the training data and iteratively evaluating instruction-demonstration sample combinations to determine the optimal prompt configuration.

9. The method according to claim 8, wherein, The preparation phase also includes extracting parameter descriptions of the target domain features from the feature management system, wherein the parameter descriptions include explanations of the parameters, data types, and value ranges.

10. The method according to claim 8, wherein, The preparation phase generates the multiple parameter sets by selecting a subset of parameters from each parameter set based on a probability distribution.

11. The method as described in claim 8, characterized in that, The preparation phase receives constraint instructions, which include at least one of the following: instructions to simplify query statements, instructions to mimic the query style of a specific user type, and instructions to use declarative sentences.

12. The method according to claim 8, wherein, The training phase also includes: The compiler provides candidate instructions that describe the program task and differ in structure and style. Iteratively combine the candidate instructions and candidate example samples; and All combinations are evaluated based on the test dataset to determine the optimal configuration.

13. The method according to claim 8, wherein, The training phase also includes: Save the optimized self-optimizing language program module to a local file; and The optimized self-optimizing language program module was integrated into the production system.

14. The method according to claim 8, wherein, The training phase also includes: The feature recognition module is used to classify the input user query into the corresponding domain category; The parameter extraction module is used to extract parameters from the input user query statement; Verify the accuracy of the extracted parameters; and Based on the validation results, an optimization program based on a large language model is used to automatically improve the feature descriptions and parameter descriptions.

15. A computer-implemented system for generating synthetic data and providing continuous validation and optimization for large language model programs, the system comprising: The preparation unit is configured to generate example parametric phrases and example constraint instructions for the target domain features; And extract feature descriptions and parameter descriptions from the feature management system; The generation unit is configured to receive an input signal from the preparation unit via a first data path, iteratively generate synthetic user query statements by combining the feature description with the parameter phrase example and the constraint instruction example, and add different synthetic user query statements to the query pool based on similarity comparison. A training unit is configured to receive input signals from the generation unit via a second data path; and to optimize a self-optimizing language program module using the synthesized user query as training data, wherein the optimization compiler bootstraps few-sample examples and iteratively evaluates instruction-demonstration example combinations. And use online and offline learning to perform continuous validation to identify errors and automatically improve feature and parameter descriptions; as well as The feedback path is configured to send a verification signal back from the training unit to the preparation unit to enable iterative improvement of feature description and parameter description.

16. The computer-implemented system according to claim 15, wherein, The performance of the large language model is evaluated by using a customized synthetic dataset as training data, and few-shot examples are generated as demonstration examples. The generation unit and the training unit work together to solve the cold start problem of the large language model.

17. The computer-implemented system of claim 15, wherein the training unit optimization comprises a modular architecture of a self-optimizing language program module, wherein each system component is constructed as a separate subprocess capable of independent optimization, including a feature recognition subprocess and a parameter extraction subprocess.

18. The computer-implemented system according to claim 15, wherein, The training unit also includes: The feature recognition subprocess is configured to process user query statements and classify the user query statements into corresponding domain categories; The error analysis subprocess is configured to analyze the error when the classification is incorrect to determine whether the error is caused by an inappropriate feature description or an ambiguous query statement; and The parameter extraction sub-process is configured to extract parameters for input to the application programming interface when the classification is correct.

19. The computer-implemented system according to claim 15, wherein, The target domain features include financial features, and the workflow is configured to process financial queries that identify the correct financial features to locate the corresponding application programming interface (API), extract parameters related to the user query, obtain calculation results from the corresponding API, and generate a summary response.

20. The computer-implemented system according to claim 15, wherein, The preparation unit uses a feature management system to maintain feature descriptions and parameter descriptions. The feature management system serves as a knowledge base. Each feature description describes the function and usage instructions of the corresponding application programming interface. The parameter description specifies the inputs required for each application programming interface, including parameter name, data type, and an explanation of the purpose of each parameter.