Large-scale model learning optimization methods, devices, equipment, media, and program products

By using real-time automated error detection and classification, and dynamically adjusting the sample weights in the large model's sample library, the problem of low efficiency and overfitting of manual annotation in vertical domains for large language models is solved, achieving high accuracy and stable user interaction answers.

CN123070243APending Publication Date: 2026-11-10CHINA UNIONPAY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611443542.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-16
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In vertical domain applications, existing large language models suffer from low efficiency in manually annotating question-and-answer pairs, resulting in poor real-time prompts, decreased accuracy of user-interacted answers, complex and computationally expensive parameter tuning, and a tendency to overfit, making it difficult to support high-concurrency and 24/7 online business scenarios.

Method used

By constructing a sample library through real-time automated error detection and classification, and dynamically adjusting the proportion of reference samples in the prompt words according to the learning weights of different categories of samples in the sample library, the task processing distribution trend of the large model is consistent, overfitting is avoided, and the accuracy of the answer is improved.

Benefits of technology

It improves the accuracy of large models in high-concurrency and 24/7 online business scenarios, ensures the stability and generalization ability of the models, reduces computing power consumption, and avoids overfitting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070243A_ABST
    Figure CN123070243A_ABST
Patent Text Reader

Abstract

This application discloses a learning optimization method, apparatus, device, medium, and program product for a large model, belonging to the field of artificial intelligence. The method includes: obtaining reference samples from a sample library; constructing a first large model based on natural language query information and the reference samples; obtaining the SQL result output by the first large model; the allocation ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library; performing multi-dimensional verification on the SQL result; classifying the samples including the SQL result and natural language query information, and storing them in the corresponding sample set in the sample library according to category; the sample categories include correct categories and incorrect categories, with the incorrect category including at least one incorrect subclass; and determining the learning weights of samples of different categories based on the feature information of samples of each category in the sample library. According to the embodiments of this application, the accuracy of SQL results obtained by users interacting with the large model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to a method, apparatus, device, medium and program product for learning and optimizing large models. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, Large Language Models (LLMs) have made breakthrough progress (hereinafter referred to as LLMs for ease of description), and can be applied to various scenarios, such as intelligent customer service, data query and analysis, and knowledge base retrieval and management. In some vertical applications, such as Natural Language to Structured Query Language (NL2SQL) and data analysis tasks, question-and-answer pairs (QA pairs) samples can be pre-annotated manually. By setting pre-annotated QA pairs samples in the prompts provided to the LLM, the LLM can be guided to output answers. However, manually annotating QA pairs is inefficient, resulting in poor real-time performance of the QA pairs in the prompts, which is inconsistent with the distribution trend of the LLM task processing, leading to a decrease in the accuracy of the answers obtained by the user interacting with the LLM. Summary of the Invention

[0003] This application provides a method, apparatus, device, medium, and program product for learning and optimizing large models, which can improve the accuracy of SQL results obtained by users interacting with large models.

[0004] In a first aspect, embodiments of this application provide a learning optimization method for a large model, comprising: receiving natural language query information input by a user; obtaining reference samples from a sample library; constructing a prompt word input first large model based on the natural language query information and the reference samples; obtaining a first SQL result output by the first large model; wherein the allocation ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library; performing multi-dimensional verification on the first SQL result; classifying the samples including the first SQL result and the natural language query information according to the obtained verification results; storing the samples in the sample library according to the categories; wherein the categories of the samples include correct categories and incorrect categories; and the incorrect categories include at least one incorrect subclass; and determining the learning weights of samples of different categories based on the feature information of samples of each category in the sample library.

[0005] Secondly, embodiments of this application provide a large-scale model learning optimization apparatus, comprising: a receiving module for receiving natural language query information input by a user; an SQL result generation module for obtaining reference samples from a sample library, constructing a prompt word input first large-scale model based on the natural language query information and the reference samples, and obtaining a first SQL result output by the first large-scale model, wherein the allocation ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library; a classification and aggregation module for performing multi-dimensional verification on the first SQL result, classifying the samples including the first SQL result and the natural language query information according to the obtained verification results, and storing them in the corresponding sample set in the sample library according to the category, wherein the sample categories include correct categories and incorrect categories, and the incorrect category includes at least one incorrect subclass; and a learning weight determination module for determining the learning weights of samples of different categories based on the feature information of samples of each category in the sample library.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the learning and optimization method of the large model in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the learning and optimization method for the large model of the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the learning and optimization method for a large model in the first aspect.

[0009] This application provides a learning optimization method, apparatus, device, medium, and program product for a large model. It selects reference samples from a sample library based on the learning weights of samples of different categories. Based on the reference samples and the user's input natural language query information, it constructs prompt words, inputs these prompt words into a first large model, and obtains the first SQL result output by the first large model. The first SQL result and the natural language query information are used as samples, and these samples are promptly stored in the corresponding sample set in the sample library according to the category determined by multi-dimensional verification of the first SQL result. This ensures that the latest samples can be referenced when the first large model executes tasks subsequently. Updates to the samples in the sample library drive updates to the learning weights of the samples. Based on the learning weights of the samples in the sample library, the distribution ratio of different categories of samples in the reference samples required to construct the prompt words is adjusted, ensuring that the samples in the prompt words match the distribution trend of the large model's task processing, thereby improving the accuracy of the SQL results obtained from the user's interaction with the large model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a learning optimization method for a large model provided in one embodiment of this application; Figure 2 A schematic diagram illustrating an example of the learning and optimization process for a large model provided in an embodiment of this application; Figure 3 A schematic diagram illustrating an example of the process for constructing a sample library and corpus provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating an example of a data screening and overfitting detection process for a sample library provided in an embodiment of this application. Figure 5 A schematic diagram of the structure of a learning and optimization device for a large model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the purpose, technical solution, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of the details in these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or necessarily used such a solution.

[0013] With the development of AI technology, large-scale models have made breakthrough progress and can be applied to various scenarios, such as intelligent customer service, data query and analysis, and knowledge base retrieval and management. In some vertical applications, such as NL2SQL and data analysis tasks, question-and-answer pairs (QA pairs) samples can be pre-annotated manually. By setting pre-annotated QA pairs in the prompts provided to the large-scale model, the model can be guided to output answers. However, manual annotation of QA pairs is inefficient and costly, resulting in poor real-time performance of the QA pairs in the prompts. This leads to a mismatch between the QA pairs and the distribution trends of the large-scale model's task processing, causing a decrease in the accuracy of the answers obtained by users interacting with the large-scale model. It also makes it difficult to support high-concurrency business scenarios and 24 / 7 online business scenarios. In traditional approaches that guide large models to output answers using prompt words, the ratio of question-answer pairs in the prompt words is static and fixed. This makes it difficult to adapt to the historical experience distribution of the large model in handling tasks. Blindly increasing the ratio of error-category samples in the prompt words can easily cause data drift, reducing the generalization ability of the large model and leading to a decrease in the accuracy of answers obtained by users interacting with the large model. Large models also struggle to learn specific techniques for high-frequency, error-prone situations, causing them to repeatedly make the same mistakes during task execution, further reducing the accuracy of their output answers. Pre-training the large model using training samples requires adjusting all model parameters, which is computationally complex and consumes a lot of computing power. Furthermore, the adjusted model also carries stability risks; if the performance is poor, retraining is necessary, significantly increasing computational costs. Moreover, large models are prone to overfitting, causing the answers output by the large language model to deviate from user intent and true logic. Overfitting is difficult to detect promptly through manual observation, further reducing the accuracy of answers obtained by users interacting with the large model.

[0014] This application provides a learning optimization method, apparatus, device, medium, and program product for a large-scale model. It can automatically detect errors, classify, and aggregate samples in real time from the Structured Query Language (SQL) results output by the large-scale model based on the user's natural language query information. No manual annotation is required, and the classification and sample aggregation are highly efficient. Newly generated samples can be promptly stored in a sample library for reference during subsequent large-scale model task execution. This ensures that the samples in the prompts align with the distribution trend of the large-scale model's task processing, improving the accuracy of answers obtained from user interaction with the large-scale model. It also supports high-concurrency and 24 / 7 online business scenarios. In this application's embodiments, the learning weights of samples of different error subclasses can be adjusted based on the distribution trend of samples formed from historical data in the sample library. This adjusts the proportion of samples of different error subclasses in the reference samples provided to the large-scale model in the prompts, ensuring that the reference samples in the prompts conform to the distribution of the large-scale model's historical experience in processing tasks. This guarantees the generalization ability of the large-scale model and improves the accuracy of answers obtained from user interaction with the large-scale model. This application can also perform error correction processing on SQL results categorized into error classes. The large model learns the error correction process by analyzing the correction prompts, the corrected SQL results, the original SQL results, and the user's natural language query information. This implicitly weights the samples of error subclasses, guiding the large model to proactively avoid frequently occurring errors and reduce the accuracy of the model's output. This application primarily adjusts the large model's task processing capabilities by classifying and storing the SQL results generated by the large model in real-time into a sample library, and by determining the reference samples used to construct prompts based on the learning weights of different categories of samples in the sample library. This solidifies the historical experience of the large model in handling tasks into its capabilities, eliminating the need for full model parameter adjustments. It improves the stability of the large model and the accuracy of answers obtained from user interaction with the large model while saving computational resources. Furthermore, this application can quantify the overfitting degree of samples in the sample library and process the sample library accordingly to avoid overfitting of the large model caused by sample overfitting, ensuring the performance stability of the large model.

[0015] The following describes the learning optimization method, apparatus, equipment, media, and program products of the large model provided in this application.

[0016] This application provides a learning optimization method for large models, which can be applied to scenarios where the ability of large models to process tasks is optimized. This large model learning optimization method can be executed by a large model learning optimization device, equipment, etc., and is not limited thereto. Figure 1 A flowchart of a large model learning optimization method provided in one embodiment of this application is shown below. Figure 1As shown, the learning optimization method for this large model may include steps S101 to S104.

[0017] In step S101, natural language query information input by the user is received.

[0018] Users can interact with a client that can invoke large models. The client can receive natural language query information input by the user. The natural language query information is a natural language structure that represents the user's query task, which is related to the business scenario and the data tables involved. For example, in a transaction business scenario, it is desirable to convert the user's natural language query information into SQL results. The data table structure involved in the transaction business is as follows: “trade_record( trade_id VARCHAR(32) -- Transaction serial number user_id VARCHAR(20) -- User ID trade_amount DECIMAL(18,2) -- Transaction amount (standard field) trade_time DATETIME -- Transaction time product_type VARCHAR(50) -- Product type region_code VARCHAR(10) -- region code status TINYINT -- Transaction status ) user_profile( user_id VARCHAR(20) user_level VARCHAR(10) -- User level register_date DATE -- Registration time risk_tolerance VARCHAR(20) -- Risk tolerance )” Correspondingly, the tasks indicated by natural language query information in transaction business scenarios may include, but are not limited to, time-based aggregation query tasks, multi-dimensional filtering query tasks, and composite calculation query tasks. Time-based aggregation query tasks may include, but are not limited to, queries of monthly / quarterly / annual transaction summaries; multi-dimensional filtering query tasks may include, but are not limited to, queries that combine filtering by user level, region, and product type; and composite calculation query tasks may include, but are not limited to, queries of cumulative amount, year-on-year and month-on-month comparisons, and percentage analysis. For example, the natural language query information entered by the user may be "Query the monthly transaction summary for August 2026," or "Please filter C1 level users and C2 level users in region B1." These natural language query information are just examples and are not limited to these.

[0019] In step S102, reference samples are obtained from the sample library. Based on the natural language query information and the reference samples, the first large model for prompt word input is constructed, and the first SQL result output by the first large model is obtained.

[0020] The sample library stores samples used to construct prompt words. These samples may include historical data obtained from the first main model. The sample library stores samples of different categories, each residing in its corresponding sample set. In some examples, the sample categories include correct and incorrect categories. If a sample belongs to the correct category, it is a correct sample, also known as a positive sample, and is stored in the sample set corresponding to that category. If a sample belongs to the incorrect category, it is an incorrect sample. An incorrect category may include at least one incorrect subclass, and incorrect samples are stored in the sample set corresponding to their respective incorrect subclass. For example, if a sample database contains 30 samples, 20 of which are correct and 10 are incorrect, and the 10 incorrect samples include 3 samples of error subclass D1, 4 samples of error subclass D2, and 3 samples of error subclass D3, then the sample database can have four sample sets: one for correct samples, one for samples of error subclass D1, one for samples of error subclass D2, and one for samples of error subclass D3. Each sample can include natural language query information and the corresponding SQL result; each sample can also include the category of the SQL result, or a category label can be configured for each sample to indicate its category. In some examples, error subclasses may include, but are not limited to, non-existent fields, obfuscated table names, missing multi-table joins, missing SQL keywords, structural incompleteness errors, bracket matching errors, function syntax errors, date format errors, aggregation logic errors, filtering logic errors, compound calculation logic errors, and time overwrite errors.

[0021] Samples can be selected from a sample library as reference samples. Reference samples include samples of the correct category and samples of the incorrect category; samples of the incorrect category include samples of each error subclass. The distribution ratio of samples of different categories in the reference samples can be determined based on the learning weights of samples of different categories in the sample library. In some examples, the learning weight of a category of samples is positively correlated with the distribution ratio of samples of that category in the reference samples; that is, the higher the learning weight of a category of samples, the higher the distribution ratio of samples of that category in the reference samples. Prompt words can be constructed based on natural language query information, the selected reference samples, and the contextual information of the natural language query information, and input into the first large language model. The contextual information of the natural language query information may include business knowledge matching the natural language query information, which can be retrieved from a pre-set knowledge base based on the natural language query information to obtain the business knowledge matching the natural language query information. The first large model can be a large model used to generate SQL results corresponding to the natural language query information. The first SQL result is the SQL result generated by the first large model based on the natural language query information and the reference samples. The first SQL result may include SQL statements; for example, the first SQL result may include the following: “SELECT SUM(trade_amount) FROM trade_record WHERE trade_time BETWEEN '2025-11-01' AND '2025-11-30'".

[0022] In step S103, the first SQL result is validated in multiple dimensions. Based on the validation results, the samples including the first SQL result and natural language query information are classified and stored in the corresponding sample set in the sample library according to the category.

[0023] Multi-dimensional validation can include validation in at least two dimensions, with each dimension potentially identifying one or more error subclasses. Multi-dimensional validation allows for the verification of the first SQL result from multiple dimensions, yielding validation results. These results can then be used to determine whether the first SQL result is erroneous, and, if so, to identify the error subclass. In some examples, multi-dimensional validation may include at least two of the following dimensions: syntax validation, field existence validation, format validation, and logical consistency validation.

[0024] Syntax validation is used to check whether the syntax of the first SQL result is correct. In some examples, validation items such as keywords, structural integrity, and bracket matching in the first SQL result can be checked. Syntax validation can be performed by a syntax parser. For example, syntax validation can be implemented by generating a parse tree using the ANother Tool for Language Recognition (ANTLR). If there are no error nodes in the parse tree, the syntax validation result is a pass; if there are error nodes in the parse tree, the syntax validation result is the error subclass corresponding to the error node.

[0025] Field existence validation is used to check if any fields are missing in the first SQL statement. In some examples, it can be used to check whether the table name and field names in the first SQL result exist in the metadata of the data tables involved in the business logic. Field existence validation can be performed through a metadata matching engine; if both the table name and field names in the first SQL result exist in the metadata, the field existence validation result is "pass"; if at least one of the table name and field names in the first SQL result does not exist in the metadata, the field existence validation result may include an error indicating that either the table name or the field name is missing. For example, field existence validation can be represented by a formula, but it is not limited to this. The formula can be expressed as MatchScore = |Schema ∩ Generated| / |Generated|, where MatchScore represents the field existence validation score, Schema represents the table name and field name in the metadata, and Generated represents the table name and field name in the first SQL result. If MatchScore=1, the field existence validation result is that the field existence validation passes. If MatchScore<1, the field existence validation result is the error subclass corresponding to the error that the table name or field name does not exist.

[0026] Format specification validation is used to verify whether the first SQL result meets the format specification. In some examples, it can validate whether date quotation marks, BETWEEN usage, function syntax, etc., in the first SQL result meet the format specification. Format specification validation can be implemented through a pre-set regular expression rule base. For example, a rule in the regular expression rule base is DATE_PATTERN = "'\d{4}-\d{2}-\d{2}'", which means the standard format of a date is 'four digits representing the year - two digits representing the month - two digits representing the day'. This is just an example; the regular expression rule base can include many different rules. If the first SQL result matches all rules in the regular expression rule base, the format specification validation result is "passed". If the first SQL result does not match all rules in the regular expression rule base, the format specification validation result includes the error subclasses corresponding to the rules that failed to match.

[0027] Logical consistency verification is used to check whether the first SQL result conforms to business rule logic. In some examples, verification items such as time range coverage, aggregation logic, and filtering conditions in the first SQL result can be checked. Logical consistency verification can be performed by the business rule engine; if the verification items in the first SQL result conform to the business rule logic of the business rule engine, the verification result of logical consistency verification is "passed"; if the verification items in the first SQL result do not conform to the business rule logic of the business rule engine, the verification result of logical consistency verification includes the error subclass corresponding to the non-conforming business rule logic. For example, logical consistency verification can be expressed by a formula, but is not limited to this. The formula can be expressed as Logic_Score = Σ(rule_i × weight_i) / Σweight_i, where Logic_Score represents the logical consistency verification score, rule_i is the score of the i-th business rule logic, representing the degree of consistency between the first SQL result and the i-th business rule logic, and weight_i represents the weight of the i-th business rule logic. If Logic_Score > 0.95, the logical consistency verification result is that the logical consistency verification passes; if Logic_Score ≤ 0.95, the logical consistency verification result includes the error subclasses corresponding to the non-compliant business rule logic.

[0028] If the validation results for each dimension in the multi-dimensional validation are all passed, then the category of the sample including the first SQL result and the natural language query information is the correct category, and the sample is stored in the sample set corresponding to the correct category in the sample library. If the validation result for at least one dimension in the multi-dimensional validation includes an error subclass, then the category of the sample including the first SQL result and the natural language query information is the error subclass in the validation result, and the sample is stored in the sample set corresponding to the error subclass in the sample library.

[0029] In step S104, the learning weights of different categories of samples are determined based on the feature information of each category of samples in the sample library.

[0030] The feature information of samples in each category reflects the characteristics of samples in that category. In some examples, the feature information of samples in a category may include statistical features and / or severity features; statistical features can characterize the statistical behavior of samples in that category, for example, statistical features may include, but are not limited to, at least one of frequency, recurrence rate, and interval duration; severity features can characterize the intensity of the influence of samples in that category, for example, severity features may include, but are not limited to, the degree of influence. For example, the feature information of samples in an error subclass may include, but is not limited to, one or more of the following: the frequency of occurrence of the error corresponding to that error subclass, the recurrence rate of the error corresponding to that error subclass, the degree of influence of the error corresponding to that error subclass, and the error interval duration of the error corresponding to that error subclass.

[0031] The learning weights for each category of samples are calculated based on the feature information of samples in the sample library. The learning weight of a sample in a category represents the attention the large model pays to samples of that category. The higher the learning weight of a sample in a category, the higher the attention the large model pays to samples of that category, and the better it can learn from samples of that category. When samples in the sample library are updated, the learning weights of the updated categories are updated accordingly. The updated learning weights and the updated samples in the sample library work together to determine the reference samples used to construct prompt words. This application realizes a closed-loop mechanism that selects reference samples based on the learning weights of samples from different categories to construct prompt words, performs large-scale model inference, the first SQL result output by the large model forms a sample classification trajectory and is stored in the sample library, and updates to the sample library drive updates to the learning weights. This mechanism can balance the accuracy of the SQL results output by the large model with the generalization ability of the large model.

[0032] It should be noted that step S104 can be executed after step S103, before step S103, or simultaneously with step S103; the timing of execution of step S104 is not limited here. The learning weights of different categories of samples in the sample library can be calculated offline or online; this is not limited here.

[0033] In this embodiment, reference samples can be selected from the sample library based on the learning weights of samples of different categories in the sample library. Prompt words are constructed based on the reference samples and the natural language query information input by the user. The prompt words are input into the first large model to obtain the first SQL result output by the first large model. The first SQL result and the natural language query information are used as samples. The samples are promptly stored in the corresponding sample set in the sample library according to the category determined by the first SQL result through multi-dimensional verification. This allows the first large model to refer to the latest samples when performing tasks in the future. The update of samples in the sample library drives the update of the learning weights of the samples. Based on the learning weights of the samples in the sample library, the distribution ratio of different categories of samples in the reference samples required to construct prompt words is adjusted so that the samples in the prompt words are consistent with the distribution trend of the large model's task processing. This improves the accuracy of the SQL results obtained by the user interacting with the large model and also ensures the generalization ability of the large model.

[0034] In some embodiments, the learning weights of samples of each error subclass in the sample library can be obtained; the sampling probability of each error subclass sample can be obtained according to the proportional relationship between the learning weights of samples of each error subclass; and samples can be selected as reference samples from the sample set corresponding to the error subclass in the sample library according to the sampling probability of the error subclass sample. The sampling probability of a sample of an error subclass can be determined as the proportion of the learning weight of a sample of an error subclass to the sum of the learning weights of all error subclass samples. The sampling probability of a sample of an error subclass can reflect the distribution ratio of the sample of that error subclass in the reference samples. For example, the sampling probability of a sample of an error subclass can be obtained according to the following formula (1): p_i(t) = w_i(t) / Σw_j(t) (1) Where p_i(t) is the sampling probability of the sample in the i-th error subclass; w_i(t) is the learning weight of the sample in the i-th error subclass; and Σw_j(t) is the sum of the learning weights of the samples in all error subclasses.

[0035] According to the sampling probability of samples of each error subclass, a corresponding number of samples can be selected as reference samples from the sample set corresponding to that error subclass in the sample library, so that the proportion of samples of each category in the reference samples conforms to the allocation ratio. Specifically, the ratio of the allocation ratio of samples of an error subclass to the allocation ratio of samples of the error category in the reference samples is equal to the sampling probability of samples of that error subclass, and the allocation ratio of samples of the error category is the sum of the allocation ratios of samples of all error subclasses in the reference samples. For example, if the sample library contains four sample sets, corresponding to the correct category, error subclass D1, error subclass D2, and error subclass D3 respectively, and if the allocation ratio of the correct category samples in the reference sample is 95%, the allocation ratio of the error category samples in the reference sample is 5%, the sampling probability of error subclass D1 is 20%, the sampling probability of error subclass D2 is 20%, and the sampling probability of error subclass D3 is 10%, then the allocation ratio of error subclass D1 samples in the reference sample is 2%, the allocation ratio of error subclass D2 samples in the reference sample is 2%, and the allocation ratio of error subclass D3 samples in the reference sample is 1%. If the total number of samples in the reference sample is 100, then 95 samples should be selected from the sample set corresponding to the correct category, 2 samples should be selected from the sample set corresponding to error subclass D1, 2 samples should be selected from the sample set corresponding to error subclass D2, and 1 sample should be selected from the sample set corresponding to error subclass D3 as the reference sample.

[0036] In some examples, the target number of samples to be selected from the sample set corresponding to each error subclass can be determined according to the sampling probability of the samples in the sample set corresponding to the error subclass; the priority parameter of the samples in the sample set corresponding to the error subclass can be determined based on the similarity parameter between the samples in the sample set corresponding to the error subclass and the natural language query information, the learning weight of the samples in the error subclass, and the sample timeliness factor; for any sample set corresponding to an error subclass, the target number of samples are selected as reference samples in descending order of priority represented by the priority parameter.

[0037] The total number of samples in the reference sample can be preset. Based on the total number of samples in the reference sample and the sampling probability of samples from the error subclass, the number of samples from each error subclass in the reference sample can be determined. Each error subclass can correspond to a target number of samples, which is the number of samples from that error subclass that should be selected in the reference sample. For example, in the sample library mentioned above, there are four sample sets, corresponding to the correct category, error subclass D1, error subclass D2, and error subclass D3, respectively. In the example where the total number of samples in the reference sample is 100, the target number of samples from error subclass D1 is 2, the target number of samples from error subclass D2 is 2, and the target number of samples from error subclass D3 is 1.

[0038] Each sample set in the sample library contains multiple samples. First, a priority parameter is determined for each sample in the sample set. This priority parameter represents the sample's priority. Samples with relatively higher priority are selected from the sample sets in the sample library as reference samples. Natural language query information in the samples corresponding to the error subclass can be converted into vectors. The user-input natural language query information is converted into query vectors. The similarity parameter between the vector corresponding to the sample and the query vector is determined as the similarity parameter between the sample in the sample set corresponding to the error subclass and the natural language query information. This similarity parameter reflects the semantic similarity between the natural language query information in the sample and the user-input natural language query information. The similarity represented by the similarity parameter is positively correlated with the priority represented by the priority parameter; the higher the similarity represented by the similarity parameter, the higher the priority represented by the priority parameter. The learning weight of the samples corresponding to the error subclass is also positively correlated with the priority represented by the priority parameter; the higher the learning weight of the samples corresponding to the error subclass, the higher the priority represented by the priority parameter. The sample timeliness factor reflects the impact of the time interval between the occurrence time of the error corresponding to the error subclass and the current time on the priority of the samples of that error subclass. In some examples, the sample timeliness factor is negatively correlated with the time interval between the occurrence time of the error corresponding to the error subclass and the current time, and positively correlated with the priority represented by the priority parameter. The longer the time interval between the occurrence time of the error corresponding to the error subclass and the current time, the smaller the sample timeliness factor, the greater the attenuation effect on the priority, and the lower the priority represented by the priority parameter. For example, the similarity parameter between the samples in the sample set corresponding to the error subclass and the natural language query information input by the user can be obtained according to the following formula (2), and the priority parameter can be obtained according to the following formulas (3) and (4): Similarity(q, QA_i)=CosineSim(Embedding(q),Embedding(QA_i.query))(2) Priority_i = w_i(t) × Similarity(q, QA_i) × Recency_i (3) Recency_i = 1 / (1+Δdays) (4) Where Similarity(q, QA_i) is the similarity parameter between the sample in the sample set corresponding to the error subclass and the natural language query information, ranging from [0,1]; Embedding(q) is the query vector converted from the natural language query information input by the user; Embedding(QA_i.query) is the vector converted from the natural language query information in the sample in the sample set corresponding to the error subclass; CosineSim() is the cosine similarity algorithm; Priority_i is the priority parameter; w_i(t) is the learning weight of the sample in the error subclass; Recency_i is the sample timeliness factor, ranging from [0,1].

[0039] After determining the priority parameters of the samples in the sample set, for any sample set corresponding to an error subclass, the top N samples can be selected as reference samples in descending order of priority, where N is the target quantity for that error subclass. For example, if the target quantity for error subclass D1 is 2, the target quantity for error subclass D2 is 2, and the target quantity for error subclass D3 is 1, then in the sample set corresponding to error subclass D1, the two samples with the highest priority are selected as reference samples; in the sample set corresponding to error subclass D2, the two samples with the highest priority are selected as reference samples; and in the sample set corresponding to error subclass D3, the one sample with the highest priority is selected as a reference sample.

[0040] By adjusting the learning weights of samples from error subclasses, the distribution ratio of samples from each category in the reference sample can be dynamically adjusted. This makes the distribution of samples from different categories in the reference sample more consistent with the historical experience of the large model in processing tasks, thus fully utilizing the historical experience of the large model in processing tasks. While ensuring the generalization ability of the large model, this also improves the accuracy of answers obtained by users interacting with the large model. By prioritizing the selection of reference samples from the sample set, the most representative samples in each error subclass can also be used as reference samples to guide and improve the reasoning ability of the large model, thereby further improving the accuracy of answers obtained by users interacting with the large model.

[0041] In some embodiments, the allocation ratio of error category samples in the reference sample can also be automatically and dynamically adjusted. Specifically, the allocation ratio of error category samples in the reference sample can be determined based on the upper limit of the error category ratio, the basic error category ratio, and the risk parameters of the natural language query information. The allocation ratio of error category samples in the reference sample can be used to determine the total number of error category samples in the reference sample. The upper limit of the error category ratio is the upper limit of the allocation ratio of error category samples in the reference sample, which can be set according to the scenario, requirements, experience, etc. For example, the upper limit of the error category ratio can be set to 30%, but it is not limited to this. The basic error category ratio is the basic allocation ratio of error category samples in the reference sample, which can be set according to the scenario, requirements, experience, etc. For example, the basic error category ratio can be set to 10%, but it is not limited to this. The risk parameters of the natural language query information can characterize the potential risk level of the large model outputting incorrect SQL results based on the natural language query information. This risk parameter can be related to one or more of the following factors: semantic complexity of the natural language query information, ambiguity of the natural language query information, stability of the table structure involved in the natural language query information, frequency of errors corresponding to error subclasses of samples similar to the natural language query information, and degree of impact of errors corresponding to error subclasses of samples similar to the natural language query information. For example, the higher the semantic complexity of the natural language query information, the larger the risk parameter; the higher the ambiguity of the natural language query information, the larger the risk parameter; the lower the stability of the table structure involved in the natural language query information, the larger the risk parameter; the more frequent the errors corresponding to error subclasses of samples similar to the natural language query information, the larger the risk parameter; the higher the degree of impact of errors corresponding to error subclasses of samples similar to the natural language query information, the larger the risk parameter. For example, the distribution ratio of error categories in the reference sample can be obtained according to the following formula (5): ratio_wrong(q) = min[ratio_max, ratio_base + η × Score_risk(q)] (5) Where ratio_wrong(q) is the proportion of samples of the error category in the reference sample; ratio_max is the upper limit of the error category proportion; ratio_base is the base proportion of the error category; Score_risk(q) is the risk parameter of natural language query information; η is the sensitivity coefficient, which can take a value of 0.5, but is not limited to this.

[0042] The allocation ratio of error category samples in the reference sample can be dynamically adjusted based on the risk of natural language query information. When there is a high risk in the natural language query information, the allocation ratio of error category samples can be increased, so that the large model pays more attention to error category samples and improves the accuracy of the SQL results output by the large model.

[0043] In some embodiments, the learning weights of samples in each error subclass can be obtained based on the existing samples in the sample library. Specifically, the learning weights of samples in each error subclass can be determined according to the baseline weight, the frequency of occurrence of the error corresponding to the error subclass, the recurrence rate of the error corresponding to the error subclass, and the degree of influence of the error corresponding to the error subclass. The baseline weight is the baseline value of the learning weight, which can be set to 1, but is not limited to this. Samples in the same error subclass correspond to the same type of error. The frequency of occurrence of the error corresponding to the error subclass can be the historical frequency of occurrence of the error corresponding to the error subclass, which can be obtained by reading historical logs and statistically analyzing the records in the historical logs; the frequency of occurrence of the error corresponding to the error subclass is positively correlated with the learning weights of the samples in the error subclass. The recurrence rate of the error corresponding to the error subclass is the frequency of repeated occurrence of the error corresponding to the error subclass, which can be obtained by the ratio of the number of repeated occurrences of the error corresponding to the error subclass to the total number of occurrences of all errors; the recurrence rate of the error corresponding to the error subclass is positively correlated with the learning weights of the samples in the error subclass. The degree of influence of the error corresponding to the error subclass can be quantified. For example, the impact of an error corresponding to an error subclass can be represented by a positive integer, ranging from [1, 3]; an impact of 1 indicates a slight impact; an impact of 2 indicates a moderate impact; and an impact of 3 indicates a severe impact. The impact of an error corresponding to an error subclass is positively correlated with the learning weight of the samples in that error subclass. For example, the learning weight of the samples in the i-th error subclass can be obtained according to the following formula (6): w_i(t)=w_base×[1+α×f_i(t) / Σf_j(t)+β×s_i+γ×r_i(t)] (6) Where w_i(t) is the learning weight of the sample in the i-th error subclass; w_base is the baseline weight; f_i(t) is the frequency of occurrence of the error corresponding to the i-th error subclass; Σf_j(t) is the sum of the frequencies of occurrence of the errors corresponding to all error subclasses; s_i is the degree of influence of the error corresponding to the i-th error subclass; r_i(t) is the recurrence rate of the error corresponding to the i-th error subclass; α, β and γ are adjustment coefficients that satisfy α+β+γ=1 and can be calibrated by Bayesian optimization.

[0044] In some embodiments, the learning weights of samples in the error subclass can be incrementally updated based on the addition of new samples to the sample set in the sample library. Specifically, when a new sample is added to the sample set corresponding to the error subclass, the learning weights of the newly added sample in the error subclass can be updated based on the learning weights obtained from the last update of the samples in that error subclass, the error interval duration, the time decay coefficient, and the weight increment.

[0045] The error interval is the time interval between a newly occurring error in the same subclass and the previous error in the same subclass. In other words, the error interval is the time interval between the error corresponding to a newly added sample and the previous error in the same subclass. A time decay mechanism can be introduced based on the error interval for updating the learning weights. The longer the error interval, the greater the impact of time decay on the learning weights, and the relatively lower the learning weights will be. The weight increment can characterize the impact of a newly added sample in an error subclass on the learning weights of the samples in that error subclass. The weight increment can be determined based on the new weight coefficient, the frequency of occurrence of the error corresponding to that subclass, and the degree of influence of the error corresponding to that subclass. The new weight coefficient is a pre-set coefficient that can be set according to the scenario, requirements, experience, etc. For example, the new weight coefficient can be set to 0.3. The weight increment is positively correlated with the frequency of occurrence of the error corresponding to that subclass. The weight increment is positively correlated with the degree of influence of the error corresponding to that subclass. For example, the update of the learning weights of the samples in the error subclass can be obtained according to the following formulas (7) and (8): w_i(t) = w_i(t-1) × exp(-λ_decay × Δt_i) + w_new(t) (7) w_new(t) = δ × [1 + log(1 + f_i(t))] × s_i (8) Where w_i(t) is the learning weight of the sample of the i-th error subclass in the current iteration; w_i(t-1) is the learning weight of the sample of the i-th error subclass in the previous iteration; λ_decay is the decay coefficient, which can be set according to the scenario, requirements, experience, etc., for example, it can be set to 0.01 / day; Δt_i is the error interval, which can be the number of days between the current error and the previous error in the same error subclass; w_new(t) is the weight increment; δ is the new weight coefficient; f_i(t) is the frequency of occurrence of the error corresponding to the i-th error subclass; s_i is the degree of influence of the error corresponding to the i-th error subclass.

[0046] For example, the error category includes the subcategory of field misuse. After the large model outputs the first SQL result, the first SQL result is validated in multiple dimensions to determine that the category of the sample including the first SQL result is the subcategory of field misuse. The original frequency of the field misuse error is 150. When the newly formed sample is aggregated into the sample set corresponding to field misuse, the frequency of the field misuse error is 150+1=151. According to the above formulas (7) and (8), the weight increment w_new = 0.3 × [1 + log(1 + 151)] × 2.5=4.52 can be calculated. If the error interval is 30 days and the learning weight of the previous field misuse sample is 1.57, then after the addition of the field misuse sample, the learning weight of the field misuse sample is 1.57 × exp(-0.01 × 30) + 4.52=5.68. The update of the learning weights of the samples with misused fields will lead to the update of the sampling probability of field misuse. The sampling probability of field misuse is 5.68 / (5.68 + 1.01 + 0.74) = 76.4%, where 1.01 and 0.74 are the learning weights of the samples of the other two error subclasses.

[0047] The learning weights for each error subclass can be determined based on the distribution of existing samples in the sample library, and can also be incrementally updated as new samples generated by the large model's online processing tasks are added to the sample library. The learning weights for different error subclasses can be dynamically adjusted based on the distribution trend of samples generated from historical data in the sample library, thereby adjusting the proportion of samples of different error subclasses in the reference samples provided to the large model. This ensures that the reference samples in the prompts conform to the distribution of historical experience of the large model's processing tasks, guaranteeing the generalization ability of the large model and improving the accuracy of answers obtained by users interacting with the large model.

[0048] In some embodiments, a reflective analysis can be performed on each error subclass to correct the first SQL result corresponding to the error subclass, and the first large model can learn this correction process to improve the reasoning ability and accuracy of the first large model's output answer. Specifically, the second large model can be invoked to perform correction reasoning based on samples of the error category, outputting correction prompts and the second SQL result; training corpus can be generated based on the samples, correction prompts, and the second SQL result; the first large model can be trained using the training corpus to incrementally adjust the local parameters of the first large model.

[0049] The second major model is the large model used for error correction. It can construct error correction prompts, including those for the first SQL result. These prompts guide the second model's analysis, automatically summarizing error patterns in the first SQL result and generating error correction information for the second SQL result. The error correction information provides targeted guidance. For example, for the first SQL result with a misused field subclass, the error correction information might include "The standard name for the transaction amount field is trade_amount, not trade_money / total_money." For the first SQL result with an incorrect time format subclass, the error correction information might include "Time range queries must use the BETWEEN 'start date' AND 'end date' format, and date values ​​must be enclosed in single quotes." After rule validation, the error correction information can be used to form training data and stored in the corpus. For example, in a transaction business scenario, the corpus may contain correction prompts such as "the standard name for the transaction amount field is trade_amount, not trade_money / total_money," "time range queries must use the BETWEEN 'start date' AND 'end date' format," "date values ​​must be enclosed in single quotes, in the format 'YYYY-MM-DD'," and "month queries need to automatically calculate the start and end dates of the current month (November: 2025-11-01 to 2025-11-30)." The second SQL result is the SQL result obtained after the first SQL result has been corrected by the second major model.

[0050] The training corpus can be structured as triples, quadruples, quintuples, septuples, etc., and is not limited here. For example, the training corpus can be structured as a triple of "first SQL result - correction prompt - second SQL result", or as a quadruple of "natural language query information - first SQL result - correction prompt - second SQL result", or as a quintuple of "natural language query information - first SQL result - error subclass - correction prompt - second SQL result", or as a septuple of "natural language query information - first SQL result - error subclass - correction prompt - second SQL result", or as a septuple of "natural language query information - first SQL result - error subclass - correction prompt - second SQL result - correction label - classification confidence". Here, the correction label is used to identify the correction process, and the classification confidence is the confidence that the first SQL result is classified as an error subclass. The first large model is trained using the training corpus, and only the local parameters of the first large model are incrementally adjusted; the full model parameters of the first large model do not need to be adjusted.

[0051] By training the first large model with training corpus, the first large model can learn the derivation process from incorrect SQL results to correct SQL results, rather than simply memorizing the correct answers. This can more effectively improve the reasoning ability of the first large model, thereby improving the accuracy of the answers obtained by users interacting with the large model.

[0052] In some embodiments, overfitting can be detected in the sample library, and the overfitting status of the samples in the sample library can be quantified into an overfitting index. The overfitting index is used to determine whether overfitting exists in the samples in the sample library. If overfitting is found, the sample library can be processed to reduce the overfitting risk of the samples in the sample library, thereby reducing the overfitting risk of using the first large model that includes the samples in the sample library. Specifically, the overfitting index of the sample library can be calculated based on the information entropy, cross-scene accuracy parameters, and the number of semantically repeated samples in the sample library. If the overfitting index indicates that the samples in the sample library are overfitted, some samples in the sample library can be deleted, and / or new samples can be added to the sample library to reduce the overfitting risk.

[0053] The information entropy of samples in the sample library can characterize the distribution of samples of different categories in the sample library. The cross-scene accuracy parameters of samples in the sample library include parameters related to the accuracy of the same sample on the first language model in different scenarios. In some examples, the cross-scene accuracy parameters may include the cross-scene accuracy change and the cross-scene accuracy baseline; the cross-scene accuracy change can characterize the change in the accuracy of the same sample on the first language model in different scenarios, and the range can be [-1,1]; the cross-scene accuracy baseline characterizes the baseline of cross-scene accuracy. There are some samples with the same or similar semantics in the sample library. Samples with the same or similar semantics can be regarded as samples with semantic repetition. The number of semantically repetitive samples can be the number of semantically repetitive samples. Semantically repetitive samples can be samples with semantic similarity higher than the similarity threshold. For example, semantically repetitive samples can be samples with semantic similarity higher than 0.9. For example, the overfitting index of the sample library can be obtained according to the following formulas (9) to (11): R_overfit(t)=λ1×(H_max-H(t)) / H_max+λ2×ΔAcc(t) / Acc_base +λ3×Σ(N_dup(t) / N_total(t)) (9) H(t) = -Σ[p_i(t) × log2(p_i(t))] (10) H_max = log2(K) (11) Where R_overfit(t) is the overfit index of the sample library, which represents the degree of overfitting of the samples in the sample library. The larger the value, the more severe the overfitting. The range of the overfit index can be [0,1]. H_max is the theoretical maximum entropy. K is the number of classes of the samples in the sample library. H(t) is the information entropy of the samples in the sample library. (H_max-H(t)) / H_max can represent the difference between the distribution of the samples in the sample library and the theoretical most uniform state. The larger the value, the more uneven the distribution of the samples in the sample library. ΔAcc(t) is the change in accuracy across scenarios. Acc_base is the accuracy base across scenarios. The baseline value; ΔAcc(t) / Acc_base can represent the degree of drastic change in cross-scene accuracy relative to the cross-scene accuracy baseline value; N_dup(t) is the number of semantically repeated samples of a certain semantic; N_total(t) represents the total number of samples in the sample library; Σ(N_dup(t) / N_total(t)) is the sum of the ratios of semantically repeated samples in the sample library to the total number of samples, which can reflect the homogeneity of samples in the sample library; p_i(t) is the sampling probability of the i-th error subclass; λ1, λ2 and λ3 are weight coefficients, which can be set to 0.4, 0.4 and 0.2 respectively, but are not limited to these.

[0054] An overfitting threshold can be preset. If the overfitting index is greater than or equal to the threshold, the overfitting index is determined to characterize overfitting of samples in the sample library. In some examples, if samples in the sample library are overfitted, some samples can be deleted to reduce the degree of overfitting. In other examples, if samples in the sample library are overfitted, a sufficient number of samples can be added to the sample library to reduce the degree of overfitting. These added samples can include those with sufficiently low semantic similarity to existing samples in the sample library, thereby increasing the diversity of samples in the library. The error rate index of different error subclasses in the sample library can be used to smooth the demand gap for diverse samples for each error subclass in the sample library; the total number of samples to be supplemented can be determined according to the overfitting index, the number of samples in the sample library, the risk threshold and the expansion coefficient; the number of samples to be supplemented for each error subclass can be determined according to the total number of samples to be supplemented and the demand gap for diverse samples for each error subclass; new diversified samples are added to the sample library according to the number of samples to be supplemented for each error subclass. The diversified samples include one or more of the following: samples with semantic similarity to existing samples in the sample library less than the diversification threshold, synthetic samples, random perturbation samples, etc. For example, the number of samples to be supplemented for each error subclass in the case of sample overfitting in the sample library can be obtained according to the following formulas (12) to (15): N_add(t) = ceil[N_total(t) × (R_overfit(t)-θ) / (1-θ) × μ] (12) p_error(k, t) = λ_smooth × p_error(k, t-1) + (1-λ_smooth) × I(error_t ∈ k) (13) D_gap(k) = max[0, p_target(k) - p_current(k)] (14) N_add(k) = N_add(t)×D_gap(k) / ΣD_gap(j) (15) Where N_add(t) is the total number of samples added; ceil represents rounding up; N_total(t) is the number of samples in the sample library; R_overfit(t) is the overfitting index; θ is the risk threshold, which can be set to 0.5, but is not limited to this; μ is the augmentation, which can be set to 1.5, but is not limited to this; p_error(k, t) is the exponential smoothing of the error rate of the k-th error subclass in this current operation; λ_smooth is the smoothing coefficient, which can be set to 0.8, but is not limited to this; p_error(k, t-1) is the exponential smoothing of the error rate of the k-th error subclass in the previous operation; I(error_t ∈ k) is an indicator function. If the error category generated this time is the kth error subclass, the value is the first value. If the error category generated this time is not the kth error subclass, the value is the second value. The first and second values ​​can be set according to the scenario, needs, experience, etc., and are not limited here; D_gap(k) is the demand gap of the kth error subclass for diverse samples; p_target(k) is the target error rate exponential smoothing of the kth error subclass; N_add(k) is the number of samples added for the i-th error subclass; ΣD_gap(j) is the sum of the demand gaps of all error subclasses for diverse samples.

[0055] In some embodiments, to prevent overfitting of samples in the sample library, data filtering processing can be performed on the samples in the sample library to improve the sample quality. Data filtering processing may include, but is not limited to, at least one of semantic duplication cleaning processing, historical distribution consistency verification processing, forgetting prevention detection processing, and cross-scenario generalization ability detection processing.

[0056] Semantic duplication cleaning can incorporate semantic vector computation to convert natural language query information in samples into vectors. Semantic similarity is then calculated using these vectors to identify redundant samples with different sentence structures but consistent semantics. A redundancy similarity threshold can be set; samples with semantic similarity exceeding this threshold are identified as redundant and removed from the sample library. By removing highly semantically repetitive redundant samples, the sample library is ensured to cover as many error subclasses as possible, and the reference samples in the prompts also cover as many error subclasses as possible, thereby improving the accuracy of the first model's output answer. This semantic duplication request processing can be performed when overfitting metrics indicate overfitting in the sample library, or it can be performed periodically or under other circumstances; it is not limited here.

[0057] Historical distribution consistency verification compares the distribution of samples of each category in the sample library with the distribution of historical data from online task processing to detect whether the overall data distribution has been disrupted due to oversampling of frequently occurring erroneous subclasses. If such disruption occurs, the sampling probability can be adjusted or general samples can be added to the sample library or reference samples to ensure the distribution of samples of each category in the sample library and reference samples aligns with the real business scenario, thus guaranteeing the generalization ability of the primary model. This historical distribution consistency verification can be performed when the overfitting index indicates overfitting of samples in the sample library, or it can be performed periodically or under other circumstances; it is not limited here.

[0058] The forgetting prevention detection process adds a certain number of correct-class samples and / or diversified samples to the sample library. Details regarding diversified samples can be found in the relevant descriptions above and will not be repeated here. The performance of the primary model on the added correct-class samples and / or diversified samples can be monitored in real time. If the accuracy of the primary model in processing correct samples and / or diversified samples decreases by more than a threshold while learning new incorrect subclass samples, an alarm can be triggered. This will adjust the proportion of samples of different classes in the sample library and the allocation ratio of samples of different classes in the reference samples, ensuring the generalization ability of the primary model and its accuracy in processing samples of multiple classes.

[0059] Cross-scenario generalization ability detection can utilize a pre-set independent test set (i.e., out-of-distribution data) to simulate unprecedented query scenarios for pre-inference. This analyzes the performance of the primary model when faced with new sentence structures in natural language queries and new table structures, assessing whether the primary model is overly dependent on specific historical data in the sample database. If it is determined that the primary model is overly dependent on specific historical data in the sample database, the sample database can be expanded by adding new and diverse samples to broaden the cognitive boundaries of the primary model and prevent it from being limited to specific historical scenarios.

[0060] To facilitate understanding, the following example illustrates the specific process of learning and optimizing large models. Figure 2 A schematic diagram illustrating an example of the learning and optimization process for a large model provided in an embodiment of this application, as shown below. Figure 2 As shown, the learning and optimization process of this large model may include steps a1 to a6.

[0061] In step a1, the entire sample is accumulated online.

[0062] The first SQL result output by the first model based on the natural language query information input by the user is obtained in real time and a sample is formed. For details, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0063] In step a2, a sample library and a corpus are constructed.

[0064] The first SQL results are classified, collected, and stored in the sample database. The second model is used to correct the first SQL results of the error category to obtain the second SQL results. The training corpus is then obtained and stored in the corpus. For details, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0065] In step a3, a reference sample is constructed from the prompt words.

[0066] Based on the learning weights of samples of different categories in the sample library, the allocation ratio of samples of different categories in the reference samples is determined. Samples are selected from the sample sets of different categories in the sample library as reference samples. For details, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0067] In step a4, the sample library is subjected to data screening and overfitting detection.

[0068] For details on data filtering and processing, and overfitting detection based on overfitting indices, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0069] In step a5, the first large model is incrementally fine-tuned.

[0070] Incremental fine-tuning of the first major model can be achieved by adjusting the distribution ratio of samples in the reference samples in the prompt words, or by incrementally fine-tuning the local model parameters of the first major model using the training corpus. For details, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0071] In step a6, the first major model and its version verification are evaluated.

[0072] The performance parameters of the first-largest model can be validated on a pre-defined validation set to test for overfitting. Performance parameters may include, but are not limited to, syntactic accuracy and logical coverage. If the performance parameters of the first-largest model fail to meet the requirements or if overfitting is observed, the previous version of the first-largest model will still be used for task processing. If the performance parameters of the first-largest model meet the requirements and overfitting is not observed, a new version of the first-largest model can be released, and the process can return to step a1 to continue accumulating new samples.

[0073] In some examples, Figure 3 A schematic diagram illustrating an example of the process for constructing a sample library and corpus provided in this application embodiment, as shown below. Figure 3 As shown, step a2 above can be further refined into steps b1 to b10.

[0074] In step b1, the first SQL result is validated in multiple dimensions. If the multi-dimensional validation passes, step b2 is executed; otherwise, step b3 is executed.

[0075] In step b2, the samples that include the first SQL result are assigned to the sample set corresponding to the correct category.

[0076] In step b3, samples including the first SQL result are categorized into the sample set corresponding to the error subclass.

[0077] In step b4, reference samples are selected from the sample library to construct prompt words.

[0078] In step b5, the inference of the first model is corrected using prompt words.

[0079] In step b6, the second major model is used to correct the first SQL of the error category to obtain the second SQL result.

[0080] In step b7, the second SQL result is validated in multiple dimensions. If the multi-dimensional validation passes, proceed to step b8; if the multi-dimensional validation fails, proceed to step b10.

[0081] In step b8, training corpora are generated. Each training corpus includes a first SQL result, correction prompts, and a second SQL result.

[0082] In step b9, a manual review is performed.

[0083] In step b10, the training data is stored in the corpus.

[0084] The specific details of steps b1 to b9 can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0085] In some examples, Figure 4 This is a schematic diagram illustrating an example of a data screening and overfitting detection process for a sample library provided in an embodiment of this application. Figure 4 As shown, step a4 above can be further refined into steps c1 to c8.

[0086] In step c1, the sample to be processed is obtained.

[0087] In step c2, semantic duplication cleaning is performed.

[0088] In step c3, historical distribution consistency verification is performed.

[0089] In step c4, forgetting prevention detection processing is performed.

[0090] In step c5, cross-scenario generalization capability detection processing is performed.

[0091] In step c6, determine whether the overfitting index of the samples in the sample library meets the standard. If it does not meet the standard, it means that the samples in the sample library are overfitting, and proceed to step c7; if it meets the standard, it means that the samples in the sample library are not overfitting, and proceed to step c8.

[0092] In step c7, a diverse range of samples are added to the sample library.

[0093] In step c8, the sample library is obtained.

[0094] The relevant content of steps c1 to c8 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0095] The large model learning optimization method adopted in the embodiments of this application can realize a closed-loop architecture of automatic sample classification and collection, dynamic prompt word construction, large model inference, overfitting detection, and adaptive optimization. No manual intervention is required throughout the process. It can achieve performance optimization of the first large model in a spiral manner, while taking into account the accuracy of SQL result generation and the generalization ability of the large model.

[0096] This application also provides a learning optimization apparatus for large models, corresponding to the learning optimization method for large models in the above embodiments. Figure 5 A schematic diagram of the structure of a large model learning optimization device provided in an embodiment of this application is shown below. Figure 5 As shown, the learning optimization device 200 of the large model may include a receiving module 201, an SQL result generation module 202, a classification and aggregation module 203, and a learning weight determination module 204.

[0097] The receiving module 201 can be used to receive natural language query information input by the user.

[0098] The SQL result generation module 202 can be used to obtain reference samples from the sample library, construct the first large model based on the natural language query information and the reference samples, and obtain the first SQL result output by the first large model. The distribution ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library.

[0099] The classification and aggregation module 203 can be used to perform multi-dimensional verification on the first SQL result. Based on the obtained verification results, the samples including the first SQL result and natural language query information are classified and stored in the corresponding sample set in the sample library according to the category. The categories of the samples include correct categories and error categories. The error category includes at least one error subclass.

[0100] The learning weight determination module 204 can be used to determine the learning weights of different categories of samples based on the feature information of each category of samples in the sample library.

[0101] In some embodiments, the SQL result generation module 202 can be used to: obtain the learning weights of samples of each error subclass in the sample library; obtain the sampling probability of samples of each error subclass based on the proportional relationship between the learning weights of samples of each error subclass; and select samples as reference samples from the sample set corresponding to the error subclass in the sample library according to the sampling probability of the samples of the error subclass.

[0102] In some examples, the SQL result generation module 202 can be used to: determine the target number of samples to be selected from the sample set corresponding to each error subclass according to the sampling probability of the samples of the error subclass; determine the priority parameter of the samples in the sample set based on the similarity parameter between the samples in the sample set corresponding to the error subclass and the natural language query information, the learning weight of the samples of the error subclass, and the sample timeliness factor; and for any sample set corresponding to an error subclass, select the target number of samples as reference samples in descending order of priority represented by the priority parameter.

[0103] In some embodiments, the learning weight determination module 204 can be used to: determine the learning weight of each sample in each error subclass based on the baseline weight, the frequency of occurrence of errors corresponding to the error subclass, the recurrence rate of errors corresponding to the error subclass, and the degree of influence of errors corresponding to the error subclass; or, when a new sample is added to the sample set corresponding to an error subclass, update the learning weight of the sample in the error subclass of the new sample based on the learning weight obtained from the last update of the sample in that error subclass, the error interval duration, the time decay coefficient, and the weight increment, wherein the error interval duration is the interval duration between the newly occurring error in that error subclass and the last error in that error subclass, and the weight increment is determined based on the new weight coefficient, the frequency of occurrence of errors corresponding to that error subclass, and the degree of influence of errors corresponding to that error subclass.

[0104] In some embodiments, multi-dimensional validation includes validation of at least two of the following dimensions: syntax validation, field existence validation, format conformity validation, and logical consistency validation.

[0105] Among them, syntax validation is used to check whether the syntax of the first SQL result is incorrect; field existence validation is used to check whether the first SQL is missing any fields; format conformity validation is used to check whether the first SQL result meets the format specifications; and logical consistency validation is used to check whether the first SQL result conforms to the business rule logic.

[0106] In some embodiments, the learning optimization device 200 of the large model may further include a training corpus acquisition module and an incremental training module.

[0107] The training corpus acquisition module can be used to: call the second model to perform corrective inference based on samples of the wrong category, output corrective prompts and the second SQL result; and generate training corpus based on the samples, corrective prompts and the second SQL result.

[0108] The incremental training module can be used to: train the first large model using the training corpus, and then incrementally adjust the local parameters of the first large model.

[0109] In some embodiments, the learning optimization apparatus 200 of the large model may further include an overfitting processing module. The overfitting processing module may be used to: calculate an overfitting index for the sample library based on the information entropy of the samples in the sample library, the cross-scene accuracy parameter, and the number of semantically repeated samples; if the overfitting index indicates that the samples in the sample library are overfitted, delete some samples in the sample library, and / or add new samples to the sample library to reduce the risk of overfitting.

[0110] It should be noted that the learning optimization device 200 of the large model is a device corresponding to the learning optimization method of the large model described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0111] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, as shown below. Figure 6 As shown, the electronic device 300 includes a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302.

[0112] In some examples, the processor 302 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0113] Memory 301 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the learning optimization method for a large model according to embodiments of this application.

[0114] The processor 302 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 301, in order to implement the large model learning optimization method in the above embodiments.

[0115] In some examples, the electronic device 300 may also include a communication interface 303 and a bus 304. For example, Figure 5 As shown, the memory 301, processor 302, and communication interface 303 are connected through bus 304 and complete communication with each other.

[0116] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 303.

[0117] Bus 304 includes hardware, software, or both, that couples components of electronic device 300 together. For example, and not as a limitation, bus 304 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 304 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0118] This application also provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are executed by a processor, they can implement the large model learning and optimization method described in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0119] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the learning optimization method of the large model in the above embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0120] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0121] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0122] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A learning optimization method for large models, characterized in that, include: Receive natural language query information input by the user; Reference samples are obtained from the sample library. Based on the natural language query information and the reference samples, a prompt word input first model is constructed to obtain the first SQL result output by the first model. The allocation ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library. The first SQL result is validated in multiple dimensions. Based on the validation results, the samples including the first SQL result and the natural language query information are classified and stored in the corresponding sample set in the sample library according to the category. The categories of the samples include correct categories and error categories. The error category includes at least one error subclass. Based on the feature information of each category of samples in the sample library, the learning weights of different categories of samples are determined.

2. The method according to claim 1, characterized in that, Obtaining reference samples from the sample library includes: Obtain the learning weights of samples for each error subclass in the sample library; Based on the proportional relationship between the learning weights of samples in each error subclass, the sampling probability of samples in each error subclass is obtained; Based on the sampling probability of the samples of the error subclass, samples are selected from the sample set corresponding to the error subclass in the sample library as the reference samples.

3. The method according to claim 2, characterized in that, The step of selecting samples as reference samples from the sample set corresponding to the error subclass in the sample library according to the sampling probability of the samples of the error subclass includes: Based on the sampling probability of samples in each error subclass, determine the target number of samples to be selected from the sample set corresponding to each error subclass; Based on the similarity parameters between the samples in the sample set corresponding to the error subclass and the natural language query information, the learning weights of the samples in the error subclass, and the sample timeliness factor, the priority parameters of the samples in the sample set are determined. For any sample set corresponding to an error subclass, the target number of samples are selected as the reference samples in descending order of priority as represented by the priority parameter.

4. The method according to claim 1, characterized in that, The step of determining the learning weights for different categories of samples based on the feature information of each category of samples in the sample library includes: The learning weights of samples in each error subclass are determined based on the baseline weights, the frequency of occurrence of errors corresponding to the error subclasses, the recurrence rate of errors corresponding to the error subclasses, and the degree of impact of errors corresponding to the error subclasses. or, When a new sample is added to the sample set corresponding to an error subclass, the learning weight of the new sample in the error subclass is updated based on the learning weight, error interval duration, time decay coefficient, and weight increment obtained from the last update of the sample in that error subclass. The error interval duration is the interval between the newly occurring error in that error subclass and the previous error in that error subclass. The weight increment is determined based on the new weight coefficient, the frequency of occurrence of the error corresponding to that error subclass, and the degree of influence of the error corresponding to that error subclass.

5. The method according to claim 1, characterized in that, Multi-dimensional validation includes validation of at least two of the following dimensions: Syntax validation, field existence validation, format conformity validation, and logical consistency validation; The syntax check is used to check whether the syntax of the first SQL result is incorrect; the field existence check is used to check whether the first SQL is missing any fields; the format conformity check is used to check whether the first SQL result meets the format specifications; and the logical consistency check is used to check whether the first SQL result conforms to the business rule logic.

6. The method according to claim 1, characterized in that, Also includes: The second model is invoked to perform corrective inference based on samples belonging to the incorrect category, and corrective prompts and the second SQL result are output. Based on the samples, correction prompts, and the second SQL result, a training corpus is generated. The first large model is trained using the training corpus to perform incremental adjustment of local parameters of the first large model.

7. The method according to claim 1, characterized in that, Also includes: Based on the information entropy, cross-scene accuracy parameters, and number of semantically repeated samples in the sample library, the overfitting index of the sample library is calculated. If an overfitting index indicates that the samples in the sample library are overfitted, some samples in the sample library may be deleted, and / or new samples may be added to the sample library to reduce the risk of overfitting.

8. A learning optimization device for a large model, characterized in that, include: The receiving module is used to receive natural language query information input by the user; The SQL result generation module is used to obtain reference samples from the sample library, construct a prompt word input first model based on the natural language query information and the reference samples, and obtain the first SQL result output by the first model. The distribution ratio of samples in the reference samples is determined based on the learning weights of samples of different categories in the sample library. The classification and aggregation module is used to perform multi-dimensional verification on the first SQL result. Based on the obtained verification results, the sample including the first SQL result and the natural language query information is classified and stored in the corresponding sample set in the sample library according to the category. The categories of the sample include correct category and error category. The error category includes at least one error subclass. The learning weight determination module is used to determine the learning weights of different categories of samples based on the feature information of each category of samples in the sample library.

9. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the learning and optimization method for large models as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the learning and optimization method for a large model as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the learning optimization method for a large model as described in any one of claims 1 to 7.