Large model scene question and answer optimization method and system based on preference learning

Abstract

The invention discloses a large model scene question and answer optimization method and system based on preference learning, and the method comprises the steps: obtaining the query data of a user in a specific scene application, and generating a preset number of candidate answers for each question; introducing a human feedback mechanism, obtaining a preference sequence of a preset number of candidate answers corresponding to each question by a human, and forming a feedback data set; and carrying out iterative training on the large model by using the feedback data set in combination with a loss function of preference alignment, so as to realize optimization and adjustment of parameters of the large model. Through the technical scheme of the invention, different business scenes can be more flexibly adapted, the effectiveness in practical application is improved, the adaptability of the model is remarkably improved, the accuracy of answers is improved, the semantic correlation of the answers is ensured, and the efficiency of model training is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model scenario question and answer optimization method based on preference learning and a large model scenario question and answer optimization system based on preference learning. Background Art

[0002] With the rapid advancement of artificial intelligence technology, the potential of big models in enterprise applications has gradually emerged. Big models have powerful computing power and wide applicability, and can handle a variety of tasks, from data analysis to automated decision-making. However, enterprises often face a key challenge when applying big models: how to closely integrate these general models with the specific business needs of the enterprise. The business scenarios and operating processes of different enterprises vary greatly, and existing big models cannot directly meet the needs of all scenarios.

[0003] At present, when developing large model applications, enterprises usually rely on open source general large models to build business scenarios. In most cases, enterprises directly call the general large model through prompt word engineering to generate scenario questions and answers. However, in order to further improve the question-answering ability of the general large model in specific scenarios, some enterprises use the company's internal annotated data for supervised fine-tuning (SFT). Although this method can improve the performance of the model to a certain extent, it still has certain limitations for certain specific application scenarios.

[0004] For enterprise applications, large models are usually required to generate answers with specific language styles and domain knowledge. However, since general large models usually lack in-depth knowledge of specific fields, and the quality and quantity of data used for fine-tuning are limited, it is difficult to fully meet these high-demand application scenarios by relying solely on prompt word engineering and fine-tuning. For example, in some professional fields, such as law, medicine, or finance, the model not only needs to accurately understand complex terminology, but also needs to generate answers based on the rules and conventions of specific fields. This kind of specialized demand is obviously beyond the capabilities of general large models, and simple fine-tuning cannot significantly improve the professionalism of the model.

[0005] Therefore, although cue word engineering and supervised fine-tuning are currently common methods for optimizing large models, for enterprise application scenarios that require high customization and domain-specific knowledge, it is still necessary to explore more effective training methods to make up for the shortcomings of general large models in specific domain knowledge. Summary of the invention

[0006] In response to the above problems, the present invention provides a large-model scenario question-answering optimization method and system based on preference learning. By introducing a human feedback mechanism and utilizing a human preference alignment algorithm, the large model is optimized and trained using feedback data to make it better adapt to the actual application scenarios of the enterprise, thereby significantly improving the adaptability of the model. At the same time, the present invention provides a complete enterprise large-model scenario-based human feedback training system, providing enterprises with a comprehensive and systematic solution for large-model scenario development.

[0007] To achieve the above object, the present invention provides a large model scenario question answering optimization method based on preference learning, comprising:

[0008] Obtain query data from users in specific scenario applications and generate a preset number of candidate answers for each question;

[0009] Introduce a human feedback mechanism to obtain human preference rankings for a preset number of candidate answers for each question, forming a feedback data set;

[0010] Combined with the loss function of preference alignment, the feedback data set is used to iteratively train the large model to achieve optimal adjustment of the parameters of the large model.

[0011] In the above technical solution, preferably, the large model scenario question answering optimization method based on preference learning also includes:

[0012] An independent validation set is used to test and validate the trained large model, and the large model is fine-tuned until the large model reaches the preset indicators.

[0013] In the above technical solution, preferably, the process of obtaining the query data of the user in the specific scenario application and generating a preset number of candidate answers for each question includes:

[0014] Acquire query data of the user in the large model of the required scenario application, wherein the query data includes question data, conversation data and search record data;

[0015] For the query data, a pre-trained large model is used to generate multiple answers by adjusting generation parameters;

[0016] Introduce specific annotated data in the corresponding scenario as the standard reference answer;

[0017] The answer generated by the large model is combined with the standard reference answer as a candidate answer, and a corresponding candidate set of questions and the candidate answers is generated.

[0018] In the above technical solution, preferably, the human feedback mechanism is introduced to obtain the preference ranking of a preset number of candidate answers corresponding to each question by humans to form a feedback data set, and the specific process includes:

[0019] Collecting human feedback results on candidate answers to each question through a human-computer interaction interface, wherein the feedback results include scoring, drag-and-drop sorting, or best answer selection;

[0020] Sort or rank the feedback results for each question and convert them into a format that can be recognized and processed;

[0021] The format files for different problems are collected and verified, and unreasonable or noisy data are filtered to obtain the feedback data set.

[0022] In the above technical solution, preferably, the loss function combined with the preference alignment uses the feedback data set to iteratively train the large model to achieve optimization adjustment of the parameters of the large model, and the specific process includes:

[0023] A loss function is set based on a preference alignment algorithm, where the loss function is Pairwise Ranking Loss or Listwise Loss;

[0024] Using the feedback data set as training data, a gradient descent optimization algorithm is used to adjust the parameters of the large model;

[0025] Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to optimize the performance of the large model.

[0026] The present invention also proposes a large model scenario question answering optimization system based on preference learning, which applies the large model scenario question answering optimization method based on preference learning disclosed in any one of the above technical solutions, including:

[0027] A candidate answer generation module is used to obtain the query data of users in a specific scenario application and generate a preset number of candidate answers for each question;

[0028] The human preference feedback module is used to introduce a human feedback mechanism to obtain the human preference ranking of a preset number of candidate answers for each question to form a feedback data set;

[0029] The preference alignment training module is used to combine the loss function of the preference alignment and use the feedback data set to iteratively train the large model to achieve optimal adjustment of the parameters of the large model.

[0030] In the above technical solution, preferably, the large model scenario question and answer optimization system based on preference learning also includes a model testing and verification module, which is used to test and verify the trained large model using an independent verification set, and fine-tune the large model until the large model reaches the preset indicators.

[0031] In the above technical solution, preferably, the candidate answer generation module is specifically used to:

[0032] Acquire query data of the user in the large model of the required scenario application, wherein the query data includes question data, conversation data and search record data;

[0033] For the query data, a pre-trained large model is used to generate multiple answers by adjusting generation parameters;

[0034] Introduce specific annotated data in the corresponding scenario as the standard reference answer;

[0035] The answer generated by the large model is combined with the standard reference answer as a candidate answer, and a corresponding candidate set of questions and the candidate answers is generated.

[0036] In the above technical solution, preferably, the human preference feedback module is specifically used for:

[0037] Collecting human feedback results on candidate answers to each question through a human-computer interaction interface, wherein the feedback results include scoring, drag-and-drop sorting, or best answer selection;

[0038] Sort or rank the feedback results for each question and convert them into a format that can be recognized and processed;

[0039] The format files for different problems are collected and verified, and unreasonable or noisy data are filtered to obtain the feedback data set.

[0040] In the above technical solution, preferably, the preference alignment training module is specifically used for:

[0041] A loss function is set based on a preference alignment algorithm, where the loss function is Pairwise Ranking Loss or Listwise Loss;

[0042] Using the feedback data set as training data, a gradient descent optimization algorithm is used to adjust the parameters of the large model;

[0043] Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to optimize the performance of the large model.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Improved adaptability of large models in enterprise scenarios: By introducing a preference learning mechanism, the present invention can optimize the performance of large models in a customized manner according to the specific needs of different enterprise scenarios. Compared with the traditional single model training method, scenario-based training enables large models to adapt to different business scenarios more flexibly, improving their effectiveness in practical applications.

[0046] 2. Enhanced accuracy and relevance of answers generated by the model: Using human feedback data, the present invention can optimize model parameters so that the answers it generates are more in line with the actual needs of users. This process not only improves the accuracy of answers, but also ensures the semantic relevance of answers, especially in scenarios that require specific domain knowledge.

[0047] 3. Effective use of human feedback improves model optimization efficiency: By ranking candidate answers and learning preferences, the present invention can effectively use human feedback to optimize the parameters of large models. This method greatly improves the efficiency of model training and avoids the inefficiency or inaccuracy that may occur in traditional methods.

[0048] 4. Supports multiple rounds of iterative optimization and improves model performance: The present invention allows multiple rounds of iterative training, and further improves the performance of the model by continuously introducing new data and fine-tuning. Compared with the one-time training method, multiple rounds of iteration can better capture and adapt to changing business needs, ensuring the long-term effectiveness of the model in practical applications.

[0049] 5. Provides a systematic enterprise large model scenario development solution: This invention not only proposes a specific training method, but also designs a complete system framework, including a training module, a feedback data acquisition module, and an inference generation module. This systematic solution enables enterprises to deploy and maintain large models more conveniently, lowers the technical threshold, and accelerates the application of large models in enterprises.

[0050] 6. Optimize resource utilization and improve model training and reasoning efficiency: The present invention makes efficient use of hardware resources, especially in GPU acceleration and distributed computing. This optimization of resource utilization not only reduces training costs, but also improves the real-time and accuracy of model reasoning, meeting the efficient needs of enterprises for large-scale models. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a large-model scenario question-answering optimization method based on preference learning disclosed in an embodiment of the present invention;

[0052] Figure 2A logical schematic diagram of a large-model scenario question-answering optimization system based on preference learning disclosed in an embodiment of the present invention.

[0053] In the figure, the corresponding relationship between each component and the reference numeral is as follows:

[0054] 1. Candidate answer generation module, 2. Human preference feedback module, 3. Preference alignment training module. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0057] like Figure 1 As shown, a large model scenario question answering optimization method based on preference learning provided by the present invention includes:

[0058] Obtain query data from users in specific scenario applications and generate a preset number of candidate answers for each question;

[0059] Introduce a human feedback mechanism to obtain human preference rankings for a preset number of candidate answers for each question, forming a feedback data set;

[0060] Combined with the preference alignment loss function, the feedback dataset is used to iteratively train the large model to achieve optimal adjustment of the large model parameters.

[0061] In this embodiment, by introducing a human feedback mechanism and utilizing a human preference alignment algorithm, the feedback data is used to optimize the training of the large model so that it can better adapt to the actual application scenarios of the enterprise, thereby significantly improving the adaptability of the model. At the same time, the present invention provides a complete enterprise large model scenario-based human feedback training system, providing a comprehensive and systematic solution for enterprises in large model scenario development.

[0062] Specifically, in specific scenario applications, the user's query data is first collected, and a large model is used to generate multiple diverse answers. In addition, the labeled data in the scenario is used as the standard answer. Finally, multiple candidate answers are generated for the same question so that these answers can be labeled with preferences through human feedback in subsequent steps. By obtaining human feedback on the above candidate answers, the quality of these candidate answers is ranked. Specifically, human preference rankings for multiple answers to the same question are collected, and these ranking data can be used in subsequent steps to optimize the parameters of the large model through preference learning. Combined with the loss function of preference alignment, the parameters of the large model are adjusted and optimized as the optimization goal, so that the large model can generate answers that are more in line with human preferences.

[0063] Among them, in order to collect feedback on human preference ranking of candidate answers, it is necessary to obtain the I / O system required for human feedback annotation. The input data is the answer data file (.json file) generated by the large model, and the output is the sorted answer data file (.json file).

[0064] Among them, for the training of large model parameters, it is necessary to use a dataset for preference alignment and a development environment required to support training, including an operating system, a container for executing the training algorithm, and hardware conditions (such as memory, CPU, and GPU). The input data includes a feedback dataset file (.json file) generated through human feedback and a model parameter file (.bin file) of the basic large model.

[0065] In the above implementation, preferably, query data of users in a specific scenario application is obtained, and a preset number of candidate answers are generated for each question. The specific process includes:

[0066] Obtain the query data of the user in the big model of the required scenario application, including question data, conversation data and search record data in actual business operations;

[0067] For query data questions, use a pre-trained large model to generate multiple answers by adjusting generation parameters (such as temperature, top-k sampling, etc.) to ensure the diversity of answers;

[0068] Introduce specific annotated data in the corresponding scenario as standard reference answers. These data are usually provided by domain experts.

[0069] The answer generated by the large model is combined with the standard reference answer as a candidate answer to generate a corresponding candidate set of one question and multiple candidate answers.

[0070] In this embodiment, the key points are:

[0071] (1) Ensure the diversity of responses to improve the effectiveness of subsequent sorting;

[0072] (2) The quality of the labeled data directly affects the final results and needs to be carefully produced by domain experts;

[0073] (3) Reasonably set generation parameters to balance the diversity and accuracy of answers.

[0074] The pre-trained big model contains the data set for big model reasoning and the environment required to support the reasoning process, including the operating system, the container for performing reasoning, and the hardware conditions (such as memory, CPU, and GPU). The input data includes the question data file (.json file) in the scene and the big model parameter file (.bin file) generated by the training module, and the output is the candidate answer data file (.json file) generated by the big model reasoning.

[0075] In the above implementation, preferably, a human feedback mechanism is introduced to obtain the preference ranking of a preset number of candidate answers corresponding to each question by humans to form a feedback data set. The specific process includes:

[0076] Collect human feedback on candidate answers to each question through a specially designed human-computer interaction interface or tool. The feedback results include scoring, drag-and-drop sorting, or best answer selection.

[0077] Sort or rank the feedback results for each question and convert them into a machine-readable format file (such as a .json file);

[0078] Collect and verify format files for different problems, filter out unreasonable or noisy data, ensure data integrity and consistency, and obtain feedback data sets.

[0079] In this implementation, the quality and quantity of human feedback are crucial, and participants should have sufficient domain knowledge. In the process of collecting human feedback, a user-friendly evaluation interface should be designed to allow human evaluators to rank or score candidate answers, reduce the user's workload, and improve the efficiency of data collection. After data collection, it is necessary to process it in a timely manner. Through verification and filtering, the feedback data is standardized and organized to ensure the validity and availability of the feedback data, which is crucial for subsequent model training.

[0080] In the above implementation, preferably, the large model is iteratively trained using the feedback data set in combination with the loss function of preference alignment to achieve optimization adjustment of the large model parameters. The specific process includes:

[0081] The loss function is set based on the preference alignment algorithm to make the answers generated by the model closer to human preferences. The loss function is Pairwise Ranking Loss or Listwise Loss.

[0082] Using the feedback data set as training data, the gradient descent optimization algorithm is used to adjust the parameters of the large model;

[0083] Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to prevent overfitting and optimize the performance of large models.

[0084] In this implementation, the choice of loss function should be adapted to the actual application scenario to ensure that the optimization goal is consistent with the business needs. When training the model, care should be taken to prevent overfitting and to reasonably select the number of training rounds and learning rate.

[0085] Specifically, in a high-performance computing environment, use the designed loss function and optimization algorithm to iteratively train large model parameters. Use GPU to accelerate the training process and save model parameters regularly. The hardware configuration must meet the requirements of large-scale model training and ensure efficient use of resources during training. Regularly monitor the changes in the loss function during training so that the training strategy can be adjusted in a timely manner.

[0086] In the above implementation, preferably, the large model scenario question answering optimization method based on preference learning also includes:

[0087] An independent validation set is used to test and verify the trained large model to evaluate its performance in real scenarios, and the large model is fine-tuned until it reaches the preset indicators.

[0088] The data of the validation set and test set should be relatively independent of the training data to accurately evaluate the generalization ability of the model.

[0089] According to the large-model scenario question-answering optimization method based on preference learning disclosed in the above-mentioned implementation mode, the implementation process of the method is specifically described by taking its application in the enterprise recruitment assistant scenario as an example.

[0090] Scene background:

[0091] Corporate recruitment assistants need to handle a large number of tasks such as resume screening, job matching, candidate evaluation, etc. These tasks not only require the system to understand the professional background of candidates, but also to give appropriate suggestions and decisions based on the specific needs and culture of the company.

[0092] Implementation steps:

[0093] Step S01: Obtain candidate answers

[0094] (1) User query collection: Recruiters enter queries such as “recommend candidates suitable for this position” or “screen candidates with advanced Python skills”.

[0095] (2) Answer generation: The system's large model generates multiple candidate answers based on the query, such as a recommended candidate list, job matching analysis, etc. At the same time, it uses the company's annotated data (such as data on previous successful recruitment cases) as a reference to generate standard answers.

[0096] (3) Candidate answer combination: Ultimately, the system generates a set of candidate answers, such as recommending different candidates for a specific position or suggesting an interview priority.

[0097] Step S02: Sorting using human feedback

[0098] (1) Feedback collection: Recruiting experts or human resources personnel evaluate and rank multiple candidate responses based on the recommendations given by the system. For example, they may sort the candidate list by match or mark suggestions that are more in line with the corporate culture.

[0099] (2) Data organization: These feedback data are converted into a standardized format for subsequent preference learning. The data contains the ranking information of each recommended answer and the expert’s preference description.

[0100] Step S03: Optimization through Preference Learning

[0101] (1) Model optimization: Use feedback data to learn preferences and adjust the parameters of the large model so that it can better reflect the actual needs of recruiters in future recommendations. For example, the model will learn how to more accurately assess the match between a candidate’s soft skills and corporate culture.

[0102] (2) Iterative training: After multiple rounds of iterative training, the system is increasingly able to capture the recruitment preferences of companies during the recommendation process, for example, it is more inclined to recommend candidates with a spirit of teamwork.

[0103] like Figure 2 As shown, the present invention also proposes a large model scenario question answering optimization system based on preference learning, which applies the large model scenario question answering optimization method based on preference learning disclosed in any one of the above embodiments, including:

[0104] The candidate answer generation module 1 is used to obtain the query data of the user in the specific scenario application and generate a preset number of candidate answers for each question;

[0105] Human preference feedback module 2 is used to introduce a human feedback mechanism to obtain human preference rankings for a preset number of candidate answers for each question to form a feedback data set;

[0106] The preference alignment training module 3 is used to combine the preference alignment loss function and use the feedback data set to iteratively train the large model to achieve optimization adjustment of the parameters of the large model.

[0107] In the above-mentioned embodiment, preferably, the large-model scenario question-answering optimization system based on preference learning also includes a model testing and verification module, which is used to test and verify the trained large model using an independent verification set, and fine-tune the large model until the large model reaches the preset indicators.

[0108] In the above implementation, preferably, the candidate answer generating module 1 is specifically used for:

[0109] Obtain the user's query data of the big model in the required scenario application, the query data includes question data, conversation data and search record data;

[0110] For questions about query data, use a pre-trained large model to generate multiple answers by adjusting the generation parameters;

[0111] Introduce specific annotated data in the corresponding scenario as the standard reference answer;

[0112] The answers generated by the large model are combined with the standard reference answers as candidate answers to generate corresponding candidate sets of questions and candidate answers.

[0113] In the above implementation, preferably, the human preference feedback module 2 is specifically used for:

[0114] Through the human-computer interaction interface, human feedback on candidate answers for each question is collected. The feedback results include scoring, drag-and-drop sorting, or best answer selection;

[0115] Sort or rank the feedback results for each question and convert them into a format that can be recognized and processed;

[0116] Collect and verify format files for different problems, filter out unreasonable or noisy data, and obtain feedback data sets.

[0117] In the above implementation, preferably, the preference alignment training module 3 is specifically used for:

[0118] Set the loss function based on the preference alignment algorithm. The loss function is Pairwise Ranking Loss or Listwise Loss.

[0119] Using the feedback data set as training data, the gradient descent optimization algorithm is used to adjust the parameters of the large model;

[0120] Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to optimize the performance of the large model.

[0121] According to the large-model scenario question and answer optimization system based on preference learning disclosed in the above-mentioned implementation mode, the functions to be implemented by each module thereof correspond to the respective steps of the large-model scenario question and answer optimization method based on preference learning disclosed in the above-mentioned implementation mode. During the implementation process, operations are performed with reference to the above-mentioned implementation mode, which will not be repeated here.

[0122] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large-model scenario question answering optimization method based on preference learning, characterized in that: include: Obtain query data from users in specific scenario applications and generate a preset number of candidate answers for each question; Introduce a human feedback mechanism to obtain human preference rankings for a preset number of candidate answers for each question, forming a feedback data set; Combined with the loss function of preference alignment, the feedback data set is used to iteratively train the large model to achieve optimal adjustment of the parameters of the large model.

2. The large-model scenario question-answering optimization method based on preference learning according to claim 1 is characterized in that: Also includes: An independent validation set is used to test and validate the trained large model, and the large model is fine-tuned until the large model reaches the preset indicators.

3. The large-model scenario question-answering optimization method based on preference learning according to claim 2 is characterized in that: The process of obtaining the query data of the user in the specific scenario application and generating a preset number of candidate answers for each question includes: Acquire query data of the user in the large model of the required scenario application, wherein the query data includes question data, conversation data and search record data; For the query data, a pre-trained large model is used to generate multiple answers by adjusting generation parameters; Introduce specific annotated data in the corresponding scenario as the standard reference answer; The answer generated by the large model is combined with the standard reference answer as a candidate answer, and a corresponding candidate set of questions and the candidate answers is generated.

4. The large-model scenario question-answering optimization method based on preference learning according to claim 3 is characterized in that: The human feedback mechanism is introduced to obtain the preference ranking of a preset number of candidate answers corresponding to each question by humans to form a feedback data set. The specific process includes: Collecting human feedback results on candidate answers to each question through a human-computer interaction interface, wherein the feedback results include scoring, drag-and-drop sorting, or best answer selection; Sort or rank the feedback results for each question and convert them into a format that can be recognized and processed; The format files for different problems are collected and verified, and unreasonable or noisy data are filtered to obtain the feedback data set.

5. The large-model scenario question-answering optimization method based on preference learning according to claim 4 is characterized in that: The loss function combined with the preference alignment uses the feedback data set to iteratively train the large model to achieve optimization adjustment of the parameters of the large model. The specific process includes: A loss function is set based on a preference alignment algorithm, where the loss function is Pairwise Ranking Loss or Listwise Loss; Using the feedback data set as training data, a gradient descent optimization algorithm is used to adjust the parameters of the large model; Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to optimize the performance of the large model.

6. A large-model scenario question-answering optimization system based on preference learning, characterized in that: The large-model scenario question-answering optimization method based on preference learning as claimed in any one of claims 1 to 5 is applied, comprising: A candidate answer generation module is used to obtain the query data of users in a specific scenario application and generate a preset number of candidate answers for each question; The human preference feedback module is used to introduce a human feedback mechanism to obtain the human preference ranking of a preset number of candidate answers for each question to form a feedback data set; The preference alignment training module is used to combine the loss function of the preference alignment and use the feedback data set to iteratively train the large model to achieve optimal adjustment of the parameters of the large model.

7. The large-model scenario question-answering optimization system based on preference learning according to claim 6 is characterized in that: It also includes a model testing and verification module, which is used to use an independent verification set to test and verify the trained large model, and to fine-tune the large model until the large model reaches the preset indicators.

8. The large-model scenario question-answering optimization system based on preference learning according to claim 7 is characterized in that: The candidate answer generation module is specifically used for: Acquire query data of the user in the large model of the required scenario application, wherein the query data includes question data, conversation data and search record data; For the query data, a pre-trained large model is used to generate multiple answers by adjusting generation parameters; Introduce specific annotated data in the corresponding scenario as the standard reference answer; The answer generated by the large model is combined with the standard reference answer as a candidate answer, and a corresponding candidate set of questions and the candidate answers is generated.

9. The large-model scenario question-answering optimization system based on preference learning according to claim 8 is characterized in that: The human preference feedback module is specifically used for: Collecting human feedback results on candidate answers to each question through a human-computer interaction interface, wherein the feedback results include scoring, drag-and-drop sorting, or best answer selection; Sort or rank the feedback results for each question and convert them into a format that can be recognized and processed; The format files for different problems are collected and verified, and unreasonable or noisy data are filtered to obtain the feedback data set.

10. The large-model scenario question-answering optimization system based on preference learning according to claim 9 is characterized in that: The preference alignment training module is specifically used for: A loss function is set based on a preference alignment algorithm, where the loss function is Pairwise Ranking Loss or Listwise Loss; Using the feedback data set as training data, a gradient descent optimization algorithm is used to adjust the parameters of the large model; Through multiple rounds of iterative training, the amount of data is gradually increased in each iteration or new data is introduced at different times to optimize the performance of the large model.

Images