A method for adjusting and optimizing large language models

Abstract

The present invention provides a method for adjusting and optimizing a large language model, including: collecting user feedback data based on the answer output by the first large language model according to the question input; cleaning and preprocessing the user feedback data and extracting feedback features, optimizing the answer generation strategy of the first large language model by a reinforcement learning method to obtain a second large language model; using the second large language model to perform type recognition according to the question input and convert it into a structured query statement, performing query expansion according to the structured query statement, generating an answer according to the expanded query statement, and the generated answer includes an explanatory text. The application of this method can integrate user feedback data into the training process of the model to optimize the answer generation strategy; it can provide the output basis and logic of the model to enhance the user's trust in the model output and promote the transparency of the model decision-making process, thereby improving the user's personalized demand adaptability and output reliability of the large language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an adjustment and optimization method for a large language model. Background Art

[0002] In the field of artificial intelligence, large language models such as GPT-3 and BERT have become the core technology for understanding and generating natural language. These models can handle complex language understanding and generation tasks such as text translation, sentiment analysis, dialogue systems, and content creation, and can simulate human language understanding and expression capabilities in various scenarios.

[0003] However, with the widespread application, the accuracy, relevance and interpretability of model-generated content have become the focus of research and development. This involves not only technical challenges, such as how to improve the model's sensitivity to context and how to deal with ambiguity and complex expressions, but also ethical and social issues, such as avoiding bias and misleading information.

[0004] It is very important to verify the correctness of content generated by large models, because models may generate content based on wrong or outdated data, or misunderstand complex or ambiguous information. If the content generated by the model is used without verification, it may mislead users. Accuracy is particularly important for sensitive or important topics, such as medical, legal, or scientific information. Accuracy verification also helps ensure that the content meets ethical standards and legal regulations, especially in preventing the spread of false information and maintaining data privacy.

[0005] Nowadays, the reliability verification technologies for content generated by large models include: rule-based verification methods, data-driven verification methods, and manual review and feedback methods. Among them, rule-based verification methods include using predefined grammatical and semantic rules to detect errors in content. This method may not be flexible and accurate enough when dealing with complex or non-standard usage, especially when dealing with diverse and innovative texts. This is because rules cannot cover all language usage, especially when faced with novel or uncommon expressions. Data-driven verification methods use data-driven technologies, such as machine learning models, to verify and correct generated content. This includes using historical data to train models to identify error patterns. However, since data-driven methods rely on a large amount of training data, this may cause the model to perform poorly when faced with unseen situations. The effectiveness of this method is limited by the quality and diversity of training data, and there may be a lack of effective data for emerging topics or rare situations. Using manual review and user feedback to verify and correct generated content is the most direct verification method, which usually involves manual review of model-generated content by linguists, domain experts, or ordinary users. Manual review is usually time-consuming and costly, and is inefficient when faced with large amounts of data, because manual review relies on human judgment, is limited by personal experience and subjectivity, and is inefficient when dealing with large amounts of data. In these existing methods for verifying the output content of large models, there is often a lack of transparent explanation of the verification process and results, which reduces users' trust and acceptance of the verification results.

[0006] In general, although existing large-model generated content reliability verification methods can identify and correct errors to a certain extent, they still lack flexibility, adaptability, real-time updating, and content interpretability.

[0007] Therefore, it is necessary to provide a large language model generation content optimization method that can solve the problems of poor adaptability to user personalized needs and poor output reliability of existing large language models. Summary of the invention

[0008] The purpose of the present invention is to provide a method for adjusting and optimizing a large language model, so as to improve the adaptability of the large language model to user personalized needs and the output reliability.

[0009] In a first aspect, the adjustment and optimization method of a large language model provided by the present invention includes: based on the answer output by the first large language model according to the user's question input, collecting user feedback data about the model output answer; cleaning and preprocessing the user feedback data and performing feedback feature extraction, optimizing the answer generation strategy of the first large language model by a reinforcement learning method according to the feedback feature extraction result, and obtaining a second large language model adjusted by the user feedback data; applying the second large language model to identify the question type according to the user's question input, performing structured conversion of the question based on the identified question type to convert the user's question input into a structured query statement represented in a set format, performing query expansion according to the structured query statement, and generating an answer according to the expanded query statement, wherein the generated answer contains an explanatory text.

[0010] The beneficial effect of the adjustment and optimization method of the large language model provided by the present invention is that: by designing and establishing a user feedback mechanism, user preferences and feedback are collected and learned, and the accuracy and relevance of the answers generated by the large language model are optimized, thereby not only improving the user's satisfaction with the generated content, but also enabling the model to better adapt to the ever-changing user needs and application environment. The mechanism for enhancing the interpretability of content is designed to explain the output basis and logic of the model to provide transparency and understanding of the model decision-making process, enhance the user's trust in the model output, and promote the transparency of the model decision-making process, so that users can better understand and trust the model's answers.

[0011] In a possible embodiment, performing feedback feature extraction on the user feedback data includes extracting key features from the user feedback data and obtaining corresponding feedback scores; optimizing the answer generation strategy of the first largest language model using a reinforcement learning method according to the feedback feature extraction result to obtain a second largest language model adjusted by the user feedback data, including: using the feedback score obtained according to the feedback feature extraction result as a reward signal according to a predefined reward mechanism; in the framework of reinforcement learning, the first largest language model adjusts the answer generation strategy and learns an answer generation strategy that can obtain a high-scoring answer based on the feedback score; optimizing the answer generation strategy of the first largest language model through adjustment and learning, thereby obtaining the second largest language model adjusted by the user feedback data.

[0012] In another possible embodiment, query expansion is performed according to the structured query statement, including: performing context annotation according to the structured query statement to add context information; and when performing query expansion, introducing the context information into the structured query statement.

[0013] In other possible embodiments, explanatory text key elements are set according to the question type; and answers are generated according to the expanded query statements, including: extracting explanatory text key elements according to the expanded query statements, and generating explanatory text content guided by the explanatory text key elements.

[0014] In a second aspect, the present invention further provides an adjustment and optimization device for a large language model, the device comprising: a collecting unit, for collecting user feedback data on the model output answer based on the answer output by the first large language model according to the user's question input;

[0015] A training unit, configured to clean and preprocess the user feedback data and extract feedback features, and optimize the answer generation strategy of the first language model using a reinforcement learning method according to the feedback feature extraction result, to obtain a second language model adjusted by the user feedback data;

[0016] The answer generation unit is used to apply the second largest language model to identify the question type according to the user's question input, perform structured conversion of the question based on the identified question type to convert the user's question input into a structured query statement represented in a set format, perform query expansion according to the structured query statement, and generate an answer according to the expanded query statement, wherein the generated answer includes explanatory text.

[0017] Among them, performing feedback feature extraction on the user feedback data includes extracting key features from the user feedback data and obtaining corresponding feedback scores; the training unit optimizes the answer generation strategy of the first largest language model by using a reinforcement learning method according to the feedback feature extraction result to obtain a second largest language model adjusted by the user feedback data, including: using the feedback score obtained according to the feedback feature extraction result as a reward signal according to a predefined reward mechanism; in the framework of reinforcement learning, the first largest language model adjusts the answer generation strategy and learns an answer generation strategy that can obtain a high-scoring answer based on the feedback score; the answer generation strategy of the first largest language model is optimized through adjustment and learning, thereby obtaining the second largest language model adjusted by the user feedback data.

[0018] The answer generation unit performs query expansion according to the structured query statement, including: performing context annotation according to the structured query statement to add context information; when executing query expansion, introducing the context information into the structured query statement.

[0019] The key elements of the explanatory text are set according to the question type; the answer generation unit generates an answer according to the expanded query sentence, including: extracting the key elements of the explanatory text according to the expanded query sentence, and generating the explanatory text content with the key elements of the explanatory text as guidance.

[0020] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for adjusting and optimizing the large language model is implemented.

[0021] In a fourth aspect, the present invention also provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned large language model adjustment and optimization method.

[0022] For the beneficial effects of the second to fourth aspects, reference may be made to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a flow chart of a method for adjusting and optimizing a large language model provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a large language model framework provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a large language model adjustment and optimization device provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0028] In view of the problems existing in the prior art, an embodiment of the present invention provides an adjustment and optimization method for a large language model.

[0029] This embodiment provides a method for adjusting and optimizing a large language model. Figure 1 and Figure 2 , the method comprising:

[0030] S101: Collect user feedback data on the answers output by the model based on the answers output by the first language model according to the user's question input.

[0031] In S101, in a possible embodiment, a user asks a question to the first language model, obtains an answer output by the first language model according to the user's question input, and then collects user feedback data about the output answer from the user through a user feedback plug-in set in the first language model.

[0032] In a specific embodiment, the first large language model collects user feedback data on the output answers by sending a questionnaire to the user. Exemplarily, the questionnaire can be set as follows: a scoring system is used for the following questions. 1-5 points, 5 points is the best, representing that the answer of the large model is the most appropriate, and 1 point is the lowest, representing that the answer of the large model is the least appropriate.

[0033] 1. Accuracy: The consistency between the generated answer and the factual or expected answer.

[0034] 2. Relevance: Evaluate the relevance of the generated content to the question asked by the user.

[0035] 3. Fluency: Evaluate the language fluency and naturalness of the generated content, and whether the grammar, semantics, and contextual logic are easier for users to understand.

[0036] 4. Explainability: Evaluate the explainability and transparency of the generated content, whether the content can clearly explain how the content was generated, the reasoning process, and the relevant background knowledge.

[0037] For the following questions, fill in the blanks and enter the user's emotional attitude towards the answer of the large model.

[0038] Bias and Fairness Feedback: Users provide feedback on possible bias or unfairness in the model output. This question is a fill-in-the-blank type, and the user enters their evaluation of the large language model's answer.

[0039] S102: Clean and preprocess the user feedback data and extract feedback features. According to the feedback feature extraction results, a reinforcement learning method is used to optimize the answer generation strategy of the first language model to obtain a second language model adjusted by the user feedback data.

[0040] In S102, in a possible embodiment, after collecting the user feedback data, the user feedback data is cleaned and preprocessed, specifically, including removing irrelevant or erroneous feedback, standardizing the feedback format, processing natural language data, etc. The data quality is ensured by cleaning and preprocessing the user feedback data, laying a foundation for subsequent processing. Feature extraction is performed on the user feedback data after the cleaning and preprocessing operations. The specific operation is to extract key features from the user feedback data, such as high-frequency words, sentence structures, emotional tendencies, etc., to identify user preferences, and correspond the user preferences to specific scores of 1-5.

[0041] By extracting features from user feedback data to obtain key features, user preferences are identified and user preferences are matched to specific scores, thereby obtaining feedback scores based on user feedback data.

[0042] In a possible embodiment, user feedback data is collected from users through a questionnaire in the form of an example in S101, and the obtained user feedback data can be processed to obtain corresponding feedback scores.

[0043] In a possible embodiment, a reinforcement learning method is used to optimize the answer generation strategy of the first largest language model according to the feedback feature extraction result to obtain a second largest language model adjusted by the user feedback data, including: according to a predefined reward mechanism, a feedback score obtained according to the feedback feature extraction result is used as a reward signal; in the framework of reinforcement learning, the first largest language model adjusts the answer generation strategy and learns an answer generation strategy that can obtain a high-scoring answer based on the feedback score; the answer generation strategy of the first largest language model is optimized through adjustment and learning, thereby obtaining the second largest language model adjusted by the user feedback data.

[0044] Exemplarily, after obtaining the feedback score by performing feedback feature extraction processing on the user feedback data, the feedback score can be used as a reward signal, and the generation strategy of the large language model can be optimized by the reinforcement learning method, that is, the answer generation strategy of the first large language model can be optimized by the reinforcement learning method, so as to obtain the second large language model whose answer generation strategy is optimized after the user feedback data is adjusted. Specifically, the process of optimizing the answer generation strategy of the first large language model by the reinforcement learning method according to the feedback feature extraction result to obtain the second large language model includes: defining a reward mechanism: setting the feedback score as a reward signal, for example, a high score (such as 4 or 5 points) represents positive feedback, which can be used as a positive reward; a low score (such as 1 or 2 points) represents negative feedback, which is used as a negative reward. Then, the agent and the environment are set. In this embodiment, the specific setting is: in the framework of reinforcement learning, the first large language model is used as an agent, and the environment that interacts with the user includes the user's input, the output of the first large language model, and the feedback score. The agent (i.e., the first large language model) maximizes the total reward it obtains by trying different answer strategies. Based on the feedback score, the first language model learns which types of answers are more likely to get high scores, and gradually adjusts its answer generation strategy to prefer these types of answers. Through continuous experimentation and learning, the answer generation strategy of the first language model can be optimized, and a more effective strategy can be developed to generate answers that users are more satisfied with, thereby obtaining the second language model adjusted by the user feedback data. It should be noted that the answer generation strategy optimization process of the large language model in this embodiment is iterative, and the generation strategy of the large language model will continue to be optimized with more user feedback. That is, as more user feedback data is collected, the large language model in the current state is regarded as the first language model to optimize the answer generation strategy, thereby obtaining the second language model whose answer generation strategy is optimized after the user feedback data is adjusted.

[0045] Using reinforcement learning methods, the large language model is able to dynamically optimize its answer generation strategy based on user ratings to better adapt to user needs and preferences.

[0046] In the adjustment and optimization method of the large language model of the present invention, during the stage of training the large language model, the user feedback data is directly integrated into the training and adjustment process of the large language model. The present invention enables the large language model to adapt to the specific needs and preferences of users more flexibly, and uses the user feedback data as a dynamic input to continuously optimize and adjust the model to ensure the continuous accuracy of the content.

[0047] S103: Apply the second largest language model to identify the question type according to the user's question input, perform structured conversion of the question based on the identified question type to convert the user's question input into a structured query statement represented in a set format, perform query expansion according to the structured query statement, and generate an answer according to the expanded query statement, wherein the generated answer includes explanatory text.

[0048] In S103, the second largest language model after optimizing the answer generation strategy is used to generate corresponding answers according to the user input question, so that the generated answers can flexibly adapt to the user's specific needs and preferences.

[0049] Exemplarily, the process of using the second language model to generate a corresponding answer according to the user input question includes: according to the user's question input, the question type is identified. Specifically, the question types include factual questions (questions requiring specific facts), reasoning questions (questions requiring the model to perform logical reasoning), opinion questions (questions asking for opinions or suggestions), and explanation questions (seeking explanations for a phenomenon or information). In a specific embodiment, question type identification can be achieved through methods such as keyword recognition, natural language processing technology, or machine learning classification models.

[0050] Then, based on the identified question type, the question is structurally transformed, and the identified question is converted into a predefined structured format. The predefined structured format includes the main entity, action, and purpose of the question. For example, factual questions include entity-attribute-value, reasoning questions include condition-result-reasoning basis, opinion-type questions include topic-opinion type-relevant factors, and explanatory questions include phenomenon-explanation requirement-relevant background. Based on the identified question type, the structural transformation of the question can obtain a structured query statement that represents the question in a set format.

[0051] Query expansion is performed based on structured query statements, and answers are generated based on the expanded query statements. Specifically, the purpose of query expansion based on structured query statements is to automatically add some standard queries based on the type and structure of the question so that the model can refer to them when generating answers. Query expansion includes expansion based on question type, expansion based on expected answers, and expansion based on knowledge domains. Methods for performing query expansion include keyword synonym replacement, introduction of relevant contextual information, and use of user historical data. The answer generated based on the expanded query statement contains explanatory text, specifically text that explains the content of the answer in detail, the reasoning process, and the standard answer based on the content, so that the generated answer is a comprehensive answer that integrates detailed explanations, logical reasoning, and factual basis. The content of the answer not only answers the user's question, but also provides sufficient information to support the answer.

[0052] In a possible embodiment, query expansion is performed according to a structured query statement, including: performing context annotation according to the structured query statement to add context information; when performing query expansion, the context information is introduced into the structured query statement. Among them, context annotation is to add necessary context information to the question, including: time context, location context, domain / topic context, situational context, and user intent context. According to the set requirements, the large language model can automatically add relevant context annotations according to the type and mode of the question when performing the operation. The context information added by the context annotation operation can add necessary details and depth to the explanatory text in the answer, making the explanation more comprehensive and accurate, in order to improve the accuracy of the large language model when processing complex queries and reduce the occurrence of hallucinations.

[0053] In a possible embodiment, the generation of the explanatory text includes: setting explanatory text key elements according to the question type. When generating an answer according to the expanded query sentence, the operations performed include: extracting explanatory text key elements according to the expanded query sentence, and generating explanatory text content with the explanatory text key elements as guidance.

[0054] Explanatory text generation is when a large language model generates an answer and generates a text that explains the answer at the same time. The specific explanation method is: using natural language processing technology, such as keyword extraction and dependency parsing, to generate explanatory text. The purpose of designing explanatory text when generating answers is to combine interpretability with output: to ensure that each generated answer has a clear and accurate explanation to help users understand the decision-making process of the large language model.

[0055] The components identified in the question structure transformation provide a framework and content direction for the creation of explanatory text. For example, factual questions usually contain entities, attributes, and values. When answering factual questions, the focus is on accurately providing specific facts. Explanatory texts are usually direct and concise, clearly answering the core inquiry in the question, and are usually based on reliable data sources or historical records. Inference-type questions include conditions, results, and reasoning. The answers to inference-type questions require demonstrating the logical reasoning process. Explanatory texts will contain the logic of how to derive the results from given conditions, and the basis of the content may be rules, precedents, or logical reasoning. Opinion-type questions usually involve topics, opinion types, and related factors. Explanatory texts should state the position taken and provide reasons and evidence to support the view. The basis of the content may include expert opinions, market surveys, or statistics. In terms of generating specific content of explanatory text, we utilize the text understanding capability of the large language model to identify key words and phrases in the input questions, determine the key elements of the explanatory text to be set as a guide based on the question type, and generate relevant content based on the huge database accumulated during the pre-training stage of the second large language model by optimizing the answer generation strategy of the first large language model through collecting user feedback data.

[0056] In a specific example of answer generation for user input questions, the generated answers to the question “Why does a rainbow appear after rain?” include:

[0057] Answer: A rainbow appears after rain, mainly due to the refraction, reflection and dispersion of light. When sunlight passes through a raindrop, it is first refracted on the surface of the raindrop, then reflected inside the raindrop, and finally refracted a second time when the light leaves the raindrop again. In this process, the light is decomposed into a spectrum of different colors, forming a rainbow.

[0058] Explanation of the text:

[0059] 1. Refraction: When light passes from one medium (such as air) to another medium (such as water), its speed and direction change. This is refraction. In the formation of a rainbow, sunlight first passes from the air to the raindrops, where it refracts.

[0060] 2. Reflection: After the sunlight enters the raindrops, part of the light is reflected back inside the raindrops. This is a key step in the formation of a rainbow.

[0061] 3. Dispersion: Different colors of light have different wavelengths, and they are refracted at different angles when passing through raindrops, causing the light to decompose into multiple colors in the spectrum.

[0062] Reasoning process:

[0063] Raindrops as refractive media: When light passes through a raindrop, it is refracted twice when entering and leaving the raindrop due to the spherical structure of the raindrop. Reflection of light: Inside the raindrop, the light is reflected back to the other side of the raindrop, which enhances the dispersion effect of the light. Dispersion creates rainbows: Because different colors of light are refracted at different angles, they are dispersed when leaving the raindrop, forming a rainbow.

[0064] Content based on:

[0065] Physics principle: The explanation is based on the basic principles of optics, such as the laws of refraction and reflection of light, and the phenomenon of dispersion. Scientific observation and experiment: The formation mechanism of rainbow has been verified by scientists through observation and experiment for many years, and it is a common optical phenomenon in nature.

[0066] In the adjustment and optimization method of the large language model provided by the present invention, a user feedback mechanism is designed to collect and learn user preferences and feedback, thereby optimizing the accuracy and relevance of the answers generated by the large language model, thereby not only improving user satisfaction with the generated content, but also enabling the model to better adapt to changing user needs and application environments. A mechanism to enhance content interpretability is designed to explain the output basis and logic of the model to provide transparency and understanding of the model's decision-making process, enhance user trust in the model output, and promote transparency of the model's decision-making process, so that users can better understand and trust the model's answers. Overall, the adjustment and optimization method of the large language model provided by the present invention can improve the accuracy and practicality of large language models (such as the GPT series) when generating content, thereby better serving user needs.

[0067] See the instruction manual Figure 3 This embodiment also provides a large language model adjustment and optimization device, which is used to implement the above method embodiment. The device includes:

[0068] The collecting unit 201 is used to collect user feedback data on the answers output by the model based on the answers output by the first language model according to the user's question input.

[0069] The training unit 202 is used to clean and preprocess the user feedback data and extract feedback features. According to the feedback feature extraction results, the answer generation strategy of the first language model is optimized by the reinforcement learning method to obtain the second language model adjusted by the user feedback data.

[0070] The answer generation unit 203 is used to apply the second largest language model to identify the question type according to the user's question input, perform structured conversion of the question based on the identified question type to convert the user's question input into a structured query statement represented in a set format, perform query expansion according to the structured query statement, and generate an answer according to the expanded query statement, wherein the generated answer includes explanatory text.

[0071] The feedback feature extraction of the user feedback data includes extracting key features from the user feedback data and obtaining corresponding feedback scores. The training unit 202 optimizes the answer generation strategy of the first language model using a reinforcement learning method according to the feedback feature extraction result to obtain a second language model adjusted by the user feedback data, including: using the feedback score obtained according to the feedback feature extraction result as a reward signal according to a predefined reward mechanism; in the framework of reinforcement learning, the first language model adjusts the answer generation strategy and learns the answer generation strategy that can obtain a high-scoring answer based on the feedback score; the answer generation strategy of the first language model is optimized through adjustment and learning, thereby obtaining the second language model adjusted by the user feedback data.

[0072] The answer generation unit 203 performs query expansion according to the structured query statement, including: performing context annotation according to the structured query statement to add context information; when executing query expansion, introducing the context information into the structured query statement.

[0073] The explanatory text key elements are set according to the question type; the answer generation unit 203 generates an answer according to the expanded query sentence, including: extracting the explanatory text key elements according to the expanded query sentence, and generating the explanatory text content with the explanatory text key elements as guidance.

[0074] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0075] In other embodiments of the present application, the present application discloses an electronic device, such as Figure 4 As shown, the electronic device 300 may include: one or more processors 301; a memory 302; a display 303; one or more applications (not shown); and one or more computer programs 304. The above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the above memory and configured to be executed by the one or more processors 301. The one or more computer programs 304 include instructions. The above instructions may be used to execute the following: Figure 1 And each step in the corresponding embodiment.

[0076] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0077] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0079] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for adjusting and optimizing a large language model, characterized in that: include: Collect user feedback data on the answers output by the first language model according to the user's question input; Cleaning and preprocessing the user feedback data and extracting feedback features, optimizing the answer generation strategy of the first language model using a reinforcement learning method according to the feedback feature extraction result, and obtaining a second language model adjusted by the user feedback data; Applying the second largest language model to identify the question type according to the user's question input, selecting a predefined structured format corresponding to the question type based on the identified question type to perform structural conversion of the question so as to convert the user's question input into a structured query statement represented in a set format, performing query expansion according to the structured query statement, and generating an answer according to the expanded query statement, wherein the predefined structured format includes an entity, an action, and a purpose of the question, and the generated answer includes an explanatory text; The extracting of feedback features from the user feedback data includes extracting features from the user feedback data to obtain key features to identify user preferences, and corresponding the user preferences to feedback scores; The method of reinforcement learning is used to optimize the answer generation strategy of the first largest language model according to the feedback feature extraction result to obtain the second largest language model adjusted by the user feedback data, including: according to a predefined reward mechanism, the feedback score obtained according to the feedback feature extraction result is used as a reward signal, and in the framework of reinforcement learning, the first largest language model adjusts the answer generation strategy and learns the answer generation strategy that can obtain a high-scoring answer based on the feedback score, and the answer generation strategy of the first largest language model is optimized through adjustment and learning, thereby obtaining the second largest language model adjusted by the user feedback data; Question types include factual questions, reasoning questions, opinion questions and explanatory questions. The predefined structured format of factual questions includes entities, attributes and values; the predefined structured format of reasoning questions includes conditions, results and reasoning basis; the predefined structured format of opinion questions includes topics, opinion types and related factors; the predefined structured format of explanatory questions includes phenomena, explanation requirements and related background.

2. The method according to claim 1, characterized in that Performing query expansion according to the structured query statement includes: Performing context annotation according to the structured query statement to add context information; When query expansion is performed, context information is introduced into the structured query statement.

3. The method according to claim 1, characterized in that: Set key elements of explanatory text according to question type; Generate answers based on the expanded query statements, including: Extract key elements of the explanatory text according to the expanded query sentence, and generate explanatory text content with the key elements of the explanatory text as guidance.

4. A large language model adjustment and optimization device, characterized in that: The device comprises: A collecting unit, configured to collect user feedback data on the answers output by the model based on the answers output by the first language model according to the user's question input; A training unit, configured to clean and preprocess the user feedback data and extract feedback features, and optimize the answer generation strategy of the first language model using a reinforcement learning method according to the feedback feature extraction result, to obtain a second language model adjusted by the user feedback data; an answer generation unit, configured to apply the second largest language model to identify the question type according to the question input by the user, select a predefined structured format corresponding to the question type based on the identified question type to perform structural conversion of the question so as to convert the question input by the user into a structured query statement represented in a set format, perform query expansion according to the structured query statement, and generate an answer according to the expanded query statement, wherein the predefined structured format includes the entity, action and purpose of the question, and the generated answer includes an explanatory text; The training unit extracts feedback features from the user feedback data, including extracting features from the user feedback data to obtain key features to identify user preferences, and corresponding the user preferences to feedback scores; The training unit optimizes the answer generation strategy of the first largest language model by a reinforcement learning method according to the feedback feature extraction result to obtain the second largest language model adjusted by the user feedback data, including: according to a predefined reward mechanism, using the feedback score obtained according to the feedback feature extraction result as a reward signal, in the framework of reinforcement learning, the first largest language model adjusts the answer generation strategy and learns the answer generation strategy that can obtain a high-scoring answer based on the feedback score, and optimizes the answer generation strategy of the first largest language model through adjustment and learning, thereby obtaining the second largest language model adjusted by the user feedback data; Question types include factual questions, reasoning questions, opinion questions and explanatory questions. The predefined structured format of factual questions includes entities, attributes and values; the predefined structured format of reasoning questions includes conditions, results and reasoning basis; the predefined structured format of opinion questions includes topics, opinion types and related factors; the predefined structured format of explanatory questions includes phenomena, explanation requirements and related background.

5. The device according to claim 4, characterized in that The answer generation unit performs query expansion according to the structured query statement, including: Performing context annotation according to the structured query statement to add context information; When query expansion is performed, context information is introduced into the structured query statement.

6. The device according to claim 4, characterized in that Set key elements of explanatory text according to question type; The answer generation unit generates an answer according to the expanded query statement, including: Extract key elements of the explanatory text according to the expanded query sentence, and generate explanatory text content with the key elements of the explanatory text as guidance.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for adjusting and optimizing the large language model according to any one of claims 1 to 3 is implemented.

8. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the electronic device executes the large language model adjustment and optimization method according to any one of claims 1 to 3.

Images