Conversation generation method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a dialogue generation method and device, equipment, a storage medium and a computer program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining multiple groups of official document question and answer pairs based on a corpus, and adding a quality score to an answer text in each group of official document question and answer pairs to obtain multiple groups of training question and answer pairs, the quality score is used for representing the quality of the content of the answer text; inputting the question text in each group of training question and answer pairs into a pre-trained big language model to obtain a model output text carrying a quality score, the pre-trained big language model being a big language model that has learned a natural language structure; the quality scores of the model output texts are input into a preset loss function to obtain loss values, model parameters of the large language model are updated based on the loss values to obtain an official document question and answer model, and the loss function is constructed based on a preference optimization model. According to the method, official document questions and answers can conform to values, intentions and preferences of human beings.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a dialogue generation method, apparatus, device, storage medium and computer program product. Background Art

[0002] Large Language Model (LLM) is a key technology in the field of artificial intelligence, which aims to understand and generate human language. Large language models can perform a wide range of tasks by training on large-scale text data. For example, large language models can be used for official document question and answer. Human preference alignment means that the goals, behaviors, and outputs of AI systems must be consistent with human values, intentions, and preferences. For LLM, human preference alignment is crucial because it is directly related to whether the answers generated by the model meet the expectations and needs of users. Official document question and answer refers to questions and answers raised in official document processing or related fields regarding official document processing. The current large language model processing does not perform human preference alignment for official document question and answer. LLM may not be able to accurately capture and reflect human preferences in official document question and answer, resulting in the generated official document answers may not meet human preferences.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a dialogue generation method, device, equipment, storage medium and computer program product, aiming to solve the technical problem that large language models do not conform to human preferences when replying to official documents.

[0005] To achieve the above object, the present application proposes a dialogue generation method, which is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes:

[0006] The dialogue generation method is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes:

[0007] Determine multiple groups of official document question-answer pairs based on the corpus, and add quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts;

[0008] Inputting the question text in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure;

[0009] The quality scores of the texts output by the model are input into a loss function constructed based on a preference optimization model to obtain a loss value, and the model parameters of the large language model are updated based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0010] In one embodiment, before the step of adding quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, the step further includes:

[0011] Inputting the answer texts in each group of the official document question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the official document question-answer pairs;

[0012] For any target official document question and answer pair in each group of the official document question and answer pairs, adjustment information based on the target initial score is obtained, and the target initial score is adjusted according to the adjustment information, and the adjusted target initial score is used as the quality score of the target official document question and answer pair, wherein the target initial score is the initial score of the answer text in the target official document question and answer pair.

[0013] In one embodiment, the annotation model includes a first input layer, an embedding layer, an encoding layer, an attention mechanism block, and a classification layer;

[0014] The step of inputting the answer texts in each group of the training question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the training question-answer pairs comprises:

[0015] Inputting the answer text in each group of the training question-answer pairs into the input layer to obtain a word index sequence vector;

[0016] Input the word index sequence into the embedding layer to obtain a word vector sequence;

[0017] Inputting the word vector sequence into the encoding layer to obtain an encoding feature vector;

[0018] Inputting the encoded feature vector into the attention mechanism block to obtain a weighted feature vector;

[0019] The weighted feature vector is input into the classification layer to obtain an initial score for the answer text.

[0020] In one embodiment, the step of determining a plurality of sets of official document question-answer pairs based on the corpus includes:

[0021] Determining a plurality of groups of corpus question-answer pairs from the question-answer pairs in the corpus, wherein the corpus question-answer pairs are question-answer pairs for an official document question-answering task;

[0022] Inputting each group of corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs;

[0023] The corpus question-answer pair and the extended question-answer pair are used as the official document question-answer pair.

[0024] In one embodiment, the generative adversarial network includes a second input layer, a generator, and a discriminator;

[0025] The step of inputting each group of corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs includes:

[0026] Input each group of the corpus question-answer pairs into the second input layer to obtain a random noise vector;

[0027] Inputting the random noise vector into the generator to generate a question-answer pair;

[0028] The generated question-answer pair is input into the discriminator to obtain a discrimination result. When the discrimination result is true, the generated question-answer pair is output through the generator, and the question-answer pair is used as an extended question-answer pair.

[0029] In one embodiment, before the step of inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores, the step further includes:

[0030] Inputting each group of the training question-answer pairs into a preset preference screening model to obtain high-quality samples, wherein the quality score of the high-quality samples is greater than a preset score threshold;

[0031] Using the high-quality samples and the training question-answer pairs as a training data set;

[0032] The step of inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores comprises:

[0033] The question text in the training data set is input into a pre-trained large language model to obtain a model output text carrying a quality score.

[0034] In addition, to achieve the above purpose, the present application also proposes a dialogue generation device, which is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes:

[0035] Determine multiple groups of official document question-answer pairs based on the corpus, and add quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts;

[0036] Inputting the question text in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure;

[0037] The quality scores of the texts output by the model are input into a loss function constructed based on a preference optimization model to obtain a loss value, and the model parameters of the large language model are updated based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a dialogue generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the dialogue generation method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dialogue generation method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the dialogue generation method described above are implemented.

[0041] In the present application, the dialogue generation device determines multiple groups of official document question and answer pairs based on the corpus, and adds quality scores to the answer texts in each group of the official document question and answer pairs to obtain multiple groups of training question and answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts; the question texts in each group of the training question and answer pairs are input into a pre-trained large language model to obtain model output texts carrying quality scores, wherein the pre-trained large language model is a large language model that has learned the natural language structure; the quality scores of the model output texts are input into a loss function constructed based on a preference optimization model to obtain loss values, and the model parameters of the large language model are updated based on the loss values ​​to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0042] In this application, a large language model is trained by using official document question and answering, and a loss function based on a preference optimization model is introduced during the training process to optimize the model parameters of the large language model, so that the obtained official document question and answer model gradually approaches human preferences and intentions. The preference alignment mechanism ensures that the official document question and answer model can better reflect human values, intentions and preferences when generating official document answers, thereby further improving the quality of official document question and answering using a large language model.

[0043] In addition, preference optimization enables the model to learn the key features of the official document question and answer task more quickly during the training process, thereby reducing unnecessary calculations and resource consumption. The official document question and answer model can utilize resources more efficiently when processing similar tasks, thereby improving the processing speed of official document question and answer.

[0044] In addition, compared to the existing training process of aligning large language models with human preferences, which relies on a reward model to provide feedback signals to guide the model to produce outputs that meet human preferences, in this application, the answer texts in the official document question and answer pairs used for model training have directly added quality scores, which can be directly used to construct loss functions and update model parameters. This application does not need to first train a reward model to evaluate the quality of the text, thereby reducing the training steps and complexity, reducing the dependence on additional computing resources, and thus reducing the consumption of computing resources for model training.

[0045] Moreover, compared to the existing reinforcement learning fine-tuning in the training process, which may cause the model to produce unexpected behaviors while pursuing high rewards, the loss function in this application is constructed based on quality scores. These scores directly reflect human expectations of the quality of answer texts. Therefore, during the training process, the model will strive to generate answer texts that meet these expectations, and will not produce unexpected behaviors like reinforcement learning fine-tuning, which helps to improve the stability and reliability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flowchart of the first embodiment of the method for generating a dialogue in this application is provided;

[0049] Figure 2 A flowchart of the second embodiment of the method for generating a dialogue in this application is provided;

[0050] Figure 3 A brief flowchart of a method for generating a dialogue according to an embodiment of the present application is provided;

[0051] Figure 4 This is a schematic diagram of the module structure of the dialogue generation device according to an embodiment of the present application;

[0052] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the dialogue generation method in the embodiment of the present application.

[0053] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of the embodiment of the present application is: based on the corpus, multiple groups of official document question and answer pairs are determined, and quality scores are added to the answer texts in each group of the official document question and answer pairs to obtain multiple groups of training question and answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts; the question texts in each group of the training question and answer pairs are input into a pre-trained large language model to obtain model output texts carrying quality scores, wherein the pre-trained large language model is a large language model that has learned the natural language structure; the quality scores of the model output texts are input into a loss function constructed based on a preference optimization model to obtain loss values, and the model parameters of the large language model are updated based on the loss values ​​to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0057] In this embodiment, for ease of description, the following description is made with the dialogue generation device as the execution subject.

[0058] Large Language Model (LLM) is a key technology in the field of artificial intelligence, which aims to understand and generate human language. Large language models can perform a wide range of tasks by training on large-scale text data. For example, large language models can be used for official document question and answer. Human preference alignment refers to the consistency of the goals, behaviors, and outputs of AI systems with human values, intentions, and preferences. Large language models need to be aligned with human values ​​and intentions, mainly to ensure that their behaviors and outputs meet human expectations and ethical standards. This alignment is crucial to avoid misleading, inaccurate, or harmful outputs. The process of human preference alignment involves multiple aspects, including but not limited to: Helpfulness: LLM should be able to provide useful information and solutions; Honesty: LLM should provide accurate information and clarify its limitations when uncertain; Harmlessness: LLM's behavior should not cause any form of harm or adverse effects. Official document question and answer refers to questions and answers raised in official document processing or related fields regarding official document processing and other aspects. The current large language model processing does not perform human preference alignment for official document question and answer. LLM may not accurately capture and reflect human preferences in official document question and answer, resulting in the generated official document answers may not meet human preferences.

[0059] When using large language models to handle official document question and answering, if human preference alignment is not performed, the following shortcomings may be encountered: inconsistent preferences, that is, LLM may not be able to accurately capture and reflect human preferences, resulting in the generated answers may not meet human preferences; generation quality fluctuations, that is, the quality of answers may fluctuate greatly, and it is difficult to ensure consistency and reliability; efficiency issues, that is, without preference optimization, LLM may require more time and resources to handle complex question and answer tasks, especially when it is necessary to understand and integrate a large amount of information.

[0060] Currently, the training of large language models for human preference alignment includes three steps: pre-training the language model: using traditional pre-training objectives to train a language model; then, training the reward model: training a reward model based on the data generated by the language model. This model receives a series of texts and returns a scalar reward, the value of which represents the human preference for the text; then, reinforcement learning fine-tuning: using strong proximal policy optimization (PPO) to fine-tune the language model. In this process, the feedback provided by the reward model is used as a reward signal to guide the model to produce outputs that are more in line with human preferences. However, the training of the reward model requires a lot of data and computing resources, and its accuracy directly affects the effect of subsequent reinforcement learning fine-tuning. The reinforcement learning fine-tuning process may also cause the model to produce unexpected behaviors while pursuing high rewards.

[0061] This application provides a solution by training a large language model using official document question and answer, and introducing a loss function based on a preference optimization model to optimize the model parameters of the large language model during the training process, so that the obtained official document question and answer model gradually approaches human preferences and intentions. The preference alignment mechanism ensures that the official document question and answer model can better reflect human values, intentions and preferences when generating official document answers, thereby further improving the quality of official document question and answering using a large language model.

[0062] In addition, preference optimization enables the model to learn the key features of the official document question and answer task more quickly during the training process, thereby reducing unnecessary calculations and resource consumption. The official document question and answer model can utilize resources more efficiently when processing similar tasks, thereby improving the processing speed of official document question and answer.

[0063] In addition, compared to the existing training process of aligning large language models with human preferences, which relies on a reward model to provide feedback signals to guide the model to produce outputs that meet human preferences, in this application, the answer texts in the official document question and answer pairs used for model training have directly added quality scores, which can be directly used to construct loss functions and update model parameters. This application does not need to first train a reward model to evaluate the quality of the text, thereby reducing the training steps and complexity, reducing the dependence on additional computing resources, and thus reducing the consumption of computing resources for model training.

[0064] Moreover, compared to the existing reinforcement learning fine-tuning in the training process, which may cause the model to produce unexpected behaviors while pursuing high rewards, the loss function in this application is constructed based on quality scores. These scores directly reflect human expectations of the quality of answer texts. Therefore, during the training process, the model will strive to generate answer texts that meet these expectations, and will not produce unexpected behaviors like reinforcement learning fine-tuning, which helps to improve the stability and reliability of the model.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a dialogue generation device, etc. The following takes the dialogue generation device as an example to illustrate this embodiment and the following embodiments. It should be noted that the dialogue generation device is connected to the corpus in communication.

[0066] Based on this, the present application embodiment provides a method for generating a dialogue, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the dialogue generation method of the present application.

[0067] In this embodiment, the dialogue generation method includes steps S10 to S30:

[0068] Step S10, determining multiple groups of official document question-answer pairs based on the corpus, and adding quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts;

[0069] It should be noted that official document Q&A refers to the activity of answering questions about the content of official documents or the process of official document processing. The content of official document Q&A may include the definition of official documents, processing principles, handling procedures, management requirements, etc. Official document Q&A pairs refer to question-and-answer units consisting of an official document question and its corresponding answer, where both the question and the answer are related to the official document. By analyzing official document Q&A pairs, we can understand the common problems in official document processing and how to solve them.

[0070] In this embodiment, a large number of question-answer pairs are collected from various official documents, policy documents, news reports and other sources to form an initial corpus; the question-answer pairs in the initial corpus are screened to remove duplicate, irrelevant or low-quality question-answer pairs to ensure that the remaining question-answer pairs are representative and relevant to document processing; quality scores are added to the question-answer pairs, and the quality scores can be used to measure the quality of the content of the answer text. In the document question-answering task, the quality score can be used as supervision information for the training model to help the model learn how to generate high-quality answers. It should be noted that the quality score of the screened answer text can be conducted by inviting experts or using existing quality assessment tools. The specific method is not limited here. The quality score can be based on the accuracy, completeness, professionalism, logic and other dimensions of the answer. The score result can be in numerical form (such as 1-5 points) or in other forms that can reflect the quality.

[0071] It is understandable that through quality scoring, the model can be better guided to learn high-quality answering methods and improve the quality of the model's answers.

[0072] Step S20, inputting the question texts in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure;

[0073] A big language model refers to a deep learning model trained using a large amount of text data. It can generate natural language text or understand the meaning of language text. The big language model learns language patterns, grammar, and semantics by processing large amounts of text data, thereby understanding and generating human language.

[0074] In this embodiment, firstly, a large language model (i.e., a pre-trained large prediction model) that has learned the natural language structure is selected, such as BERT and other models. The pre-trained large prediction model has acquired powerful language understanding and generation capabilities through learning a large amount of text data. The question texts in the screened training question-answer pairs are used as inputs and sequentially input into the pre-trained large language model. The model generates corresponding answer texts (hereinafter referred to as model output text for distinction) based on the input question texts, which serve as a basis for subsequent model optimization.

[0075] Step S30, inputting the quality scores of the texts output by the model into the loss function constructed based on the preference optimization model to obtain a loss value, and updating the model parameters of the large language model based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0076] A preference optimization model is a method or framework used to optimize a system or model to better conform to human preferences. In official document question-answering tasks, the preference optimization model can be used to adjust the output of the model to make it more consistent with human expectations and requirements for the answer text.

[0077] A loss function is constructed based on a preference optimization model (such as the policy gradient method in reinforcement learning). The loss function compares the quality score of the answer text output by the model with the quality score of the actual answer text, and calculates a loss value. The size of the loss value reflects the degree of difference between the model output and the actual situation. The back propagation algorithm is used to pass the loss value back to each layer of the large language model. The model parameters of the model are updated according to the size and direction of the loss value, and the iteration is performed multiple times until the loss value reaches a smaller threshold or no longer decreases significantly. After multiple iterative updates, the final official document question and answer model is obtained.

[0078] Exemplarily, the process of constructing a loss function based on a preference optimization model may be: calculating the difference between the quality score of the model output text and the quality score of the sample answer in the training sample corresponding to the model output text, specifically, calculating the difference, ratio or distance metric between the two, etc., which is not limited here; constructing a loss function based on the quality score difference. Common types of loss functions include: mean square error, cross entropy loss, etc. It should be noted that since quality assessment may involve multiple dimensions, it may be necessary to construct a multi-objective loss function to simultaneously optimize multiple quality features.

[0079] It can be understood that by using official document question and answer to train the large language model and introducing a loss function based on the preference optimization model to optimize the model parameters of the large language model during the training process, the obtained official document question and answer model gradually approaches human preferences and intentions. The preference alignment mechanism ensures that the official document question and answer model can better reflect human values, intentions and preferences when generating official document answers, thereby further improving the quality of official document question and answering using a large language model.

[0080] In addition, preference optimization enables the model to learn the key features of the official document question and answer task more quickly during the training process, thereby reducing unnecessary calculations and resource consumption. The official document question and answer model can utilize resources more efficiently when processing similar tasks, thereby improving the processing speed of official document question and answer.

[0081] In addition, compared to the existing training process of aligning large language models with human preferences, which relies on a reward model to provide feedback signals to guide the model to produce outputs that conform to human preferences, in this embodiment, the answer text in the official document question and answer pairs used for model training has directly added quality scores, which can be directly used to construct a loss function and update model parameters. This application does not need to first train a reward model to evaluate the quality of the text, thereby reducing the training steps and complexity, reducing the dependence on additional computing resources, and thus reducing the consumption of computing resources for model training.

[0082] Furthermore, compared to the existing reinforcement learning fine-tuning during training, which may cause the model to produce unexpected behaviors while pursuing high rewards, the loss function in this embodiment is constructed based on quality scores. These scores directly reflect human expectations for the quality of answer texts. Therefore, during the training process, the model will strive to generate answer texts that meet these expectations, and will not produce unexpected behaviors like reinforcement learning fine-tuning, which helps to improve the stability and reliability of the model.

[0083] In a feasible implementation manner, before the step S10: inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores, the step further includes:

[0084] Step S01, inputting each group of the training question-answer pairs into a preset preference screening model to obtain high-quality samples, wherein the quality score of the high-quality samples is greater than a preset score threshold;

[0085] A preference screening model is a machine learning model that can screen out samples that meet specific preferences or standards from a given set of samples. In official document question-answering systems or other text processing tasks, preference screening models can evaluate the quality of samples based on preset quality standards (such as clarity, accuracy, relevance, etc.) and screen out high-quality samples for subsequent use.

[0086] Determine the preference screening model and train it based on a large amount of annotated data. The annotated data may include an evaluation of the quality of the question-answer pairs, such as scores for clarity, accuracy, relevance and other indicators. The specific training process is not described here. Preprocess each group of training question-answer pairs, such as removing irrelevant information, word segmentation, and stop words, to ensure the consistency and accuracy of the data. The data may also be formatted to meet the input requirements of the preference screening model. Model evaluation and screening: Input the preprocessed training question-answer pairs into the preference screening model. The model will perform a quality assessment on each question-answer pair according to the preset quality standards and output a quality score. According to the preset score threshold (such as 80 points, 90 points, etc.), screen out question-answer pairs with a quality score higher than the threshold as high-quality samples.

[0087] It can be understood that by screening out high-quality samples from the original training question-answer pairs to improve the effect and performance of subsequent model training, by introducing a preference screening model, samples that meet the requirements can be automatically identified and retained, reducing the interference of noise data, thereby improving the quality of the model.

[0088] Step S02, using the high-quality samples and the training question-answer pairs as a training data set;

[0089] The high-quality samples are merged with the original training question-answer pairs to form a new training dataset. The new training dataset contains both high-quality samples and retains the diversity of the original dataset, which helps to improve the generalization ability of the model.

[0090] It can be understood that by constructing a comprehensive training dataset containing high-quality samples and original data, the advantages of high-quality samples can be fully utilized while maintaining the diversity of original data, providing richer and more comprehensive data support for subsequent model training.

[0091] The step S10: inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores, comprises:

[0092] Step S101, inputting the question text in the training data set into a pre-trained large language model to obtain a model output text carrying a quality score.

[0093] The question text is extracted from the training data set and necessary preprocessing is performed (such as word segmentation, removal of stop words, etc.); the processed question text is input into the pre-trained large language model; the large language model generates the answer text based on the input question text and outputs a quality score at the same time. The quality score is used to measure the quality of the answer text.

[0094] It is understandable that by screening out high-quality samples through the preference screening model, the interference of noise data can be effectively reduced and the quality of the training data set can be improved, thereby improving the performance of the large language model and enabling it to more accurately understand and generate text content in official document questions and answers.

[0095] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 Before the step S10 of adding quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, the method further includes:

[0096] Step A01, inputting the answer texts in each group of the official document question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the official document question-answer pairs;

[0097] It should be noted that the annotation model can perform quality assessment on the answer text in the official document question and answer and output an initial score. Specifically, it can be a classifier or regressor based on machine learning, or a neural network model based on deep learning, and there is no limitation here.

[0098] The answer texts in each group of official document question-and-answer pairs are preprocessed, including removing irrelevant information, word segmentation, and removing stop words, to ensure the consistency and accuracy of the data. The preprocessed answer texts are then input into the annotation model. The model calculates the quality score of each answer text based on internal algorithms and parameters, and outputs these scores as initial scores.

[0099] The preset annotation model is used to conduct a preliminary quality assessment of the answer text in the official document question and answer session, which provides basic data for subsequent quality score adjustments and ensures the objectivity and consistency of the evaluation process.

[0100] Step A02, for any target official document question and answer pair in each group of the official document question and answer pairs, obtain adjustment information based on the target initial score, and adjust the target initial score according to the adjustment information, and use the adjusted target initial score as the quality score of the target official document question and answer pair, wherein the target initial score is the initial score of the answer text in the target official document question and answer pair.

[0101] For each target document question-and-answer pair (i.e., the document question-and-answer pair whose score needs to be adjusted), obtain adjustment information based on its initial score. The adjustment information can come from multiple aspects, such as expert evaluation, user feedback, historical data, etc. The obtained adjustment information is integrated and analyzed to determine the specific adjustment method and magnitude of the target initial score. The target initial score is adjusted based on the integrated adjustment information. The adjustment process can be linear (such as directly adding or subtracting a fixed value) or nonlinear (such as applying a function for transformation). The adjusted score is used as the quality score of the target document question-and-answer pair and saved for subsequent use.

[0102] The initial scores are further adjusted and optimized to more accurately reflect the quality of the official document question-answer pairs, and the output of the annotation model is corrected by introducing external information (such as expert evaluation and user feedback), thereby improving the accuracy and reliability of the quality scores.

[0103] It is understandable that by making full use of the preset annotation model and external adjustment information, a more accurate quality assessment of the answer text in the official document question and answer system can be performed, further ensuring the objectivity and fairness of the evaluation process. Moreover, as an objective tool, the annotation model can reduce the impact of human factors on the evaluation results, and provide a reliable basis for the subsequent processing of the official document question and answer system (such as sorting, recommendation, etc.), thereby improving the overall performance of the system and user experience.

[0104] In one feasible implementation, the annotation model includes a first input layer, an embedding layer, an encoding layer, an attention mechanism block, and a classification layer;

[0105] The step A01: inputting the answer texts in each group of the training question-answer pairs into a preset annotation model to obtain the initial scores of the answer texts in each group of the training question-answer pairs, comprises:

[0106] Step A011, inputting the answer text in each group of the training question-answer pairs into the input layer to obtain a word index sequence vector;

[0107] The annotation model includes the first input layer, embedding layer, encoding layer, attention mechanism block and classification layer. The first input layer is the first layer of the annotation model, which is responsible for receiving the original input data. In text processing tasks, the input layer usually receives a text sequence and converts it into a numerical form that the model can process, such as a word index sequence. The embedding layer is used to convert the output of the input layer (such as a word index) into a dense, low-dimensional word vector. These word vectors can capture the semantic relationship between words and are the basis for subsequent encoding layer processing. The encoding layer is responsible for further processing the word vector output by the embedding layer and extracting high-level features of the text, which may include semantic information and syntactic structure of the text. The encoding layer is usually composed of multiple neural network layers, such as convolutional layers, recurrent layers or Transformer layers. The attention mechanism is a technology used to enhance the performance of neural network models, allowing the model to dynamically focus on important parts when processing input data. In text processing tasks, the attention mechanism block can calculate the weight of each word or sentence in the input text and generate weighted feature vectors based on these weights, thereby paying more attention to important information. Classification layer: The classification layer is the last layer of the neural network model, which is responsible for converting the feature vector output by the encoding layer or the attention mechanism block into the final classification result or score. In the annotation model, the classification layer usually uses a fully connected layer (also called a dense layer) and a softmax function to achieve quality scoring.

[0108] First, the answer texts in the training question-answer pairs are preprocessed, such as word segmentation and stop word removal. Vocabulary construction: A vocabulary is constructed based on the preprocessed text, and a unique index is assigned to each word. Each answer text is then converted into a corresponding word index sequence, that is, a sequence consisting of the index of each word in the vocabulary. The generated word index sequence is input into the first input layer of the annotation model to obtain a word index sequence vector.

[0109] Step A012, inputting the word index sequence into the embedding layer to obtain a word vector sequence;

[0110] The output of the input layer (that is, the word index sequence) is input into the embedding layer, and the corresponding word vector sequence is obtained by looking up the embedding matrix (or word embedding table), thereby converting the word index sequence into a dense, low-dimensional word vector sequence, providing a basis for subsequent encoding layer processing.

[0111] Step A013, inputting the word vector sequence into the encoding layer to obtain an encoding feature vector;

[0112] The word vector sequence output by the embedding layer is input into the encoding layer. The encoding layer extracts high-level features of the text, such as semantic information, syntactic structure, etc., by processing the word vector sequence. Then the encoding layer represents the extracted features as encoded feature vectors as the input of the subsequent attention mechanism block or classification layer.

[0113] Step A014, inputting the encoded feature vector into the attention mechanism block to obtain a weighted feature vector;

[0114] The attention mechanism block first calculates the attention weight of each element in the encoded feature vector, which can be achieved by calculating the correlation or similarity between elements, which is not limited here; weighted summation: according to the calculated attention weight, the elements in the encoded feature vector are weighted summed to obtain a weighted feature vector. The attention mechanism is used to enhance the model's ability to focus on important information, so that the model can pay more attention to the important parts of the answer text, thereby improving the accuracy of annotation.

[0115] Step A015, inputting the weighted feature vector into the classification layer to obtain an initial score for the answer text.

[0116] The weighted feature vector output by the attention mechanism block is input into the classification layer. The classification layer processes the weighted feature vector through a fully connected layer and a softmax (normalization) function to calculate the initial score of the answer text.

[0117] It is understandable that by introducing deep learning technologies such as embedding layer, encoding layer and attention mechanism block, the annotation model can more accurately capture the semantic information and important features in the answer text, thereby improving the accuracy of annotation. In addition, the attention mechanism block can calculate the importance weight of each word or sentence in the answer text, which helps to understand the decision-making process of the model and provide explainability.

[0118] In a feasible implementation, the step S10: determining a plurality of sets of official document question-answer pairs based on the corpus, comprises:

[0119] Step S101, determining a plurality of groups of corpus question-answer pairs from the corpus question-answer pairs, wherein the corpus question-answer pairs are question-answer pairs for an official document question-answering task;

[0120] Screening out question-answer pairs related to the official document question-answering task from the existing corpus may include removing noise data, screening out question-answer pairs related to official documents, and other operations, which are not limited here; in the preprocessed corpus, select multiple groups of question-answer pairs as initial corpus question-answer pairs, which serve as the basis for subsequent steps.

[0121] Step S102, inputting each group of the corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs;

[0122] Generative Adversarial Network (GAN) is a deep learning model consisting of two parts: Generator and Discriminator. The core idea of ​​GAN is to generate high-quality data samples by letting these two networks compete and confront each other. The goal of the generator is to generate realistic data samples from random noise (usually Gaussian noise or uniform noise). During the training process, the generator continuously tries to generate data samples that are closer and closer to the real data distribution. The task of the discriminator is to distinguish whether the input data sample comes from the real data set or is generated by the generator. It is a binary classifier that outputs a probability value, indicating the probability that the input data sample is real data. The training of GAN is an iterative process. In each iteration, the discriminator is first fixed, and the generator is trained to generate data samples that can deceive the discriminator. Then the generator is fixed and the discriminator is trained to better distinguish between real data and generated data. This process is repeated until the generator can generate sufficiently realistic data samples, making it difficult for the discriminator to distinguish them. The specific training process is not described here.

[0123] In this embodiment, the generative adversarial network is used to generate multiple sets of extended question-answer pairs. Specifically, the selected corpus question-answer pairs are input into the GAN model as real data, and the generator is trained to generate new question-answer pairs that are similar to but different from the real question-answer pairs. These newly generated question-answer pairs can increase the number and diversity of official document question-answer pairs, thereby helping to improve the performance of subsequent models on official document question-answering tasks.

[0124] Step S103: Using the corpus question-answer pair and the extended question-answer pair as the official document question-answer pair.

[0125] The screened corpus question-answer pairs and the generated extended question-answer pairs are merged to form a new official document question-answer pair dataset, which contains both the question-answer pairs in the original corpus and the extended question-answer pairs generated by the GAN model, thereby improving the diversity and richness of the dataset, and enabling the model to learn more features and patterns, thereby improving its accuracy and robustness in the official document question-answering task. It is understandable that the merged dataset can be subjected to necessary preprocessing and formatting operations to ensure that it meets the requirements of subsequent steps.

[0126] In one feasible implementation, the generative adversarial network includes a second input layer, a generator, and a discriminator;

[0127] The step S102: inputting each group of the corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs, comprises:

[0128] Step S1021, inputting each group of the corpus question-answer pairs into the second input layer to obtain a random noise vector;

[0129] The second input layer in the generative adversarial network refers to the part that receives the raw data input. The second input layer receives each group of corpus question-answer pairs. Its function is to convert these question-answer pairs into a format that the network can process, usually by encoding them into a series of numerical features, and then further converting them into a random noise vector. This random noise vector will be used as the input of the generator to generate new question-answer pairs. The generator can be a deep learning model, such as a recurrent neural network, a long short-term memory network, or a variational autoencoder. The generator receives a random noise vector as input and outputs a new question-answer pair, which should be as close as possible to the real-world question-answer pair to deceive the discriminator.

[0130] The corpus question-answer pair is input into the second input layer and converted into a random noise vector through a series of encoding operations (such as fully connected layers, activation functions, etc.), which will serve as the input of the generator.

[0131] Step S1022, inputting the random noise vector into the generator to generate a question-answer pair;

[0132] The random noise vector is input into the generator, which converts it into a new question-answer pair through a series of decoding operations (such as deconvolution layers, upsampling, etc., which may be the decoding process of RNN for text generation) and outputs the generated question-answer pair: the generator outputs the generated question-answer pair, which should be as close to the real-world question-answer pair as possible.

[0133] Step S1023, input the generated question-answer pair into the discriminator to obtain a discrimination result. When the discrimination result is true, the generated question-answer pair is output through the generator, and the question-answer pair is used as an extended question-answer pair.

[0134] The generated question-answer pair is input into the discriminator, which outputs a probability value through a series of classification operations (such as convolutional layer, fully connected layer, etc.), indicating the probability that this question-answer pair is real data. The discriminant result is judged as follows: If the probability value output by the discriminator is greater than a certain threshold (for example, set to 0.5 or higher), the generated question-answer pair is considered to be real, that is, the discriminant result is true, and the extended question-answer pair is output: When the discriminant result is true, the generated question-answer pair is output through the generator and added to the dataset as an extended question-answer pair.

[0135] It is understandable that the generative adversarial network can generate more and more realistic question-answer pairs through the mutual confrontation and training of the generator and the discriminator, thereby improving the quality of the generated question-answer pairs. By generating extended question-answer pairs through the generative adversarial network, the number and diversity of official document question-answer pairs can be significantly increased, which helps to improve the generalization ability of the model in official document question-answering tasks. In addition, by adjusting the parameters and training strategies of the generative adversarial network, official document question-answer pairs that meet specific needs or styles can be generated to meet different application scenarios and needs.

[0136] For example, to help understand the implementation process of the dialog generation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flow chart of a method for generating a dialogue is provided. This embodiment adopts a pre-trained large-scale language model as the basic architecture, and specifically selects the QWEN model. Specifically:

[0137] 1. Collection and construction of data sets (i.e., determining multiple sets of official document question-answer pairs based on the corpus, and adding quality scores to the answer texts in each set of official document question-answer pairs to obtain multiple sets of training question-answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts). The specific process is as follows:

[0138] Collection of preference data: Using a professional manual annotation team and advanced large-scale models, we collected high-quality preference data for the specific scenario of official document question and answer. In each set of question and answer data, the annotation team will select the best and worst samples from multiple possible answers. In the process of sample annotation, a semi-automatic annotation system can be introduced based on the existing corpus, combining the preliminary annotation based on the large-scale language model with the review of human experts. By preliminarily screening the answers generated by the model, human annotators only need to adjust or confirm a small amount of key data, thereby achieving a more efficient and consistent data annotation process.

[0139] Data preprocessing: The collected data was strictly cleaned and formatted to ensure the quality and consistency of the data set. The carefully processed data was divided into a training set and a validation set to facilitate subsequent model training and evaluation. For example, there is a data set for a document question-and-answer task, which contains several questions and their corresponding multiple answers. Experts select the best answer and the worst answer for each question. The specific steps are as follows: Question 1: How to improve work efficiency? Answer A: Arrange time reasonably and make a detailed work plan. Marked as excellent. Answer B: Drink more coffee and stay up late to work overtime. Marked as poor. Question 2: How to write a good document? Answer A: Follow the format of official document writing and use concise and clear language. Marked as excellent. Answer B: Use gorgeous words and try to write more complex. Marked as poor.

[0140] Preference data input model: Question 1 and its good and bad answer pairs (answer A and answer B) are input into the model, preference loss is calculated and model parameters are updated. Question 2 and its good and bad answer pairs (answer A and answer B) are input into the model, preference loss is calculated and model parameters are updated.

[0141] In this embodiment, the existing corpus can be used to introduce generative adversarial networks (GANs) or data enhancement technology to generate more and more diverse question-answer pairs. After manual screening, these extended data can enrich the content of the training set and enhance the model's ability to answer official document questions in different scenarios.

[0142] In this embodiment, preference data and corpus can also be jointly iterated for training, and the official document corpus can be combined with preference data for joint training. A large number of question-answer pairs in the corpus are used to automatically generate candidate answers, and the optimal samples are screened out based on the pre-trained preference screening model and added to the preference data set. In this way, the size of the preference data set can be greatly expanded while ensuring data quality, so that the DPO model can be optimized on a larger scale.

[0143] 2. DPO fine-tuning (i.e., inputting the question texts in each group of training question-answer pairs into a pre-trained large language model to obtain model output texts with quality scores, wherein the pre-trained large language model is a large language model that has learned the natural language structure; inputting the quality scores of the model output texts into the loss function constructed based on the preference optimization model to obtain loss values, and updating the model parameters of the large language model based on the loss values ​​to obtain an official document question-answering model, wherein the official document question-answering task is processed according to the official document question-answering model). The specific process is as follows:

[0144] Initialize model: Load a pre-trained large language model.

[0145] Define the preference loss function: According to the DPO formula, define the preference loss function to optimize the parameters of the model. In this embodiment, the DPO (Direct Preference Optimization) method is used to directly optimize the preference without going through the reward modeling step. By directly optimizing human preference data, the performance of LLM in official document question and answer is improved. The analytical mapping from the reward function to the optimal strategy is used, allowing the direct use of human preference data for a simplified optimization process. In this embodiment, the loss function based on the DPO algorithm is:

[0146]

[0147] Among them, L DPO where π denotes the objective function used to optimize the model parameters θ; θ is the large language model to be trained; π refis the reference model used to constrain π θ Generated text; D is the training data set, which contains the input x and the corresponding high-quality scoring text (ie, reasonable text) y w and low quality scoring text (i.e. unreasonable text) y l ; σ is the sigmoid function, which maps the input value to [0,1]; β is a hyperparameter used to control the weight of unreasonable text relative to reasonable text in the loss function; Used to ensure that the generated text is consistent with the reference model. It can be understood that DPO reparameterization is equivalent to having an implicit reward function: Parameter model π θ The optimization of is equivalent to the optimization of the reward model under this variable change.

[0148] Model training: Use preference data to fine-tune the model. During the training process, the model adjusts parameters based on the preference loss function so that its output is consistent with human preferences. The specific steps are as follows:

[0149] Take a pair of good and bad samples (and) from the training set.

[0150] Calculate the score (sum) for each sample.

[0151] Calculate the loss value according to the preferred loss function, backpropagate the loss value and update the model parameters.

[0152] 3. Model verification: Evaluate the performance of the model on the validation set to ensure that the model is not overfitted during the fine-tuning process. Verify the effectiveness of the DPO method by comparing the performance indicators before and after fine-tuning.

[0153] 4. Effect evaluation. Design comparative experiments, and evaluate the performance of the model before and after fine-tuning through official document question-answering tasks. Specific evaluation indicators include: preference consistency, whether the answers generated by the model are in line with human preferences; generation quality, the accuracy, coherence and relevance of the answers; processing efficiency, the response time and resource consumption of the model when processing complex question-answering tasks. Compare the effects before and after fine-tuning, and evaluate the performance of the DPO method in improving the LLM official document question-answering capabilities, which may include: preference consistency improvement, the fine-tuned model can more accurately capture and reflect human preferences when answering official document question-answering tasks; generation quality improvement, the fine-tuned model has higher quality when generating answers, and the coherence and relevance are significantly improved; efficiency improvement, the fine-tuned model has a shorter response time and reduced computing resource consumption when processing complex question-answering tasks.

[0154] In this embodiment, the DPO (Direct Preference Optimization) method is used to directly optimize preferences without going through the reward modeling step. By directly optimizing human preference data, the performance of LLM in official document question answering is improved.

[0155] In this embodiment, compared with LLM without human preference alignment, after using DPO to improve LLM's official document question-and-answer capabilities, LLM's output is more in line with human preferences, improving user satisfaction; LLM is more controllable when generating answers and can better follow user instructions and preferences; LLM shows better ability to control the emotions of generated results and improve the response quality of summaries and single-round conversations. In addition, the computing resources required for model training and reasoning are reduced, and the efficiency of processing question-and-answer tasks is improved. Compared with RLHF using PPO, using DPO to improve LLM's official document question-and-answer capabilities uses a simpler and more direct training method that does not require explicit reward modeling steps, nor does it require sampling from the model during fine-tuning, saving a lot of resources.

[0156] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the dialogue generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0157] This application also provides a dialogue generation device, please refer to Figure 4 The dialogue generation device is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes:

[0158] A determination module 10 is used to determine multiple groups of official document question-answer pairs based on the corpus, and add a quality score to the answer text in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality score is used to characterize the quality of the content of the answer text;

[0159] An input module 20, for inputting the question texts in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure;

[0160] The updating module 30 is used to input the quality scores of the texts output by the model into the loss function constructed based on the preference optimization model to obtain a loss value, and update the model parameters of the large language model based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0161] Optionally, the dialogue generating device further comprises a scoring module, which is used to:

[0162] Inputting the answer texts in each group of the official document question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the official document question-answer pairs;

[0163] For any target official document question and answer pair in each group of the official document question and answer pairs, adjustment information based on the target initial score is obtained, and the target initial score is adjusted according to the adjustment information, and the adjusted target initial score is used as the quality score of the target official document question and answer pair, wherein the target initial score is the initial score of the answer text in the target official document question and answer pair.

[0164] Optionally, the annotation model includes a first input layer, an embedding layer, an encoding layer, an attention mechanism block and a classification layer; and the scoring module is further used for:

[0165] Inputting the answer text in each group of the training question-answer pairs into the input layer to obtain a word index sequence vector;

[0166] Input the word index sequence into the embedding layer to obtain a word vector sequence;

[0167] Inputting the word vector sequence into the encoding layer to obtain an encoding feature vector;

[0168] Inputting the encoded feature vector into the attention mechanism block to obtain a weighted feature vector;

[0169] The weighted feature vector is input into the classification layer to obtain an initial score for the answer text.

[0170] Optionally, the determining module 10 is further configured to:

[0171] Determining a plurality of groups of corpus question-answer pairs from the question-answer pairs in the corpus, wherein the corpus question-answer pairs are question-answer pairs for an official document question-answering task;

[0172] Inputting each group of corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs;

[0173] The corpus question-answer pair and the extended question-answer pair are used as the official document question-answer pair.

[0174] Optionally, the generative adversarial network includes a second input layer, a generator and a discriminator; the input module 20 is further used for:

[0175] Input each group of the corpus question-answer pairs into the second input layer to obtain a random noise vector;

[0176] Inputting the random noise vector into the generator to generate a question-answer pair;

[0177] The generated question-answer pair is input into the discriminator to obtain a discrimination result. When the discrimination result is true, the generated question-answer pair is output through the generator, and the question-answer pair is used as an extended question-answer pair.

[0178] Optionally, the dialogue generation device further includes a training module, which is used to:

[0179] Inputting each group of the training question-answer pairs into a preset preference screening model to obtain high-quality samples, wherein the quality score of the high-quality samples is greater than a preset score threshold;

[0180] Using the high-quality samples and the training question-answer pairs as a training data set;

[0181] The input module 20 is also used for:

[0182] The question text in the training data set is input into a pre-trained large language model to obtain a model output text carrying a quality score.

[0183] The dialogue generation device provided by the present application adopts the dialogue generation method in the above embodiment, which can solve the technical problem that the large language model does not conform to human preferences when replying to official documents. Compared with the prior art, the beneficial effects of the dialogue generation device provided by the present application are the same as those of the dialogue generation method provided by the above embodiment, and other technical features in the dialogue generation device are the same as those disclosed in the above embodiment method, which will not be repeated here.

[0184] The present application provides a dialog generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the dialog generation method in the above-mentioned embodiment.

[0185] Reference below Figure 5 , which shows a schematic diagram of the structure of a dialog generation device suitable for implementing the embodiment of the present application. The dialog generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The dialog generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0186] like Figure 5 As shown, the dialog generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the dialog generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the dialog generation device to communicate with other devices wirelessly or by wire to exchange data. Although the dialog generation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0187] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0188] The dialogue generation device provided by the present application adopts the dialogue generation method in the above embodiment, which can solve the technical problem that the large language model does not conform to human preferences when replying to official documents. Compared with the prior art, the beneficial effects of the dialogue generation device provided by the present application are the same as the beneficial effects of the dialogue generation method provided by the above embodiment, and the other technical features in the dialogue generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0189] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more groups of embodiments or examples in a suitable manner.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0191] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the dialog generation method in the above-mentioned embodiment.

[0192] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more sets of wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0193] The computer-readable storage medium may be included in the dialog generation device; or may exist independently without being assembled into the dialog generation device.

[0194] The computer-readable storage medium carries one or more groups of programs. When the one or more groups of programs are executed by the dialogue generation device, the dialogue generation device: determines multiple groups of official document question and answer pairs based on the corpus, and adds quality scores to the answer texts in each group of the official document question and answer pairs to obtain multiple groups of training question and answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts; inputs the question texts in each group of the training question and answer pairs into a pre-trained large language model to obtain model output texts carrying quality scores, wherein the pre-trained large language model is a large language model that has learned the natural language structure; inputs the quality scores of the model output texts into a loss function constructed based on a preference optimization model to obtain a loss value, and updates the model parameters of the large language model based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

[0195] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more groups of executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0198] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned dialogue generation method, and can solve the technical problem that the large language model does not conform to human preferences when replying to official documents. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the dialogue generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0199] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned dialogue generation method when executed by a processor.

[0200] The computer program product provided by this application can solve the technical problem that the large language model does not conform to human preferences when replying to official documents. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the dialogue generation method provided by the above embodiment, which will not be repeated here.

[0201] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a dialogue, characterized in that: The dialogue generation method is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes: Determine multiple groups of official document question-answer pairs based on the corpus, and add quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality scores are used to characterize the quality of the content of the answer texts; Inputting the question text in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure; The quality scores of the texts output by the model are input into a loss function constructed based on a preference optimization model to obtain a loss value, and the model parameters of the large language model are updated based on the loss value to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

2. The method for generating a dialogue according to claim 1, wherein: Before the step of adding quality scores to the answer texts in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, the method further includes: Inputting the answer texts in each group of the official document question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the official document question-answer pairs; For any target official document question and answer pair in each group of the official document question and answer pairs, adjustment information based on the target initial score is obtained, and the target initial score is adjusted according to the adjustment information, and the adjusted target initial score is used as the quality score of the target official document question and answer pair, wherein the target initial score is the initial score of the answer text in the target official document question and answer pair.

3. The method for generating a dialogue according to claim 2, wherein: The annotation model includes a first input layer, an embedding layer, an encoding layer, an attention mechanism block and a classification layer; The step of inputting the answer texts in each group of the training question-answer pairs into a preset annotation model to obtain an initial score for the answer texts in each group of the training question-answer pairs comprises: Inputting the answer text in each group of the training question-answer pairs into the input layer to obtain a word index sequence vector; Input the word index sequence into the embedding layer to obtain a word vector sequence; Inputting the word vector sequence into the encoding layer to obtain an encoding feature vector; Inputting the encoded feature vector into the attention mechanism block to obtain a weighted feature vector; The weighted feature vector is input into the classification layer to obtain an initial score for the answer text.

4. The method for generating a dialogue according to claim 1, wherein: The step of determining a plurality of sets of official document question-answer pairs based on the corpus comprises: Determining a plurality of groups of corpus question-answer pairs from the question-answer pairs in the corpus, wherein the corpus question-answer pairs are question-answer pairs for an official document question-answering task; Inputting each group of corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs; The corpus question-answer pair and the extended question-answer pair are used as the official document question-answer pair.

5. The method for generating a dialogue according to claim 4, wherein: The generative adversarial network includes a second input layer, a generator and a discriminator; The step of inputting each group of corpus question-answer pairs into a preset generative adversarial network to obtain multiple groups of extended question-answer pairs includes: Input each group of the corpus question-answer pairs into the second input layer to obtain a random noise vector; Inputting the random noise vector into the generator to generate a question-answer pair; The generated question-answer pair is input into the discriminator to obtain a discrimination result. When the discrimination result is true, the generated question-answer pair is output through the generator, and the question-answer pair is used as an extended question-answer pair.

6. The method for generating a dialogue according to any one of claims 1 to 5, characterized in that: Before the step of inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores, the step further includes: Inputting each group of the training question-answer pairs into a preset preference screening model to obtain high-quality samples, wherein the quality score of the high-quality samples is greater than a preset score threshold; Using the high-quality samples and the training question-answer pairs as a training data set; The step of inputting the question texts in each group of the training question-answer pairs into the pre-trained large language model to obtain the model output texts carrying the quality scores comprises: The question text in the training data set is input into a pre-trained large language model to obtain a model output text carrying a quality score.

7. A dialogue generation device, characterized in that: The dialogue generation device is applied to a dialogue generation device, the dialogue generation device is connected to a corpus in communication, and the dialogue generation includes: A determination module, used to determine multiple groups of official document question-answer pairs based on the corpus, and add a quality score to the answer text in each group of the official document question-answer pairs to obtain multiple groups of training question-answer pairs, wherein the quality score is used to characterize the quality of the content of the answer text; An input module, used for inputting the question text in each group of the training question-answer pairs into a pre-trained large language model to obtain a model output text carrying a quality score, wherein the pre-trained large language model is a large language model that has learned a natural language structure; An updating module is used to input the quality scores of the texts output by the model into a loss function constructed based on a preference optimization model to obtain a loss value, and based on the loss value, update the model parameters of the large language model to obtain an official document question and answer model, wherein the official document question and answer task is processed according to the official document question and answer model.

8. A dialogue generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dialog generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dialogue generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the dialog generation method according to any one of claims 1 to 6 are implemented.

Images