Dialogue generation method and device based on improved DPO training and readable medium

Abstract

The invention discloses a dialogue generation method and device based on improved DPO training and a readable medium, and the method comprises the steps: carrying out the supervised fine tuning of a large language model, and obtaining a fine-tuned large language model; taking the fine-tuned large language model as a reference model; constructing a preference dialogue data set; and freezing parameters of the reference model, taking the fine-tuned large language model as a to-be-trained strategy model, and performing improved DPO training on the to-be-trained strategy model based on the preference dialogue data set and the reference model to adjust parameters of the to-be-trained strategy model so as to obtain a trained large language model. A loss function used in the process of improving DPO training is mixed loss, and the mixed loss comprises preference loss, fine tuning loss and anchor point scoring loss; and obtaining a to-be-replied historical dialogue and inputting the to-be-replied historical dialogue into the trained large language model to obtain a corresponding reply statement. According to the method, more stable training can be realized, and over-fitting is avoided.

Description

Technical Field

[0001] This invention relates to the field of dialogue generation, and more specifically to a dialogue generation method, apparatus, and readable medium based on improved DPO training. Background Technology

[0002] Large language models are a type of self-supervised generative pre-trained language model (PLM). Their core idea is to learn general language representations from massive amounts of text through self-supervised learning, and then fine-tune them to adapt them for downstream tasks. Existing large language models such as Deepseek and Qwen are autoregressive generative models based on unidirectional Transformers. This is because autoregressive generative models maximize the likelihood probability of sequences and model them using conditional probability. Predicting the next token (character) allows for a step-by-step generation effect, making it highly suitable for query-answer dialogue modes.

[0003] Another key point is how to better match the output of a large language model with user needs. This requires preference learning, which is to learn preferences from chosen-rejected sample pairs through reinforcement learning. Direct Preference Optimization (DPO) is a method that avoids specifically training a reward model, directly optimizing human preferences and avoiding the complex reward modeling of reinforcement learning. Current DPO training suffers from the following problems:

[0004] 1. Traditional DPO training methods rely only on pairwise preference data (such as preference selection responses and rejected responses), making it difficult to handle multi-dimensional preferences (such as factuality, security, fluency, etc.).

[0005] 2. A single loss function can easily lead to overfitting of the model to a certain preference dimension, while ignoring other important indicators.

[0006] 3. Existing hybrid training methods typically use simple weighted loss and lack dynamic adjustment mechanisms for preference conflicts. Summary of the Invention

[0007] The purpose of this application is to propose a dialogue generation method, apparatus, and readable medium based on improved DPO training to address the aforementioned technical problems.

[0008] In a first aspect, the present invention provides a dialogue generation method based on improved DPO training, comprising the following steps:

[0009] Supervised fine-tuning of the large language model yields a fine-tuned large language model; the fine-tuned large language model is then used as a reference model.

[0010] Construct a preference dialogue dataset; freeze the parameters of the reference model, use the fine-tuned large language model as the policy model to be trained, and perform improved DPO training on the policy model to be trained based on the preference dialogue dataset and the reference model to adjust the parameters of the policy model to be trained, so as to obtain the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor rating loss.

[0011] The system retrieves the historical dialogues to be replied to and inputs them into a trained large language model to obtain the corresponding reply statements.

[0012] As a preferred approach, in the forward propagation process of improved DPO training, the preference loss adopts DPO loss, the fine-tuning loss adopts cross-entropy loss, and the anchor point rating loss adopts mean squared error loss. In the backpropagation process, the gradient of the loss function with respect to the parameters of the policy model to be trained is calculated through the backpropagation algorithm, and the optimizer updates the parameters of the policy model to be trained based on the gradient.

[0013] As a preferred method, the construction process of the DPO loss is as follows:

[0014] The historical dialogues from the sample of the preference dialogue dataset are respectively input into the reference model and the policy model to be trained. The conditional probabilities of the reference model predicting the preferred response based on the historical dialogues, the conditional probabilities of the reference model predicting the rejected response based on the historical dialogues, and the conditional probabilities of the policy model to be trained based on the parameters are obtained respectively. The following is based on the conditional probability of the preferred response predicted from historical dialogues, and the policy model to be trained in terms of parameters. The conditional probability of a rejected response is predicted based on historical dialogues, and the DPO loss is constructed as shown in the following formula:

[0015] ;

[0016] in, Indicates DPO loss, and These represent the conditional probabilities of the reference model predicting the preferred response based on historical dialogues and the conditional probabilities of the reference model predicting the rejected response based on historical dialogues, respectively. and These represent the policy model to be trained in terms of parameters. The following is a prediction of the conditional probability of the preferred response based on historical dialogues, and the policy model to be trained in terms of parameters. The conditional probability of being rejected is predicted based on historical dialogues.

[0017] As a preferred option, the process for constructing the fine-tuning loss is as follows:

[0018] The policy model to be trained has parameters In the process of predicting preferred responses based on historical dialogues, the conditional probability of each output token is obtained based on all input tokens before each time step, and a fine-tuning loss is constructed, as shown in the following formula:

[0019] ;

[0020] in, This represents the fine-tuning loss, where T represents the length of the text sequence. This represents the output token at time step t in the text sequence. This represents all input tokens up to the t-th time step. This indicates that the policy model to be trained has parameters According to Predicted The conditional probability.

[0021] As a preferred approach, the construction process for the anchor point scoring loss is as follows:

[0022] The preference dialogue dataset also includes labeled evaluation scores of real preference selection responses, which will be used to train the policy model on the parameters. The predicted preference responses from historical dialogues are input into a linear layer to obtain the corresponding predicted scores. These scores are then combined with the actual preference response evaluation scores to construct an anchor rating loss, as shown in the following equation:

[0023] ;

[0024] Where N represents the number of samples in the preference dialogue dataset, The evaluation score represents the actual preference choice response. This indicates that the policy model to be trained has parameters The predicted score is obtained by passing the preference selection response obtained from historical dialogues through a linear layer.

[0025] As a preferred option, the hybrid loss is the sum of the preference loss, fine-tuning loss, and anchor point rating loss.

[0026] Secondly, the present invention provides a dialogue generation apparatus based on improved DPO training, comprising:

[0027] The fine-tuning module is configured to perform supervised fine-tuning on the large language model to obtain a fine-tuned large language model; the fine-tuned large language model is then used as a reference model.

[0028] The preference training module is configured to build a preference dialogue dataset; freeze the parameters of the reference model, use the fine-tuned large language model as the policy model to be trained, and perform improved DPO training on the policy model to be trained based on the preference dialogue dataset and the reference model to adjust the parameters of the policy model to be trained, so as to obtain the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor point rating loss.

[0029] The generation module is configured to retrieve the historical dialogues to be replied to and input them into a trained large language model to obtain the corresponding reply statements.

[0030] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0031] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0032] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The dialogue generation method based on improved DPO training proposed in this invention uses a hybrid loss consisting of preference loss, fine-tuning loss and anchor point scoring loss to improve the DPO training of the fine-tuned large language model. With a small amount of paired preference data, it can make full use of chosen samples to train a better model based on preference data. The convergence of the hybrid loss will be better and more stable in the later stage of training, without generating too much oscillation. It is highly adaptable to the application of large language models and can quickly meet the dialogue service needs of customers in any limited scenario. It can be applied efficiently in e-commerce, law, education, medical care and other fields.

[0035] (2) The dialogue generation method based on improved DPO training proposed in this invention can alleviate pattern collapse. Using DPO loss alone for preference training may make large language models too focused on "distinguishing preferences" and ignore the naturalness and diversity of generation, such as always generating short and safe answers. Using hybrid loss can take into account both preferences and rationality.

[0036] (3) The dialogue generation method based on improved DPO training proposed in this invention uses hybrid loss to achieve more stable training. DPO loss depends on pairwise preference data. If the data noise is large or the coverage is limited, the fine-tuning loss based on SFT can provide a more stable gradient signal and avoid overfitting. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the dialogue generation method based on improved DPO training, which is an embodiment of this application.

[0039] Figure 2 The training result diagram shows the training process of the dialogue generation method based on improved DPO training, which is an embodiment of this application, using hybrid loss.

[0040] Figure 3 A training result graph showing the results of using a single DPO loss during the training process of a large language model;

[0041] Figure 4 This is a schematic diagram of an apparatus for a dialogue generation method based on improved DPO training, as an embodiment of this application.

[0042] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0044] Figure 1 An embodiment of this application illustrates a dialogue generation method based on improved DPO training, comprising the following steps:

[0045] S1. Supervised fine-tuning of the large language model yields the fine-tuned large language model; the fine-tuned large language model is then used as the reference model.

[0046] Specifically, in the embodiments of this application, the large language model uses the Qwen2.5-7B-Instruct model as an example, and supervised fine-tuning (SFT) is performed based on the Qwen2.5-7B-Instruct model. The fine-tuned large language model is denoted as... The reference model (ref_model) used for the next improved DPO training is denoted as... ,Will Equivalent to In the subsequent improved DPO training process, this The parameters are fixed, while The parameters are used as the policy model to be trained to improve the DPO training process. Initial parameters, subsequent The parameters will change during the training of the improved DPO. The role of this reference model is to provide a stable policy baseline to prevent the policy model from "crashing" or severely degrading during training.

[0047] S2, construct a preference dialogue dataset; freeze the parameters of the reference model, use the fine-tuned large language model as the policy model to be trained, and perform improved DPO training on the policy model to be trained based on the preference dialogue dataset and the reference model to adjust the parameters of the policy model to be trained, so as to obtain the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor rating loss.

[0048] In a specific embodiment, during the forward propagation process of improving DPO training, the preference loss adopts DPO loss, the fine-tuning loss adopts cross-entropy loss, and the anchor point rating loss adopts mean squared error loss. During the backpropagation process, the gradient of the loss function with respect to the parameters of the policy model to be trained is calculated through the backpropagation algorithm, and the optimizer updates the parameters of the policy model to be trained based on the gradient.

[0049] Specifically, embodiments of this application improve the loss function for DPO training, thereby enabling the policy model to be trained better and improving its performance and effectiveness in applications. The improved loss function for DPO training employs a hybrid loss, comprising the following components:

[0050] 1) DPO loss The pairwise preference loss of the traditional DPO makes the implicit reward (reward_chosen) for the chosen answer significantly higher than that for the rejected answer (reward_rejected). The specific loss function will be detailed in the training process below.

[0051] 2) Fine-tuning loss : Directly maximize the probability of the model generating the chosen answer (i.e., the negative log-likelihood loss of traditional large language models), ).

[0052] 3) Anchor point scoring loss : Token-wise reward-weighted average of chosen responses.

[0053] In a specific embodiment, the process of constructing the DPO loss is as follows:

[0054] The historical dialogues from the sample of the preference dialogue dataset are respectively input into the reference model and the policy model to be trained. The conditional probabilities of the reference model predicting the preferred response based on the historical dialogues, the conditional probabilities of the reference model predicting the rejected response based on the historical dialogues, and the conditional probabilities of the policy model to be trained based on the parameters are obtained respectively. The following is based on the conditional probability of the preferred response predicted from historical dialogues, and the policy model to be trained in terms of parameters. The conditional probability of a rejected response is predicted based on historical dialogues, and the DPO loss is constructed as shown in the following formula:

[0055] ;

[0056] in, Indicates DPO loss, and These represent the conditional probabilities of the reference model predicting the preferred response based on historical dialogues and the conditional probabilities of the reference model predicting the rejected response based on historical dialogues, respectively. and These represent the policy model to be trained in terms of parameters. The following is a prediction of the conditional probability of the preferred response based on historical dialogues, and the policy model to be trained in terms of parameters. The conditional probability of being rejected is predicted based on historical dialogues.

[0057] Specifically, in the data preparation phase, a preference dialogue dataset (prompt, chosen, rejected) is constructed, denoted as D={(x,y)}. chosen ,y rejected )}, where x is the input history dialogue (which may include prompts), y chosen It is a human preference choice response, y rejected The response was rejected. The data sources are manual annotation and user interaction logs; an example of the sample format is shown below:

[0058] {

[0059] "prompt": [

[0060] {"role": "user", "content": "What should I do if a small tooth is growing inside my teeth?"}

[0061] {"role": "assistant", "content": "Have you had a dental check-up before?"},

[0062] {"role": "user", "content": "I can see my teeth on my gums now, they haven't come in for a long time"}

[0063] ],

[0064] "chosen": ["If a tooth doesn't erupt, it's likely an impacted wisdom tooth."] <sep>How old are you this year?

[0065] "rejected": ["Um, how old are you this year? Which tooth specifically is it?"]

[0066] }

[0067] In improving the DPO training process, triples in each batch of the preference dialogue dataset are... Calculate separately:

[0068] and ; and ;

[0069] In the embodiments of this application, there is no explicit reward model. The "reward" is defined by the logarithmic ratio of the probabilities of the current policy model and the reference model for a certain response. If the current policy model favors a certain answer more than the reference model, the corresponding "reward" is higher.

[0070] );

[0071] );

[0072] The two formulas above represent the "rewards" of the chosen and rejected samples for the current input historical dialogue (x), respectively. It is a scaling factor, a parameter that can be set during training to prevent training from crashing.

[0073] Next, by calculating the difference between the two "rewards," we can measure the difference between the preferred response and the rejected response. Then, by adding a sigmoid function and a negative logarithm to minimize the formula, we construct the DPO loss as follows:

[0074] .

[0075] In a specific embodiment, the process of constructing the fine-tuning loss is as follows:

[0076] The policy model to be trained has parameters In the process of predicting preferred responses based on historical dialogues, the conditional probability of each output token is obtained based on all input tokens before each time step, and a fine-tuning loss is constructed, as shown in the following formula: ;

[0077] in, This represents the fine-tuning loss, where T represents the length of the text sequence. This represents the output token at time step t in the text sequence. This represents all input tokens up to the t-th time step. This indicates that the policy model to be trained has parameters According to Predicted The conditional probability.

[0078] In a specific embodiment, the process of constructing the anchor point scoring loss is as follows:

[0079] The preference dialogue dataset also includes labeled evaluation scores of real preference selection responses, which will be used to train the policy model on the parameters. The predicted preference responses from historical dialogues are input into a linear layer to obtain the corresponding predicted scores. These scores are then combined with the actual preference response evaluation scores to construct an anchor rating loss, as shown in the following equation:

[0080] ;

[0081] Where N represents the number of samples in the preference dialogue dataset, The evaluation score represents the actual preference choice response. This indicates that the policy model to be trained has parameters The predicted score is obtained by passing the preference selection response obtained from historical dialogues through a linear layer.

[0082] In a specific implementation, the hybrid loss is the sum of the preference loss, fine-tuning loss, and anchor rating loss.

[0083] Specifically, the core optimization of the embodiments of this application lies in the innovative addition of fine-tuning loss and anchor point rating loss to the conventional DPO loss for preference learning. This allows the model to learn the expressions of chosen samples more smoothly and focus more on fitting preference data, without being too aggressive and causing a decrease or degradation in generalization ability.

[0084] Furthermore, the fine-tuning loss uses cross-entropy loss, and the anchor point rating loss uses mean squared error loss. This anchor point rating loss adds a multi-task training logic to the entire preference training task, making the preference learning of chosen samples more compact. In the preference loss, the policy model to be trained is adjusted according to the parameters... The conditional probability of choosing a response based on historical dialogues is then predicted. The policy model to be trained has parameters Generate preference selection response below The product of the conditional probabilities of each output token, the policy model to be trained in the parameters The conditional probability of being rejected is predicted based on historical dialogues. The policy model to be trained has parameters The next generation was rejected. The product of the conditional probabilities of each output token; the reference model predicts the conditional probabilities of preferred responses based on historical dialogues. It is a reference model that generates preference selection responses. The reference model calculates the conditional probability of a rejected response by multiplying the conditional probabilities of each output token, based on historical dialogues. The reference model generated a rejected response. It is the product of the conditional probabilities of each output token. The fine-tuning loss uses the parameters of the policy model being trained. Generate preference selection response below The conditional probability of each output token. In the anchor point rating loss, an additional linear layer is used to adjust the parameters of the policy model to be trained. The conditional probability of choosing a response based on historical dialogues is then predicted. The data is compressed to a batch size of 1 to obtain a one-dimensional numerical value as the predicted score. This value is then combined with the evaluation scores of the labeled actual preference selection responses to construct the anchor score loss.

[0085] Therefore, the mixed loss It is the sum of the three losses: DPO loss, fine-tuning loss, and anchor point scoring loss, as shown in the following formula:

[0086] ;

[0087] During backpropagation, the parameters of the loss function with respect to the policy model are calculated using the backpropagation algorithm. The gradient is used to update the policy model using an optimizer (such as AdamW). The parameters.

[0088] refer to Figure 2 and Figure 3 The top image represents the results of the mixed loss method, while the bottom image represents the training results of the original preference learning method using only the preference loss method. The training results using the mixed loss method show more stable overall convergence and smoother training in the later stages. In contrast, although the training results using only the DPO loss method have a lower convergence loss, the training process is very unstable and even shows obvious overfitting in the later stages, which is enough to prove that the training effect of the mixed loss method is better.

[0089] S3: Obtain the historical dialogue to be replied to and input it into the trained large language model to obtain the corresponding reply statement.

[0090] Specifically, the trained large language model, trained using the improved DPO training method described above, will be deployed and applied. By inputting the historical dialogue to be replied into the trained large language model, a reply statement that is more in line with user preferences can be output.

[0091] Further reference Figure 4 As an implementation of the methods shown in the above figures, this application provides an embodiment of a dialogue generation method apparatus based on improved DPO training. This apparatus embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0092] This application provides a dialogue generation apparatus based on improved DPO training, comprising:

[0093] Fine-tuning module 1 is configured to perform supervised fine-tuning on the large language model to obtain the fine-tuned large language model; the fine-tuned large language model is used as the reference model.

[0094] Preference training module 2 is configured to construct a preference dialogue dataset; freeze the parameters of the reference model, use the fine-tuned large language model as the policy model to be trained, and perform improved DPO training on the policy model to be trained based on the preference dialogue dataset and the reference model to adjust the parameters of the policy model to be trained, thereby obtaining the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor point rating loss.

[0095] Module 3 is configured to retrieve the historical dialogues to be replied to and input them into a trained large language model to obtain the corresponding reply statements.

[0096] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device of this embodiment includes a processor 501 and a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0097] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0098] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0099] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 501, implement the above method.

[0100] This invention also provides a computer program product, including a computer program that, when executed by a processor 501, implements the above-described method.

[0101] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0102] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0103] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0104] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 501 to execute some steps of the methods of the various embodiments of this application.

[0105] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor 501. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 501 executing the steps, or as a combination of hardware and software modules within the processor 501 executing the steps.

[0106] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0107] Bus 503 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0108] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0109] An exemplary storage medium is coupled to processor 501, enabling processor 501 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 501. Processor 501 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 501 and storage medium can exist as discrete components in an electronic device or host device.

[0110] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.< / sep>

Claims

1. A dialogue generation method based on improved DPO training, characterized in that, Includes the following steps: Supervised fine-tuning of the large language model yields a fine-tuned large language model; the fine-tuned large language model is then used as a reference model. Construct a preference dialogue dataset; The parameters of the reference model are frozen, and the fine-tuned large language model is used as the policy model to be trained. Based on the preference dialogue dataset and the reference model, the policy model to be trained is trained by improved DPO to adjust the parameters of the policy model to be trained, so as to obtain the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor point rating loss. The historical dialogues to be replied to are obtained and input into the trained large language model to obtain the corresponding reply statements.

2. The dialogue generation method based on improved DPO training according to claim 1, characterized in that, In the forward propagation process of the improved DPO training, the preference loss adopts the DPO loss, the fine-tuning loss adopts the cross-entropy loss, and the anchor point rating loss adopts the mean squared error loss. In the backpropagation process, the gradient of the loss function with respect to the parameters of the policy model to be trained is calculated by the backpropagation algorithm, and the optimizer updates the parameters of the policy model to be trained according to the gradient.

3. The dialogue generation method based on improved DPO training according to claim 2, characterized in that, The process of constructing the DPO loss is as follows: The historical dialogues from the samples of the preference dialogue dataset are respectively input into the reference model and the policy model to be trained, to obtain the conditional probability of the reference model predicting the preference selection response based on the historical dialogues, the conditional probability of the reference model predicting the rejection response based on the historical dialogues, and the conditional probability of the policy model to be trained in terms of parameters. The following is based on the conditional probability of the preferred response predicted from historical dialogues, and the policy model to be trained in terms of parameters. The conditional probability of a rejected response is predicted based on historical dialogues, and the DPO loss is constructed as shown in the following formula: ； in, Indicates DPO loss, and These represent the conditional probability of the reference model predicting a preferred response based on historical dialogues and the conditional probability of the reference model predicting a rejected response based on historical dialogues, respectively. and These represent the parameters of the policy model to be trained. The conditional probability of the preferred response is predicted based on historical dialogues, and the policy model to be trained is then compared with the parameters. The conditional probability of being rejected is predicted based on historical dialogues.

4. The dialogue generation method based on improved DPO training according to claim 1, characterized in that, The process of constructing the fine-tuning loss is as follows: In the policy model to be trained, in terms of parameters In the process of predicting preferred responses based on historical dialogues, the conditional probability of each output token is obtained based on all input tokens before each time step, and a fine-tuning loss is constructed, as shown in the following formula: ； in, This represents the fine-tuning loss, where T represents the length of the text sequence. This represents the output token at time step t in the text sequence. This represents all input tokens up to the t-th time step. This indicates that the policy model to be trained has parameters According to Predicted The conditional probability.

5. The dialogue generation method based on improved DPO training according to claim 1, characterized in that, The process of constructing the anchor point scoring loss is as follows: The preference dialogue dataset also includes labeled evaluation scores of real preference selection responses, which are used to train the policy model in terms of parameters. The predicted preference responses from historical dialogues are then input into a linear layer to obtain the corresponding predicted scores. These scores are then combined with the actual preference responses to construct an anchor point rating loss, as shown in the following equation: ； Where N represents the number of samples in the preference dialogue dataset. The evaluation score represents the actual preference choice response. This indicates that the policy model to be trained has parameters The predicted score is obtained by passing the preference selection response obtained from historical dialogues through a linear layer.

6. The dialogue generation method based on improved DPO training according to claim 1, characterized in that, The hybrid loss is the sum of preference loss, fine-tuning loss, and anchor point rating loss.

7. A dialogue generation device based on improved DPO training, characterized in that, include: The fine-tuning module is configured to perform supervised fine-tuning on the large language model to obtain the fine-tuned large language model. The finely tuned large language model is used as a reference model; The preference training module is configured to build a preference dialogue dataset. The parameters of the reference model are frozen, and the fine-tuned large language model is used as the policy model to be trained. Based on the preference dialogue dataset and the reference model, the policy model to be trained is trained by improved DPO to adjust the parameters of the policy model to be trained, so as to obtain the trained large language model. The loss function used in the improved DPO training process is a hybrid loss, which includes preference loss, fine-tuning loss and anchor point rating loss. The generation module is configured to acquire the historical dialogue to be replied to and input it into the trained large language model to obtain the corresponding reply statement.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images