Large model dialogue generation optimization method and electronic device

Abstract

The present disclosure provides a large model dialogue generation optimization method and an electronic device. The large model dialogue generation optimization method comprises: obtaining a plurality of dialogue tracks generated by interaction between a target model and a user simulator, each dialogue track comprising a plurality of dialogue turns; calculating a global reward value of each dialogue track through a first value head of a value evaluation model, and performing normalization processing on the global reward value to obtain a global advantage value of each dialogue track; calculating a local reward value of each dialogue turn through a second value head of the value evaluation model, and performing normalization processing on the local reward value to obtain a local advantage value of each dialogue turn; forming a total advantage value of each dialogue turn according to the global advantage value and the local advantage value; and updating parameters of the target model based on the total advantage value through a policy optimization algorithm. The embodiments of the present disclosure can improve the interaction quality of the large model and optimize the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a method and electronic device for optimizing large-scale dialogue generation. Background Technology

[0002] Currently, industrial dialogue system optimization typically employs reinforcement learning methods, simply adding the language reward for each round to the final business reward as the training signal. However, this method suffers from gradient dominance: the high volume of business rewards overwhelms the subtle language rewards, leading to stiff and repetitive dialogue generated by the model; it also suffers from reward hijacking, where the model may pursue local high scores while neglecting the final goal, making it difficult to converge to the optimal strategy.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method and electronic device for optimizing large-scale model dialogue generation, which decouples reward signals at different time scales to balance dialogue quality and business objectives, improves the interaction quality of large models, and optimizes user experience.

[0005] According to a first aspect of the present disclosure, a method for optimizing large-scale model dialogue generation is provided, comprising: acquiring multiple dialogue trajectories generated by the interaction between a target model and a user simulator, each dialogue trajectory containing multiple dialogue rounds; calculating a global reward value for each dialogue trajectory using a first value head of a value evaluation model, and normalizing the global reward value to obtain a global advantage value for each dialogue trajectory; calculating a local reward value for each dialogue round using a second value head of the value evaluation model, and normalizing the local reward value to obtain a local advantage value for each dialogue round; forming a total advantage value for each dialogue round based on the global advantage value and the local advantage value; and updating the parameters of the target model based on the total advantage value using a policy optimization algorithm. The user simulator is constructed based on a large language model and pre-configured with user profile parameters, including user attribute parameters and decision preference parameters; when generating each dialogue trajectory, the user simulator simulates user behavior based on a set of user profile parameters and engages in multiple rounds of dialogue interaction with the target model.

[0006] According to a second aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any one of the preceding embodiments based on instructions stored in the memory.

[0007] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the large model dialogue generation optimization method as described in any of the preceding claims.

[0008] According to a fourth aspect of this disclosure, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the method as described in any of the preceding claims.

[0009] This embodiment employs two independent value networks to calculate and normalize global and local rewards respectively. This ensures that reward signals at different scales are balanced in magnitude, preventing a single-scale signal from dominating the model update direction. Consequently, while pursuing the ultimate business goal, the target model continuously optimizes the language quality of each round of dialogue, achieving a unity of natural fluency and business orientation in the target model's output, thus optimizing the user experience.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0012] Figure 1 This is a flowchart of a large-model dialogue generation optimization method in an exemplary embodiment of this disclosure.

[0013] Figure 2 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0014] Figure 3 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0015] Figure 4 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0016] Figure 5 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0017] Figure 6 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0018] Figure 7This is a block diagram of a large-model dialogue generation optimization apparatus in an exemplary embodiment of the present disclosure.

[0019] Figure 8 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0021] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of a large-model dialogue generation optimization method in an exemplary embodiment of this disclosure.

[0024] refer to Figure 1 Large-scale model dialogue generation optimization methods 100 may include: Step S1: Obtain multiple dialogue trajectories generated by the interaction between the target model and the user simulator. Each dialogue trajectory contains multiple dialogue rounds. Step S2: Calculate the global reward value for each dialogue trajectory using the first value head of the value assessment model, and normalize the global reward value to obtain the global advantage value for each dialogue trajectory. Step S3: Calculate the local reward value for each dialogue round using the second value head of the value assessment model, and normalize the local reward value to obtain the local advantage value for each dialogue round. Step S4: Calculate the total advantage value for each dialogue round based on the global advantage value and the local advantage value; Step S5: Based on the total advantage value, update the parameters of the target model using a strategy optimization algorithm.

[0025] This embodiment employs two independent value networks to calculate and normalize global and local rewards respectively. This ensures that reward signals at different scales are balanced in magnitude, preventing a single-scale signal from dominating the model update direction. Consequently, while pursuing the ultimate business goal, the target model continuously optimizes the language quality of each round of dialogue, achieving a unity of natural fluency and business orientation in the target model's output, thus optimizing the user experience.

[0026] The following section provides a detailed explanation of each step in the large-model dialogue generation optimization method 100.

[0027] In step S1, multiple dialogue trajectories generated by the interaction between the target model and the user simulator are obtained, and each dialogue trajectory contains multiple dialogue rounds.

[0028] In an exemplary embodiment, the target model refers to the large language model to be optimized, used to generate response information in multi-turn dialogues, such as customer service robots, shopping assistants, consultants and other conversational artificial intelligence systems.

[0029] In this embodiment, the user simulator is built based on a large language model and pre-configured with user profile parameters, including user attribute parameters and decision preference parameters. User attribute parameters include static features such as age, gender, occupation, and region; decision preference parameters include dynamic behavioral features such as price sensitivity, brand preference, risk tolerance, and decision hesitation.

[0030] When generating each dialogue trajectory, the user simulator simulates user behavior based on a set of user profile parameters and engages in multiple rounds of dialogue interaction with the target model.

[0031] Specifically, the user simulator generates initial user questions based on a set of configured user profile parameters and inputs them into the target model. In each round of interaction, the user simulator receives the response information output by the target model and generates the next round of user questions based on the current user profile parameters and the historical dialogue context, thereby simulating the dialogue behavior patterns of real users under specific profile characteristics.

[0032] Each dialogue trajectory corresponds to a set of user profile parameters, and different dialogue trajectories may correspond to the same or different sets of user profile parameters. By configuring diverse user profile parameters, a high-fidelity dialogue dataset covering different user types can be constructed, providing rich training samples for optimizing the target model.

[0033] In this embodiment of the disclosure, the dialogue trajectory refers to the complete interaction process between the user simulator and the target model from the first interaction to the end of the dialogue. For example, it is the complete dialogue sequence from the user simulator initiating the first round of questions, through multiple rounds of question and answer, until the preset termination condition is reached (such as completing the business goal, exceeding the maximum number of rounds, or the user actively terminating).

[0034] A dialogue turn refers to the basic interactive unit that constitutes the dialogue trajectory. Each turn includes a question output by the user simulator and an answer output by the target model in response to that question. For example, a dialogue turn is formed when the user simulator asks "Is there a discount on this product?" and the target model answers "There is a limited-time discount currently being offered".

[0035] In step S2, the global reward value of each dialogue trajectory is calculated using the first value head of the value assessment model, and the global reward value is normalized to obtain the global advantage value of each dialogue trajectory.

[0036] In this embodiment, the global reward value is a comprehensive evaluation of the final outcome of the entire dialogue trajectory, which can reflect both whether the dialogue has achieved the expected business goals and whether it meets the specification requirements.

[0037] Figure 2 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0038] refer to Figure 2 In an exemplary embodiment, step S2, which calculates the global reward value for each dialogue trajectory using the first value head of the value assessment model, may include: Step S21: Obtain the business conversion indicator score and compliance indicator score at the end of each dialogue trajectory; Step S22: The weighted sum of the business conversion indicator score and the compliance indicator score is used as the global reward value for this dialogue trajectory.

[0039] In this embodiment of the disclosure, the business conversion indicator score is a quantitative value used to measure whether the dialogue has achieved the expected business goals. For example, whether the user's problem was successfully resolved in the customer service scenario, whether the user completed the purchase in the shopping guide scenario, and whether the user left contact information in the consultation scenario. Its score range is usually a real number between 0 and 1. The compliance indicator score is a quantitative value used to measure whether the dialogue content meets the preset standard requirements. For example, whether the dialogue contains illegal expressions, whether it involves sensitive topics, and whether it meets industry regulatory requirements. Its score range is also usually a real number between 0 and 1.

[0040] In step S21, the business conversion indicator score and compliance indicator score at the end of each dialogue trajectory are obtained. Specifically, the business conversion indicator score can be determined based on the user's behavior status at the end of the dialogue (such as placing an order, making an appointment, leaving information, etc.), and the compliance indicator score can be determined based on the review results of the dialogue content.

[0041] In step S22, the global reward value of the i-th dialogue trajectory can be calculated using formula (1): = + (1) in, This represents the global reward value for the i-th dialogue trajectory. This represents the business conversion metric score for the i-th dialogue trajectory. This represents the compliance score for the i-th dialogue trajectory. and The preset weighting coefficients satisfy... It may be adjusted according to actual business needs.

[0042] By weighting and integrating business conversion metrics with compliance metrics, the global reward value can guide the target model to generate dialogue content within a compliance framework while pursuing business conversion.

[0043] Figure 3 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0044] refer to Figure 3 In an exemplary embodiment, after calculating the global reward value for each dialogue trajectory using the first value head of the value assessment model, the method further includes: Step S23: Calculate the cumulative reward of the global reward value for each dialogue trajectory using the first discount factor, and use the cumulative reward as the updated global reward value; Normalizing the global reward value means normalizing the updated global reward value.

[0045] In this embodiment of the disclosure, the discount factor refers to a decay coefficient used to adjust the degree of influence of future rewards on the current decision, and its value is typically between 0 and 1. The closer the discount factor is to 1, the more the model values ​​long-term returns; the closer the discount factor is to 0, the more the model values ​​short-term returns.

[0046] Cumulative reward refers to the sum of all rewards after the current moment, after being reduced by a discount factor. It is used to measure the contribution of the current decision to the overall future returns. For example, in a dialogue trajectory containing 10 rounds, the cumulative reward of the first round of dialogue is equal to the reward value of the first round itself, plus the reward value of the second round multiplied by the discount factor, plus the reward value of the third round multiplied by the square of the discount factor, and so on, up to the reward value of the tenth round multiplied by the 9th power of the discount factor.

[0047] In an exemplary embodiment, the formula for calculating the cumulative reward of the global reward value is, for example: (2) in This represents the global cumulative reward value for the t-th dialogue round in the i-th dialogue trajectory. Let represent the global reward value of the i-th dialogue trajectory, β represent the first discount factor used for the global reward, T represent the total number of dialogue rounds contained in the i-th dialogue trajectory, and t represent the sequence number of the current dialogue round.

[0048] In this embodiment of the disclosure, β=1.0 is set so that when calculating the total advantage value in subsequent rounds, the terminal reward can be returned to each conversation without decay.

[0049] Specifically, for all dialogue rounds along the same dialogue path, the same first discount factor is applied. Furthermore, in the embodiments disclosed herein Therefore, the global cumulative reward value for each dialogue round is equal to the global reward value for that dialogue trajectory. ,have: (3) Therefore, the terminal reward is transmitted back to each round of the dialogue without decay, so that each round of dialogue can feel the full pressure to achieve the final goal. This guides the target model to always keep its focus on the business goal in multiple rounds of interaction and avoids neglecting the final conversion due to the early rounds.

[0050] Figure 4 This is a sub-flowchart of step S2 in an exemplary embodiment of this disclosure.

[0051] refer to Figure 4 In an exemplary embodiment, step S2, which normalizes the global reward value to generate a global advantage value for each dialogue trajectory, includes: Step S24: Within a training batch, perform an independent first normalization operation on the set of global reward values ​​to form a global advantage value for each dialogue trajectory. The first standardization operation includes: for each global reward value, subtracting the mean of the set of global reward values ​​and dividing by the standard deviation of the set of global reward values.

[0052] In this embodiment of the disclosure, a training batch refers to a set of training samples used for a single model parameter update, typically containing multiple dialogue trajectories. During deep reinforcement learning training, the model does not update its parameters immediately after each dialogue trajectory is generated. Instead, it collects a certain number of dialogue trajectories, statistically calculates the reward signals of these trajectories, and then updates the parameters uniformly. This set of samples constitutes a training batch.

[0053] In an exemplary embodiment, the set of global reward values ​​can be standardized using formula (4): (4) in, This represents the global advantage value of the i-th dialogue trajectory. This represents the global reward value for the i-th dialogue trajectory, or the global reward value updated based on the first discount factor. This represents the mean of all global reward values ​​within this training batch. This represents the standard deviation of all global reward values ​​within this training batch. For a very small positive number (e.g.) ), used to avoid division by zero errors.

[0054] For example, suppose a training batch contains 3 dialogue trajectories, and their global reward values ​​are respectively , , First, calculate the average global reward value within this batch. Standard deviation Then, the global advantage value for each dialogue trajectory is calculated using formula (4):

[0055] Therefore, after the first standardized operation: Dialogue trajectory 1 (successful business conversion and good compliance, 95 points) received a significant positive global advantage value (1.15), indicating that its performance far exceeded the average level of the same batch; Dialogue trajectory 2 (business conversion failure or serious compliance issues, 20 points) received a significantly negative global advantage value (-1.32), indicating that its performance was far below the average level of the same batch; Dialogue track 3 (medium performance, 65 points) achieved a global advantage score close to zero (0.16), indicating that its performance was comparable to the average level of its batch.

[0056] Through the first standardization operation, the originally significantly different global reward values ​​(95, 20, 65) are transformed into relative advantage values ​​(1.15, -1.32, 0.16) expressed in standard deviation, clearly presenting the performance differences between different trajectories as standardized scores. This approach eliminates the inconsistency in the units of global reward values ​​caused by differences in business scenarios between different dialogue trajectories, ensuring that the global advantage value reflects the relative performance of the trajectory compared to other trajectories in the same batch, rather than its absolute value. This approach stabilizes the update direction of the policy gradient, avoiding training instability caused by excessive fluctuations in global reward values.

[0057] Furthermore, the global advantage value after standardization is within the same dimensional range as the local advantage value calculated in subsequent steps, which facilitates the weighted fusion of the two to calculate the total advantage value for each dialogue round, preventing the gradient update direction from being dominated by a signal of a certain scale due to dimensional differences in related technologies.

[0058] By employing a dual-scale independent normalization (HIAN) mechanism, reward signals at different time scales are decoupled and standardized separately. This fundamentally solves the gradient dominance problem caused by high-volume, high-variance conversation-level rewards overshadowing subtle single-round language rewards in traditional methods. After normalization, both global and local dominance values ​​are expressed in standard deviation, reflecting the "degree of superiority or inferiority relative to the average level of the same batch" at their respective scales, rather than absolute numerical values. This allows the model to simultaneously perceive the long-term pressure from the ultimate business goal and the short-term constraints from the quality of each round of dialogue when updating its strategy. While pursuing conversion, it maintains the natural flow and compliance of the dialogue, ultimately achieving a dual optimization of dialogue quality and business objectives.

[0059] In step S3, the local reward value for each dialogue round is calculated using the second value head of the value assessment model, and the local reward value is normalized to obtain the local advantage value for each dialogue round.

[0060] In this embodiment of the disclosure, the local dominance value is a quantitative indicator used to measure the quality of language in each dialogue round relative to other rounds in the same batch, reflecting the language performance level of the target model in generating responses in each round of dialogue.

[0061] Figure 5 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0062] refer to Figure 5 In an exemplary embodiment, step S3, which calculates the local reward value for each dialogue round using the second value head of the value assessment model, includes: Step S31: Obtain scores for multiple language quality indicators for one dialogue round. The multiple language quality indicators include at least one of repetition, fluency, use of interjections, and length compliance. Step S32: Calculate the local reward value for this dialogue round based on the weighted sum of scores from multiple language quality indicators.

[0063] exist Figure 5 In the illustrated embodiment, a round of dialogue is scored using multiple language quality metrics to measure the local reward value of that round of dialogue. The multiple language quality metrics include at least one of repetition, fluency, and compliance.

[0064] The repetition rate includes intra-sentence repetition and inter-sentence repetition, used to measure whether there is repetition of words, phrases, or sentences in the responses generated by the model in the current round. Intra-sentence repetition focuses on repetition within the same sentence, such as "okay, okay, okay"; inter-sentence repetition focuses on repetition with previous rounds, such as repeating the exact same phrases as in previous rounds. The higher the repetition rate, the worse the language quality, and the lower the corresponding score.

[0065] The repetition score is obtained by calculating the repetition rate in the current round of responses. Specifically, for intra-sentence repetition, the proportion of repeated words or phrases in the current response is counted; for inter-sentence repetition, the similarity between the current response and responses in previous rounds of dialogue is compared to calculate the proportion of repeated content. The higher the repetition rate, the lower the repetition score.

[0066] Fluency is a quantitative metric used to measure whether the responses generated by a model are grammatically and semantically fluent and natural, including whether the sentence structure is complete, whether the word collocation is reasonable, and whether there are any grammatical errors. The higher the fluency, the better the language quality, and the higher the corresponding score.

[0067] Fluency scores can be obtained by calculating the perplexity of the current answer using the language model built into the second value computation network. The lower the perplexity, the more fluent and natural the answer, and the higher the fluency score. Alternatively, scores can be given based on rules, such as sentence length and grammatical structure completeness.

[0068] The usage rate of modal particles refers to a quantitative indicator used to measure whether the responses generated by the model contain appropriate modal particles (such as "um," "ah," "oh," "ne," etc.) and whether their frequency of use is reasonable. The use of modal particles can enhance the naturalness and friendliness of the conversation, but overuse or inappropriate use can reduce the quality of language.

[0069] The score for the use of modal particles can be calculated by statistically analyzing the frequency and positional distribution of modal particles in the current answer and comparing it with preset modal particle usage rules. For example, a modal particle density threshold can be set; a higher score can be given when the density of modal particles is within a reasonable range, and a lower score can be given when there are too few or too many modal particles.

[0070] Length compliance is a quantitative metric used to measure whether the length of the responses generated by the model meets the preset requirements. Responses that are too long may lead to information redundancy and increase the user's reading burden; responses that are too short may lead to insufficient information and an inability to effectively respond to user needs.

[0071] Length compliance scores are calculated by comparing the actual length of the current answer with a preset ideal length range. For example, minimum and maximum length thresholds can be set; a higher score is given when the answer length is within the ideal range, and a lower score is given according to the degree of deviation when the answer length exceeds the range.

[0072] In some embodiments, language quality metrics may also include compliance.

[0073] Compliance is a quantitative metric used to measure whether the answers generated by the model meet preset standards, such as whether there are any inappropriate expressions, whether they involve sensitive topics, or whether they comply with industry regulatory requirements. Higher compliance indicates better language quality and a higher score.

[0074] Compliance scoring can be achieved by matching the current answer against a pre-set sensitive word library or a violation word library. The fewer sensitive words matched, the higher the compliance score. Alternatively, a trained compliance classification model can be used to judge the compliance of the answer.

[0075] The scores for each language quality indicator can be calculated using the scoring module built into the second value head of the value assessment model.

[0076] In this embodiment, the value assessment model, for example, is developed based on a large language model and has two independent value heads: a first value head and a second value head. The first and second value heads are trained separately on different reward signals, enabling accurate assessment of global-scale business conversion and compliance performance, as well as local-scale language quality performance. The two value heads are independent and maintain their own network parameters, ensuring that reward signals at different scales do not interfere with each other during the assessment process.

[0077] By calculating and weighting the scores of multiple language quality metrics for a single dialogue turn, a fine-grained local reward value for that turn can be obtained, enabling the target model to perceive the language performance in each round of dialogue. This intensive reward signal guides the model to continuously optimize the generation quality of each round of dialogue while pursuing long-term business goals, avoiding issues such as stiff, repetitive, or non-compliant language due to excessive focus on the final transaction.

[0078] After obtaining the local reward value for a dialogue turn, continue to calculate the local reward values ​​for other dialogue turns in the same trajectory or dialogue turns in other trajectories, until the calculation of the local reward value for each dialogue turn in all multiple dialogue trajectories is completed.

[0079] Figure 6 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0080] refer to Figure 6 In an exemplary embodiment, after calculating the local reward value for each dialogue round using the second value head of the value assessment model in step S3, the method further includes: Step S33: Calculate the cumulative reward of the local reward value for each dialogue round using the second discount factor, and use the cumulative reward as the updated local reward value; Normalizing the local reward value involves normalizing the updated local reward value, where the second discount factor is less than the first discount factor.

[0081] In this embodiment of the disclosure, calculating the cumulative reward of the local reward value for each dialogue round using a second discount factor (e.g., α=0.99) means summing the local reward values ​​of the current round and all subsequent rounds according to time decay. Specifically, for the t-th round of the i-th dialogue trajectory containing T rounds, the formula for calculating its local cumulative reward is, for example: = +α +α 2 +…+α T-t (5) in, Let represent the local reward value in the t-th round of the i-th dialogue trajectory. This represents the second discount factor, where α < β (the first discount factor).

[0082] By using a second discount factor, which is less than the first discount factor, to calculate the cumulative return of the local reward value, the earlier the round, the greater its impact on subsequent language quality, and the higher its cumulative return; the later the round, the smaller its impact on the future, and the lower its cumulative return. This can reflect the time decay characteristics of the impact of the round sequence on the completion of business objectives, so that the model can perceive the differentiated contribution of different rounds in the language quality dimension when updating the strategy, and avoid treating all rounds equally.

[0083] Furthermore, when α=0.99 and β=1, a dual-scale design can be achieved where local rewards decay over time while global rewards are returned without decay, ensuring that the model continuously optimizes language performance in each round while pursuing long-term business goals.

[0084] Finally, the cumulative reward calculated based on the local reward value of the trajectory for a dialogue turn is used as the updated local reward value for that dialogue turn. In subsequent steps, the updated local reward value is normalized, which allows the normalization operation to be applied to the cumulative reward that already includes time decay information, further enhancing the model's ability to perceive the contribution of language quality in different turns.

[0085] Furthermore, consistent with the logic of normalizing the global reward value, in the exemplary embodiment, normalizing the local reward value to generate a local advantage value for each dialogue round includes: within a training batch, performing an independent second normalization operation on the set of local reward values ​​to form a local advantage value for each dialogue round; the second normalization operation includes: for each local reward value, subtracting the mean of the set of local reward values ​​and dividing by the standard deviation of the set of local reward values.

[0086] In an exemplary embodiment, the set of local reward values ​​can be subjected to a second normalization operation using formula (6): (4) in, This represents the local advantage value of the i-th dialogue trajectory in the t-th round of dialogue. This represents the local reward value of the i-th dialogue trajectory in the t-th round, or the local reward value updated based on the second discount factor. This represents the mean of all global reward values ​​within this training batch. This represents the standard deviation of all global reward values ​​within this training batch. For a very small positive number (e.g.) ), used to avoid division by zero errors.

[0087] Assuming a training batch contains 3 dialogue trajectories, their round distribution and local reward values ​​updated based on the second discount factor are given. as follows: Track 1: Contains 3 dialogue rounds, with local reward values ​​updated based on the second discount factor as follows: , , ; Track 2: Contains 2 dialogue rounds, with local reward values ​​updated based on the second discount factor as follows: , ; Track 3: Contains 3 dialogue rounds, with local reward values ​​updated based on the second discount factor as follows: , , .

[0088] First, calculate the mean of the local cumulative reward value for all dialogue rounds (3+2+3=8 rounds) within this training batch. and standard deviation The values ​​are 0.6625 and 0.18 respectively. Then, the local reward value will be updated based on the second discount factor. Replace in formula (6) Calculate the local advantage value for each dialogue turn:

[0089] Therefore, after the second standardized operation: Rounds with significantly higher-than-average language quality (e.g., 1.60 in round 1 of trajectory 3 and 1.04 in round 1 of trajectory 1) achieved significant positive local dominance values; Rounds with language quality comparable to the average level (e.g., round 2 of trajectory 1 -0.07, round 2 of trajectory 3 0.21) achieved local dominance values ​​close to zero; Rounds with language quality significantly below average (such as round 3 of trajectory 1, -1.46, and round 3 of trajectory 3, -1.18) achieved significant negative local dominance values.

[0090] Through the second standardization operation, the locally accumulated reward values, which originally had limited absolute numerical differences, are transformed into relative advantage values ​​measured in standard deviations. This allows the differences in language quality between different rounds to be clearly presented as standardized scores. This approach stabilizes the update direction of the policy gradient and avoids training instability caused by differences in the absolute numerical values ​​of local reward values.

[0091] Furthermore, the local advantage value and the global advantage value after standardization are within the same dimension range, which facilitates the weighted fusion of the two to calculate the total advantage value for each dialogue round. This prevents the signal of a certain scale from dominating the gradient update direction due to the difference in dimensions. This is the core inventive concept of this application in solving the gradient dominance problem through a dual-scale independent normalization mechanism.

[0092] In step S4, the total advantage value for each dialogue round is formed based on the global advantage value and the local advantage value.

[0093] In an exemplary embodiment, the normalized local advantage value of each dialogue round can be weighted and summed with the normalized global advantage value of the dialogue trajectory in that dialogue round to form the total advantage value for each dialogue round. For example, a preset first weight coefficient and a second weight coefficient can be obtained; the product of the global advantage value and the first weight coefficient, and the product of the local advantage value and the second weight coefficient, are added together to obtain the total advantage value.

[0094] For example, continuing from the previous example, suppose a training batch contains 3 dialogue trajectories, whose global advantage value and local advantage value for each round have been calculated through the aforementioned steps: Global advantage value of trajectory 1:

[0095] Global advantage value of trajectory 2:

[0096] Global advantage value of trajectory 3:

[0097] The local dominance values ​​for each dialogue round in each trajectory are shown in the example above: Track 1: , ,

[0098] Track 2: ,

[0099] Track 3: , ,

[0100] Set the first weighting factor of the fusion. Second weighting coefficient Then the total advantage value for each dialogue round can be calculated using formula (7): (7) The total advantage value for each round is calculated as follows: Track 1:

[0101] Track 2:

[0102] Track 3:

[0103] Through the weighted fusion process described above, the total advantage value of each dialogue round simultaneously includes language quality information from the local scale and business objective information from the global scale.

[0104] For example, although Trajectory 1 had a high global advantage value (1.15) in Round 1, it also had a high local advantage value (1.04), thus achieving the highest total advantage value (1.106). Trajectory 1 in Round 3, despite being within the same high-transaction trajectory, had a significantly lower total advantage value (0.106) due to its language quality being significantly below average (local advantage value -1.46), reflecting the constraint of language quality. Conversely, Trajectory 2 had poor overall transaction performance (global advantage value -1.32), but its language quality in Round 1 was acceptable (local advantage value 0.49), resulting in a negative total advantage value (-0.596), reflecting the guiding role of business objectives.

[0105] This weighted fusion mechanism enables the model to simultaneously consider the language quality of each round of dialogue and the ultimate business objective when updating its strategy, avoiding the dominance of a single-scale signal in the gradient update direction, thereby achieving dual optimization of dialogue quality and business conversion. Therefore, this embodiment of the disclosure solves the problems of gradient dominance and reward hijacking through this dual-scale credit allocation framework.

[0106] In step S5, the parameters of the target model are updated using a strategy optimization algorithm based on the total advantage value.

[0107] In an exemplary embodiment, in step S5, the total advantage value of each dialogue round can be substituted into the loss function of the near-end policy optimization algorithm to calculate the gradient; then, based on the gradient, the parameters of the target model are updated through the backpropagation algorithm.

[0108] Specifically, the loss function of the near-end policy optimization algorithm is constructed based on the total advantage value of each dialogue turn. It compares the probability ratio of generating a response in the same state between the current model policy and the previous model policy, and weights this ratio with the total advantage value. This loss function also includes a pruning mechanism to limit the magnitude of a single policy update, preventing training instability due to excessively large updates.

[0109] After calculating the loss function, the gradient of the loss function with respect to the target model parameters is calculated using the backpropagation algorithm, and the model parameters are updated along the gradient direction. The learning rate is used to control the step size of parameter updates, ensuring that the model can stably converge to the optimal policy.

[0110] By substituting the total advantage value of each dialogue turn into the loss function of the proximal policy optimization algorithm, the target model can be made to increase the probability of dialogue actions that bring high total advantage values ​​and decrease the probability of dialogue actions that bring low total advantage values ​​when updating parameters.

[0111] Since the overall advantage value has already incorporated global-scale business objective information and local-scale language quality information through the aforementioned steps, the model can simultaneously perceive constraints and guidance from both scales during the update process, continuously optimizing the language performance of each round of dialogue while pursuing the final business conversion.

[0112] This strategy optimization approach based on dual-scale total advantage value enables the target model to gradually learn dialogue strategies that meet both language quality requirements and effectively achieve business goals through multiple rounds of interaction, ultimately achieving a comprehensive improvement in dialogue generation quality and a significant improvement in business conversion results.

[0113] In summary, the embodiments of this disclosure construct a dual-scale credit allocation framework to decouple reward signals at different time scales and adopt a scale-independent advantage normalization mechanism to achieve balanced optimization of multi-objective signals. This can effectively solve the technical problems of gradient dominance, reward hijacking, and training instability in traditional methods.

[0114] At the technical implementation level, the embodiments of this disclosure generate diverse dialogue trajectories through a high-fidelity user simulator, and perform refined reward modeling from both the global business objective dimension and the local language quality dimension. On this basis, a double discount factor is introduced to calculate the cumulative rewards at different scales, and the global advantage value and the local advantage value are mapped to a unified dimension space through independent standardization operations within the batch. Finally, the total advantage value of each dialogue round is formed by weighted fusion to guide the strategy optimization and update of the target model.

[0115] Experimental results show that optimizing the target model using the scheme of this disclosure embodiment enables the target model to continuously optimize the language performance of each round of dialogue while pursuing the final business conversion. This achieves significant technical effects such as a 6.82% increase in conversion rate, an 11.36% reduction in the number of dialogue rounds, and an 82.28% reduction in inter-sentence repetition rate. Furthermore, it possesses high fidelity simulation and a high degree of correlation with the real environment, providing better optimization support for the application of large-model dialogue generation technology in industrial scenarios.

[0116] Corresponding to the above method embodiments, this disclosure also provides a large model dialogue generation optimization apparatus, which can be used to execute the above method embodiments.

[0117] Figure 7 This is a block diagram of a large-model dialogue generation optimization apparatus in an exemplary embodiment of the present disclosure.

[0118] refer to Figure 7 The large-model dialogue generation optimization device 700 may include: The training data generation module 71 is configured to acquire multiple dialogue trajectories generated by the interaction between the target model and the user simulator, with each dialogue trajectory containing multiple dialogue rounds; The global advantage independent calculation module 72 is configured to calculate the global reward value of each dialogue trajectory through the first value head of the value assessment model, and normalize the global reward value to obtain the global advantage value of each dialogue trajectory. The local advantage independent calculation module 73 is configured to calculate the local reward value for each dialogue round through the second value head of the value assessment model, and normalize the local reward value to obtain the local advantage value for each dialogue round. The round advantage calculation module 74 is configured to form the total advantage value for each dialogue round based on the global advantage value and the local advantage value. The model update module 75 is configured to update the parameters of the target model based on the total advantage value using a policy optimization algorithm.

[0119] The user simulator is built based on a large language model and pre-configured with user profile parameters, including user attribute parameters and decision preference parameters. When generating each dialogue trajectory, the user simulator simulates user behavior based on a set of user profile parameters and engages in multi-round dialogue interaction with the target model.

[0120] Since the functions of the device 700 have been described in detail in their respective method embodiments, they will not be repeated here.

[0121] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0122] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0123] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0124] The following reference Figure 8 To describe an electronic device 800 according to this embodiment of the present invention. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0125] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processor 810, at least one memory 820, and a bus 830 connecting different system components (including memory 820 and processor 810).

[0126] The memory stores program code that can be executed by the processor 810, causing the processor 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processor 810 can perform the methods shown in the embodiments of this disclosure.

[0127] The memory 820 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8201 and / or cache 8202, and may further include read-only memory (ROM) 8203.

[0128] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0129] Bus 830 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0130] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0132] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0133] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0135] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0137] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0139] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. A method for optimizing large-scale dialogue generation, characterized in that, include: Acquire multiple dialogue trajectories generated by the interaction between the target model and the user simulator, with each dialogue trajectory containing multiple dialogue rounds; The global reward value of each dialogue trajectory is calculated using the first value head of the value assessment model, and the global reward value is normalized to obtain the global advantage value of each dialogue trajectory. The local reward value for each dialogue round is calculated using the second value head of the value assessment model, and the local reward value is normalized to obtain the local advantage value for each dialogue round. The total advantage value for each dialogue round is formed based on the global advantage value and the local advantage value. Based on the total advantage value, the parameters of the target model are updated using a policy optimization algorithm; The user simulator is built based on a large language model and pre-configured with user profile parameters, including user attribute parameters and decision preference parameters. When generating each dialogue trajectory, the user simulator simulates user behavior based on a set of user profile parameters and engages in multi-round dialogue interaction with the target model.

2. The large-model dialogue generation optimization method as described in claim 1, characterized in that, The calculation of the global reward value for each dialogue trajectory using the first value head of the value assessment model includes: Obtain the business conversion metric score and compliance metric score at the end of a conversation; The global reward value for this dialogue trajectory is formed by the weighted sum of the business conversion indicator score and the compliance indicator score.

3. The large-model dialogue generation optimization method as described in claim 1, characterized in that, The calculation of the local reward value for each dialogue round using the second value head of the value assessment model includes: Obtain scores for multiple language quality metrics for a single dialogue turn, wherein the multiple language quality metrics include at least one of repetition, fluency, use of interjections, and length compliance; The local reward value for this dialogue round is formed by a weighted sum of scores from multiple language quality metrics.

4. The large-model dialogue generation optimization method as described in claim 2, characterized in that, After calculating the global reward value for each dialogue trajectory using the first value head of the value assessment model, the method further includes: The cumulative reward of the global reward value for each dialogue trajectory is calculated using the first discount factor, and the cumulative reward is used as the updated global reward value. The normalization process for the global reward value is to normalize the updated global reward value.

5. The large-model dialogue generation optimization method as described in claim 4, characterized in that, After calculating the local reward value for each dialogue round using the second value head of the value assessment model, the following is also included: The cumulative reward of the local reward value for each dialogue round is calculated using a second discount factor, and the cumulative reward is used as the updated local reward value. The normalization process for the local reward value is to normalize the updated local reward value. Wherein, the second discount factor is less than the first discount factor.

6. The large-model dialogue generation optimization method as described in any one of claims 1, 2, or 4, characterized in that, The global reward value is normalized to obtain the global advantage value for each dialogue trajectory, including: Within a training batch, the set of global reward values ​​is subjected to an independent first normalization operation to form a global advantage value for each dialogue trajectory; The first standardization operation includes: for each global reward value, subtracting the mean of the set of global reward values ​​and dividing by the standard deviation of the set of global reward values.

7. The large-model dialogue generation optimization method as described in any one of claims 1, 3, or 5, characterized in that, The local reward values ​​are normalized to obtain the local advantage value for each dialogue round, including: Within a training batch, the set of local reward values ​​is subjected to an independent second normalization operation to form a local advantage value for each dialogue round; The second standardization operation includes: for each of the local reward values, subtracting the mean of the set of local reward values ​​and dividing by the standard deviation of the set of local reward values.

8. The large-model dialogue generation optimization method as described in claim 1, characterized in that, The step of updating the parameters of the target model based on the total advantage value using a policy optimization algorithm includes: The total advantage value for each dialogue round is substituted into the loss function of the near-end policy optimization algorithm to calculate the gradient; Based on the gradient, the parameters of the target model are updated using the backpropagation algorithm.

9. The large-model dialogue generation optimization method as described in claim 1, characterized in that, The total advantage value for each dialogue round is formed based on the global advantage value and the local advantage value, including: Obtain the preset first and second weighting coefficients; The total advantage value is obtained by adding the product of the global advantage value and the first weight coefficient, and the product of the local advantage value and the second weight coefficient.

10. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1-9 based on instructions stored in the memory.

Images