Conversation processing method and device and computer equipment

Abstract

The invention provides a dialogue processing method and device and computer equipment, and the method comprises the steps: determining first information based on current input information; cutting the first information by using a cutting model to obtain second information, the second information representing key information of the first information, and the second information serving as input of a preset dialogue model to generate reply information corresponding to the current input information; wherein the cutting model is obtained by performing reinforcement learning training by using a preset multi-dimensional reward function, and the multi-dimensional reward function is used for updating parameters of the to-be-trained cutting model at least once. In this way, the first information is clipped through the clipping model trained by using a multi-dimensional reward function and a reinforcement learning feedback mechanism, so that the output second information keeps key information, and meanwhile, the semantic integrity and the length controllability are considered.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a dialogue processing method, apparatus, and computer equipment. Background Technology

[0002] In recent years, Large Language Models (LLMs) have been widely used in fields such as intelligent customer service, dialogue systems, education, and healthcare due to their powerful natural language understanding and generation capabilities. Related technologies typically employ methods such as deleting early dialogue content or compressing context to control the input length of LLMs. However, this often results in incomplete retention of key information and a high risk of progressive loss of core semantics, leading to a significant decrease in the accuracy of dialogue responses. Summary of the Invention

[0003] This application provides a dialogue processing method, apparatus, and computer device.

[0004] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a dialogue processing method, the method comprising: Based on the current input information, determine the first piece of information; The first information is cropped using a cropping model to obtain the second information. The second information represents the key information of the first information. The second information is used as the input of a preset dialogue model to generate the response information corresponding to the current input information. The pruning model is trained using a preset multi-dimensional reward function through reinforcement learning. The multi-dimensional reward function is used to update the parameters of the pruning model at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the pruned key information, the second reward value represents the length compliance of the pruned key information, the third reward value represents the redundancy of the pruned key information, and the fourth reward value represents the format compliance of the pruned key information.

[0005] Secondly, embodiments of this application provide a dialogue processing apparatus, including: The determination module is used to determine the first information based on the current input information; The trimming module is used to trim the first information using a trimming model to obtain second information. The second information represents the key information of the first information. The second information is used as input to a preset dialogue model to generate response information corresponding to the current input information. The pruning model is trained using a preset multi-dimensional reward function through reinforcement learning. The multi-dimensional reward function is used to update the parameters of the pruning model at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the pruned key information, the second reward value represents the length compliance of the pruned key information, the third reward value represents the redundancy of the pruned key information, and the fourth reward value represents the format compliance of the pruned key information.

[0006] Thirdly, embodiments of this application provide a computer device, the computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement some or all of the steps in the above method. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating the implementation flow of the dialogue processing method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the processing procedure of the dialogue processing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the training process of the cropping model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the dialogue processing device provided in the embodiments of this application.

[0008] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the embodiments of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the embodiments of this application.

[0010] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0011] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of this application belong. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0013] The technical solutions in the embodiments of this application will now be clearly and completely described with reference to the accompanying drawings.

[0014] Current large language models have a fixed context window limit in dialogue interaction. When the accumulated context length from multiple rounds of dialogue exceeds the window limit, context pruning is necessary to ensure the model functions correctly. Context pruning techniques mainly fall into three categories: Rule-based pruning: Based on the "time decay" principle, the latest round of dialogue is retained, or the content at the end is truncated based on a fixed length, such as directly deleting the earliest N rounds of dialogue; Supervised fine-tuning pruning: A dedicated pruning model is trained by manually labeling “original dialogue-pruned dialogue” sample pairs, enabling the model to learn fixed pruning rules; Simple semantic trimming: The text summarization model is used to compress the context, extracting long texts into short texts before inputting them into the large model.

[0015] The above pruning techniques have the following problems: On the one hand, semantic preservation and length control are difficult to balance. Rule-based pruning is prone to accidentally deleting key information, leading to semantic breaks. Simple semantic compression often loses details, while supervised fine-tuning, although focusing on semantic integrity, often has difficulty effectively controlling the output length. On the other hand, in multi-turn dialogue interactions, if the context is repeatedly pruned, the core information will gradually decay, leading to context collapse.

[0016] If a fixed trimming strategy is relied upon and cannot be dynamically adjusted according to the length of the conversation or the scenario, the redundancy recognition capability is limited and it is difficult to cover various forms of redundancy such as repeated questions and piled-up interjections, resulting in the trimming results still containing a large amount of invalid content.

[0017] To address at least some of the aforementioned deficiencies in related technologies, embodiments of this application provide a dialogue processing method, such as... Figure 1 As shown, Figure 1This application provides a schematic diagram of the implementation flow of a dialogue processing method, which includes the following steps S11 and S12: Step S11: Determine the first information based on the current input information.

[0018] Here, the current input information can refer to the information entered at the current moment. This current input information can be the information selected by the user from multiple alternative information, or it can be information entered by the user through other means, or it can be obtained from a preset dialogue model. This application does not impose specific limitations on this. In the embodiments of this application, the dialogue model can be a Large Language Model (LLM), a Retrieval-based Dialogue Model, a Generative Dialogue Model, etc. For example, the dialogue model can be a generative pre-trained model based on the Transformer architecture (self-attention transformation architecture) (such as the GPT series (Generative Pre-trained Transformer), LLaMA (Large Language Model Meta AI), Qwen, etc.). The dialogue model can also be other models that can provide corresponding responses. The specific dialogue model can be selected according to the actual situation, and this application does not impose limitations.

[0019] The first information can be the same as or different from the current input information. The method for determining the first information can be any suitable method. In some implementations, the current input information can be directly used as the first information. In some implementations, the first information can be determined based on the current input information and historical dialogue information.

[0020] Step S12: The first information is pruned using a pruning model to obtain second information. The second information represents the key information of the first information. The second information is used as input to a preset dialogue model to generate the response information corresponding to the current input information. The pruning model is trained using a preset multi-dimensional reward function through reinforcement learning. The multi-dimensional reward function is used to update the parameters of the pruning model to be trained at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the pruned key information. The second reward value represents the length compliance of the pruned key information. The third reward value represents the redundancy of the pruned key information. The fourth reward value represents the format compliance of the pruned key information.

[0021] Here, the pruning model refers to the model trained using reinforcement learning (RL). Its training process is based on a pre-defined multi-dimensional reward function, which quantifies and scores the pruning results from multiple complementary evaluation dimensions, thereby guiding the continuous optimization of the pruning model during training.

[0022] It is important to understand that the first reward value is used to measure the semantic performance of the pruned output, that is, whether the core content of the original information is completely preserved and its meaning is accurately conveyed, ensuring that key facts, logical relationships and user intent are not distorted or lost; the second reward value is used to evaluate whether the length of the pruned output meets the preset constraints, which can be a preset maximum number of tokens or a preset maximum number of characters, to achieve concise expression while meeting the requirements; the third reward value is used to suppress redundant information and eliminate repetitive, verbose or irrelevant content as much as possible; the fourth reward value is used to ensure that the output format meets the preset format requirements.

[0023] Key information refers to the optimal content to retain in the first piece of information determined by the trimming model.

[0024] In some implementations, a pruning model can be used to directly prune redundant information in the first information and retain the information as the second information.

[0025] In some implementations, a cropping model can be used to extract key information from the first information to obtain the second information.

[0026] In some implementations, key information from the first information can be obtained first using a preset extraction model, and then the obtained key information can be transmitted to a trimming model. The trimming model can then trim redundant information from the key information, and the retained information can be used as the second information. The extraction model can be a pre-trained language model, a large language model, etc., and is not limited thereto in this application.

[0027] The parameters of the pruning model are the learnable parameters within the pruning model, such as the number of network layers and weights.

[0028] For example, if the output of the cropping model does not fully retain key information during training, resulting in a low first reward value, then the parameters for retaining key information in the cropping model need to be adjusted to increase the cropping model's attention to semantics.

[0029] For example, during training, if the length of the output of the pruning model exceeds the length threshold, resulting in non-compliant length and a low second reward value, the weights of core semantics and non-core semantics related to key information in the pruning model can be adjusted (e.g., increasing the weight of core semantics to make it more likely that core semantics are retained in key information; decreasing the weight of non-core semantics to make it less likely that non-core semantics are retained in key information) to reduce the length of the output of the pruning model.

[0030] After the first piece of information is cropped, the resulting second piece of information can be input into the dialogue model to obtain the response information corresponding to the current input information. It is understood that the response information can be generated directly by the dialogue model, or it can be selected from multiple alternative response information generated by the dialogue model. The response information can be text information, voice information, image information, etc., and this application does not impose any limitations on this.

[0031] In some implementations, first information can be determined based on the received current input information, and then the first information is transmitted to an external trimming model. The trimming model trims the first information to obtain second information and sends the second information to a large language model. The large language model then uses the second information to generate the response information corresponding to the current input information. In practice, the current input information and response information of this round can be cached (e.g., stored in the large language model, other databases, etc.) to support the next round of dialogue interaction.

[0032] In this embodiment, on the one hand, by introducing a multi-dimensional reward function to train the pruning model through reinforcement learning, the trained pruning model can retain key information while maintaining semantic integrity, effectively eliminating redundant content, and thus significantly improving the quality of the pruned context. On the other hand, the second information pruned by the pruning model is used as input to the dialogue model to generate response information corresponding to the current input information, ensuring that the generated response information highly matches the current input information, further enhancing the accuracy and adaptability of the dialogue response. Furthermore, based on the multi-dimensional reward function, the parameters of the pruning model to be trained are updated at least once, enabling the pruning model to achieve dynamic optimization and improving the training efficiency of the pruning model.

[0033] In some implementations, the step S12 of "cutting the first information using a clipping model to obtain the second information" includes: if the length of the first information is greater than a preset length threshold, cutting the first information using a clipping model to obtain the second information.

[0034] Here, the length threshold can be a pre-set length threshold or the maximum window value of the clipping model.

[0035] It should be understood that the maximum window size of the pruning model can be determined based on the model architecture and training data. The maximum window size of the pruning model is a reasonable value that, within the hard upper limit allowed by the model architecture, takes into account the length distribution of the training data, the resource limitations of the deployed hardware, and actual needs, ensuring that it can cover the vast majority of inputs without excessively wasting computational resources. The maximum window size can be 2048 tokens, 4096 tokens, or 8192 tokens, and can be adjusted according to requirements; this application does not impose any restrictions on this.

[0036] In practice, the processing procedure of this dialogue processing method can be as follows: Figure 2 As shown, where: Step S21: Determine the first information based on the current input information; Step S22: Determine whether the first information is greater than the maximum window value of the clipping model. If yes, proceed to step S23; otherwise, proceed to step S24. Step S23: Use the pruning model to prune the first information to obtain the second information, and input the second information into the large language model; Step S24: Input the first information into the large language model; Step S25: Use the large language model to generate the response information corresponding to the current input information.

[0037] In some implementations, the pruning model can be an agent model trained under the veRL (Volcano Engine Reinforcement Learning for LLMs) framework, where the pruning model is mainly used to remove redundant information and retain key information.

[0038] In this embodiment, by only starting the pruning model when the length of the first information exceeds the length threshold, unnecessary pruning operations can be effectively avoided, computing resources can be saved and operating efficiency can be improved, thereby significantly improving the response quality and user experience of large models in practical applications.

[0039] In some implementations, step S11 includes: determining first information based on the current input information and the at least one round of historical dialogue information, if it is determined that at least one round of historical dialogue information exists.

[0040] Here, historical dialogue information can be information entered by the user at a historical moment and the corresponding response information (i.e., reply information). In some implementations, historical dialogue information can be stored in the dialogue model, the pruning model, or any suitable storage object, which is not limited in this application.

[0041] In some implementations, if at least one round of historical dialogue information exists, all historical dialogue information can be extracted and concatenated with the current input information to obtain the first information. For example, if the user's current input is: "I previously subscribed to a monthly membership and want to upgrade to an annual membership. What's the price difference? How do I pay?", this can be concatenated with the historical dialogue information "User: What types of memberships are there? Customer Service: There are two types: monthly membership for 29 yuan and annual membership for 199 yuan" to form the first information: "User: What types of memberships are there? Customer Service: There are two types: monthly membership for 29 yuan and annual membership for 199 yuan. User: I previously subscribed to a monthly membership and want to upgrade to an annual membership. What's the price difference? How do I pay?".

[0042] In some implementations, if there is multi-round historical dialogue information, the historical dialogue information with the highest similarity to the current input information can be used as the target historical dialogue information, and the target historical dialogue information can be concatenated with the current input information to obtain the first information.

[0043] In some implementations, if there is multi-turn historical dialogue information, the user can select the historical dialogue information to be spliced ​​from the multi-turn historical dialogue information according to their needs, and splice the historical dialogue information to be spliced ​​with the current input information to obtain the first information.

[0044] Understandably, if there is no historical dialogue information, the current input information can be used as the first information.

[0045] In this embodiment of the application, by concatenating historical dialogue information with current input information, misunderstandings or erroneous responses caused by the lack of historical dialogue information can be effectively avoided, and the risk of possible semantic deviation can be reduced.

[0046] In some implementations, after obtaining the response information corresponding to the current input information, the dialogue processing method further includes: generating new historical dialogue information based on the current input information, the response information corresponding to the current input information, and at least one round of historical dialogue information.

[0047] Here, new historical dialogue information may include the current input information and the corresponding response information, or it may include the current input information, the response information corresponding to the current input information, and at least one round of historical dialogue information.

[0048] In some implementations, if at least one round of historical dialogue information exists, the similarity (or correlation) between the current round of dialogue information (including the current input information and the corresponding response information) and each round of historical dialogue information can be determined. If all similarities are lower than a preset similarity threshold (e.g., 80%, 90%), the current round of dialogue information is used as new historical dialogue information. If at least one similarity is not less than the similarity threshold, the historical dialogue information with a similarity not less than the similarity threshold is used as the target historical dialogue information. The current round of dialogue information and at least one round of target historical dialogue information are then concatenated to obtain new historical dialogue information.

[0049] Understandably, if there is no historical dialogue information at least once, the current input information and the corresponding response information can be concatenated to obtain new historical dialogue information.

[0050] In some implementations, a large language model can be used to concatenate the current round of dialogue information with at least one round of target historical dialogue information to obtain new historical dialogue information.

[0051] In some implementations, any suitable model, such as a recurrent neural network (RNN), a bidirectional encoder representation from transformers (BERT), or a memory network, can be used to concatenate the current round of dialogue information with at least one round of target historical dialogue information to obtain new historical dialogue information.

[0052] In some implementations, the current round of dialogue information and at least one round of target historical dialogue information can be concatenated according to dialogue roles, time sequence, degree of relevance, etc., to obtain semantically complete new historical dialogue information. In practice, this can be used as the historical dialogue information for the new round of dialogue, allowing the pruning model to use it in the new round of interaction.

[0053] In this embodiment, by generating new historical dialogue information based on the current input information, the response information corresponding to the current input information, and at least one round of historical dialogue information, the latest interaction content can be continuously integrated to ensure that historical information is dynamically updated rather than statically truncated. This effectively avoids the loss of core semantics or context collapse caused by repeated pruning, enabling the large language model to more accurately understand user intent and dialogue progress in subsequent rounds, reducing misunderstandings or repeated questions.

[0054] In some implementations, the processing method further includes: training a first pruning model based on a training dataset to obtain a second pruning model; and performing reinforcement learning training on the second pruning model based on a multi-dimensional reward function to obtain a pruning model.

[0055] Here, the training dataset can refer to a sample dataset generated by calling the optimized prompt words, or it can be a real dialogue fragment extracted from user interaction, or it can be a question-and-answer pair obtained by reconstructing a public dataset through instruction. This application does not limit the way the training dataset is generated.

[0056] The first pruning model can be a large language model (such as Qwen3-1.7B) or other lightweight models. The second pruning model is obtained after supervised training of the first pruning model, and the final pruning model is obtained after reinforcement training of the second pruning model. In other words, the first pruning model is an untrained model or an initial model.

[0057] In some implementations, a request is sent to a pre-defined open-source model (such as Qwen-Max, GPT-4, etc.) to simulate and generate multi-turn dialogues, and to actively introduce typical redundant content (such as repeated questions, piled-up interjections, irrelevant chatter, information backtracking, etc.) to output the original dataset. The original dataset contains a large number of triple samples (i.e., the original dialogue, the labeled optimal retained content, and the core semantic summary), and needs to cover a variety of scenarios (such as customer service, daily chat, work communication, etc.) to ensure that the trained model has generalization ability.

[0058] Furthermore, the generated raw dataset can be cleaned, such as by manually sampling each triplet sample and deleting samples with labeling errors (e.g., missing core information), unrealistic dialogues (e.g., logical breaks, role confusion, or fictional content that does not conform to common sense), or those that are redundant or oversimplified. Qualified triplet samples can also be converted to a specific format (e.g., JSONL, XML). Qualified triplet samples can also be split into training and validation datasets according to a certain ratio. This application does not impose any limitations on the sampling ratio, the number of rounds of dialogue generation, the splitting ratio, or the number of generated samples.

[0059] It should be noted that by converting qualified triplet samples into a special format, memory can be saved, the independence of the samples can be guaranteed, errors in the entire file due to format errors can be avoided, and the samples can be standardized without additional conversion.

[0060] In some implementations, the first pruning model can be trained using SFT (Supervised Fine-Tuning). This involves training the model with a training dataset and preset first training parameters, enabling it to initially grasp the basic rules of "preserving core information and adding structured labels." It should be understood that the first training parameters may include, but are not limited to, training batch size, learning rate, training epochs, and the selected loss function. These parameters can be set according to requirements and are not limited herein.

[0061] For example, set the batch size to 4 and the learning rate to 2e in the first training parameter. -5 The training rounds are 3, the loss function is cross-entropy loss, and the first pruning model is trained for 3 rounds using the training dataset and the first training parameters to obtain the trained first pruning model.

[0062] In some implementations, the first pruning model after training can be used as the second pruning model, or the first pruning model after retraining can be used as the second pruning model.

[0063] In some implementations, after training the first pruning model, validation samples can be input into the trained first pruning model to obtain the output results corresponding to the validation samples. If the accuracy of the optimal retained content in the output results reaches a preset threshold, the training is considered successful. At this time, the trained first pruning model can be used as the second pruning model. The preset threshold can be any suitable value, which is not limited in this application. If the accuracy of the optimal retained content in the output results does not reach the preset threshold, the training is considered unsuccessful and retraining is required. During retraining, emphasis should be placed on reducing redundant statements.

[0064] For example, the original content in the validation sample is "Customer Service: Hello! User: Hello. Customer Service: How can I help you? User: I want to unsubscribe from your Gold Membership. Customer Service: Okay, is it effective immediately? User: Yes, I'll unsubscribe now." The optimal retained content with tags is "<Retain> User: I want to unsubscribe from your Gold Membership. <Retain>, <Retain> User: Yes, I'll unsubscribe now. <Retain>", and the core semantic summary is "Unsubscribe from Gold Membership, Effective Immediately".

[0065] When this validation sample is input into the first pruning model, the labeled optimal retained content in the output is "<Retain> User: I want to unsubscribe from your Gold Membership. <Retain>, <Retain> Customer Service: Okay, is it effective immediately? <Retain>, <Retain> User: Yes, unsubscribe now. <Retain>", meaning that the output contains redundant statements, and the accuracy between the output and the labeled optimal retained content in the validation set is less than the preset threshold (e.g., 70%). Therefore, the training at this point has not been successful.

[0066] In some implementations, the second pruning model is trained using reinforcement learning. The reinforcement learning training can employ the veRL framework, the TRL (Transformer Reinforcement Learning) framework, or any other suitable learning framework, which is not limited in this application.

[0067] In some implementations, when training the second pruning model using reinforcement learning, a GSPO (Generalized Safe Policy Optimization) strategy, a GRPO (Group Relative Policy Optimization) strategy, a PPO (Proximal Policy Optimization) strategy, or any suitable strategy can be used, which is not limited in this application.

[0068] In some implementations, the second pruning model is trained using a preset reinforcement learning framework, training strategy, preset multi-dimensional reward function, and second training parameters until a preset stopping condition is met, thus obtaining the pruned model. The multi-dimensional reward function is used to evaluate the model output during reinforcement learning. It should be understood that the preset second training parameters may include, but are not limited to, LoRA (Low-Rank Adaptation), learning rate, and training epochs. The second training parameters can be set according to requirements and are not limited herein.

[0069] In some implementations, the second pruning model can be trained using samples (such as samples in the training dataset, samples in the validation set, etc.), a multi-dimensional reward function, and a second training parameter. After training, the training effect of the second pruning model can be verified. For example, the validation samples are input into the trained second pruning model to obtain the output results of the validation samples. If at least one of the following is satisfied: the accuracy of the optimal retained content in the output results reaches a preset accuracy threshold, the obtained reward value is greater than or equal to a preset reward threshold, or the length compliance rate of the optimal retained content reaches a preset compliance threshold, then training is stopped, and the trained second pruning model is used as the final pruning model. The preset accuracy threshold, preset reward threshold, and preset length threshold can be set according to requirements, and are not limited in this application.

[0070] For example, the second pruning model has a LoRA rank of 16 and a learning rate of 1e. -4The second training parameters are trained for 5 training epochs. After training, the validation samples are input into the second pruning model. The optimal retained content in the output is then compared with the optimal retained content of the validation samples. It is found that: the accuracy of the output is less than the preset accuracy threshold, the reward value of the output is less than the preset reward threshold, and the length compliance rate of the optimal retained content reaches the preset compliance threshold. Although the length compliance rate of the optimal retained content reaches the preset compliance threshold, the accuracy and reward value do not meet the conditions. Therefore, the second pruning model needs to continue training with LoRA rank 16 and a learning rate of 1e. -4 The training parameters are trained in 5 training rounds, and the internal parameters (such as the weight values ​​of the neural network) are adjusted based on the output results of the validation sample.

[0071] For example, the second pruning model has a LoRA rank of 16 and a learning rate of 1e. -4 The second training parameter is trained for 5 training rounds. After training, the validation sample is input into the second pruning model. At this time, the optimal retained content in the output result is compared with the optimal retained content of the validation sample. It can be seen that: the accuracy of the output result is greater than the preset accuracy threshold, the reward value of the output result is greater than the preset reward threshold, and the length compliance rate of the optimal retained content reaches the preset compliance threshold. At this time, the length compliance rate of the optimal retained content reaches the preset compliance threshold, the optimal retained content reaches the preset accuracy threshold, and the reward value is greater than or equal to the preset reward threshold. Therefore, the second pruning model meets the compliance conditions and training can be stopped.

[0072] In practice, the training process of this cropping model can be as follows: Figure 3 As shown, steps S31 to S36 are included, wherein: Step S31, construct the training dataset; Step S32: Input the training samples into the first cropping model; Step S33: Supervised training is performed on the first cropping model; Step S34: Use the first cropping model after training as the second cropping model; Step S35: Perform reinforcement learning training on the second pruning model based on the multi-dimensional reward function; Step S36: Use the trained second cropping model as the cropping model.

[0073] In this embodiment, by introducing a two-stage training mechanism that combines supervised training and reinforcement learning, the second pruning model can quickly master the basic pruning rules and continuously optimize the strategy through a reward feedback mechanism. This enables precise pruning of information, thereby balancing text length control and semantic preservation, and ultimately improving semantic coherence and response quality in multi-turn dialogues.

[0074] In some implementations, "training the second pruning model using reinforcement learning based on a multi-dimensional reward function to obtain the pruning model" includes: inputting samples into the second pruning model to obtain the key prediction information of the samples; determining the reward value of the samples based on the multi-dimensional reward function and the key prediction information of the samples; and updating the parameters of the second pruning model at least once based on the reward value of the samples to obtain the pruning model.

[0075] Here, the sample can be a sample from the training dataset or the validation dataset. This sample can be a triple sample, which can contain the original dialogue, labeled optimal retained content, and core semantic summary. It should be noted that the original dialogue can be a multi-turn dialogue with redundancy; the labeled optimal retained content is the optimal content retained after pruning; and the core semantic summary is a highly condensed keyword summary of key information.

[0076] The key information for this prediction may include the predicted labeled optimal retention content and the core semantic summary of the prediction.

[0077] In some implementations, the sample is input into the second pruning model to output predicted key information, and the reward value of the sample is determined based on the predicted key information, the key information of the sample, the preset minimum length, etc.

[0078] In some implementations, the reward value of the sample can be determined based on at least two of a first reward value, a second reward value, a third reward value, and a fourth reward value. This application does not limit the method of determining the sample reward value. Specifically, the first reward value is determined based on the predicted key information and the key information of the sample; the second reward value is determined based on the length of the predicted key information; the third reward value is determined based on the length of the predicted key information and a preset minimum length; and the fourth reward value is determined based on the tagging information in the predicted key information.

[0079] For example, the reward value of a sample can be determined based on a first reward value and a second reward value. Alternatively, the sum, average, or standard deviation of the first and second reward values ​​can be used as the reward value for the sample. Another example is using a weighted sum of the first and second reward values ​​as the reward value for the sample.

[0080] For example, the reward value of a sample can be determined based on the first reward value, the second reward value, and the third reward value.

[0081] For example, the sample reward value can be determined based on the first reward value, the second reward value, the third reward value, and the fourth reward value.

[0082] In some implementations, the parameters of the second pruning model are iteratively updated based on the reward value of the sample until a preset stopping condition is met. The parameters of the second pruning model can be adjustable learnable parameters within the pruning model, and these learnable parameters (such as the weights of a neural network) are adjusted according to the reward value of the sample.

[0083] In this embodiment, on the one hand, the reinforcement learning training of the second pruning model based on a multi-dimensional reward function effectively replaces the traditional evaluation mechanism that relies on a single indicator; on the other hand, the parameters of the second pruning model are dynamically updated based on the reward value during the training process, thereby dynamically adjusting the pruning strategy of the model, quantitatively evaluating the training effect of the second pruning model, and ensuring that the pruning model obtained after training achieves the expected performance.

[0084] In some implementations, "determining the reward value of a sample based on a multi-dimensional reward function and the sample's predicted key information" includes: determining a first reward value based on the sample's predicted key information and the sample's key information; determining a second reward value based on the length of the sample's predicted key information; determining a third reward value based on the length of the sample's predicted key information and a preset minimum length; determining a fourth reward value based on the labeling information in the sample's predicted key information; and determining the sample's reward value based on the first, second, third, and fourth reward values.

[0085] Here, the first reward value can be determined based on the semantic completeness R3 and / or semantic accuracy R1 of the predicted key information.

[0086] The key information for sample prediction includes the optimal content to be retained in the output of the second pruning model after the input sample is fed into it.

[0087] In some implementations, semantic accuracy R1 can be determined by the character matching degree between the predicted optimal retained content and the optimal retained content of the sample. The character matching degree can be the ratio between a first number of characters and a second number of characters, ranging from 0 to 1, where the first number of characters is the number of matching characters between the predicted optimal retained content and the optimal retained content of the sample, and the second number of characters is the number of characters in the predicted optimal retained content. Alternatively, the character matching degree can be the ratio between a first sum and a second sum, ranging from 0 to 1, where the first sum is the weighted sum of each target key character in the predicted optimal retained content, and the second sum is the weighted sum of each target key character in the optimal retained content of the sample, with each target key character in the predicted optimal retained content corresponding to one target key character in the optimal retained content of the sample. It is understood that each key character in both the predicted optimal retained content and the optimal retained content of the sample can be pre-assigned corresponding weights. In practice, other suitable methods can also be used to calculate the character matching degree, which is not limited in this application.

[0088] In some implementations, the "optimal retained content of the original text" (i.e. the optimal retained content of the sample) is used as a benchmark. The number of matching characters between the marked retained content (i.e. the predicted optimal retained content) and the benchmark is calculated. The semantic accuracy R1 is determined based on the ratio of the number of matching characters to the number of characters in the retained content (i.e. the number of characters in the optimal retained content of the sample). If there is no corresponding benchmark segment for the retained content, the number of matching characters is counted as 0. The value range is from 0 to 1.

[0089] In some implementations, semantic integrity R3 can be determined based on a first similarity between the predicted optimal retained content and the optimal retained content of the sample, with a value ranging from 0 to 1. For example, the first similarity, a weighted average of the first similarity, etc., can be used as semantic integrity.

[0090] In some implementations, semantic integrity R3 can be determined based on a second similarity between the core semantic summary of a sample and the predicted core semantic summary, with a value ranging from 0 to 1. For example, the second similarity, a weighted average of the second similarity, etc., can be used as semantic integrity R3.

[0091] The first reward value can be determined in any suitable way. For example, semantic completeness or a weighted average of semantic completeness can be used as the first reward value. Another example is semantic accuracy or a weighted average of semantic accuracy. Yet another example is a weighted sum or mean of semantic accuracy and semantic completeness.

[0092] For example, in a customer service scenario, the optimal content to be retained in a sample includes keywords such as membership level, upgrade method, and price difference. If the predicted optimal content to be retained in a sample can accurately cover these keywords and maintain the integrity of the original text (i.e., the first similarity or second similarity is higher than the set similarity threshold), then a higher first reward value can be obtained.

[0093] The length of the key information predicted for a sample can refer to the length of the optimal content to be retained in the prediction.

[0094] In some implementations, length compliance (i.e., the second reward value R2) can be determined by judging whether the length of the predicted optimal retained content exceeds the maximum window value of the second pruning model. For example, if the length of the predicted optimal retained content exceeds the maximum window value of the second pruning model, the length is non-compliant, and the second reward value R2 can be a preset value; if the length of the predicted optimal retained content does not exceed the maximum window value of the second pruning model, the second reward value can be determined based on the original dialogue. The preset value can be 0 or other suitable values, which are not limited in this application. For example, if the length of the predicted optimal retained content output by the second pruning model is 12000 tokens, and the maximum window value is 8192 tokens, then the length of the predicted optimal retained content is greater than the maximum window value, so the length is determined to be non-compliant, and the second reward value R2 can be 0.

[0095] The preset minimum value refers to the minimum retention length necessary to ensure semantic integrity. This preset minimum value can be 70%, 75%, etc., of the optimal retention content of the sample, and can also be dynamically adjusted according to needs (such as scenarios). For example, in a customer service scenario, the preset minimum value can be 2000 tokens, while in daily conversations, the preset minimum value can be set to 1000 tokens. This application does not impose any restrictions on this.

[0096] It should be noted that if the length of the predicted key information is less than the preset minimum length, it indicates that the predicted key information may have been over-pruned, resulting in the loss of core semantics and thus adversely affecting information understanding. Conversely, if the length of the predicted key information is much greater than the preset minimum length, it indicates that there may be redundant content in the predicted key information that has not been effectively removed.

[0097] The third reward value R4 can be determined in any suitable way.

[0098] In some implementations, the third reward value can be determined based on a first ratio between the first difference and the predicted optimal retention length. For example, the first ratio and a weighted average of the first ratio can be used as the third reward value R4. Here, the first difference is the difference between the predicted optimal retention length and a preset minimum value. In some implementations, the third reward value R4 can be determined based on a second ratio between the first difference and a preset minimum value. For example, the second ratio and a weighted average of the second ratio can be used as the third reward value R4.

[0099] In some implementations, the third reward value R4 can be determined based on a third ratio. For example, if the third ratio is greater than a preset value (e.g., 0, 0.1, etc.), then the third ratio is used as the third reward value; if the third ratio is not greater than the preset value, then the preset value is used as the third reward value. The third ratio is the ratio between the first difference and the maximum window value of the second pruning model.

[0100] In some implementations, the third reward value R4 can be determined using the following formula (1-1): (1-1); in, To predict the length of the optimal content to retain, To preset the minimum value, This is the maximum window size.

[0101] Understandably, the third reward value R4 ranges from 0 to 1.

[0102] The tagging information refers to the special tags added to the output by the second cropping model (such as <retain>...<retain>, etc.). <keep> …… <keep>(etc.) are used to clearly identify which content is retained. In practice, this application does not limit the specific form of the special marking.

[0103] In some implementations, the key information for the prediction of a sample includes the best-preserved content with tags. The format compliance (i.e., the fourth reward value R5) is determined by verifying whether the reserved special tags are compliant (e.g., whether the special tags are paired, whether they are preset special tags, whether there are special tags, etc.).

[0104] In some implementations, if the retained special marker format is compliant, the format compliance can be a first preset value; if it is not compliant, the format compliance can be a second preset value. The first preset value can be 1, and the second preset value can be 0. The first and second preset values ​​can also be other suitable values, which are not limited in this application. It is understood that the second preset value is different from the first preset value. For example, if there are unpaired special markers in the predicted key information of the sample, the fourth reward value R5 is determined to be 0. Or, for example, if there are no special markers in the predicted key information of the sample, the fourth reward value R5 is determined to be 0.

[0105] The reward value for this sample can be determined in any suitable way. The reward value of the sample serves as a feedback signal during reinforcement learning training, guiding the model to optimize its pruning strategy and enabling the model to continuously improve its performance through multiple rounds of training.

[0106] In some implementations, the reward value of the sample can be determined based on any two of the first reward value, the second reward value, the third reward value, and the fourth reward value. For example, the weighted sum or mean of any two can be used as the reward value of the sample.

[0107] In some implementations, the reward value of the sample can be determined based on any three of the first, second, third, and fourth reward values. For example, the weighted sum or mean of any three values ​​can be used as the reward value of the sample.

[0108] In some implementations, the sample reward value can be determined based on a first reward value, a second reward value, a third reward value, and a fourth reward value. For example, the weighted sum or mean of these four reward values ​​can be used as the reward value for the sample.

[0109] In some implementations, the reward value Rtotal of the sample can be calculated according to the following formulas (1-2), where: (1-2); in, , , , and These are the initial values ​​of the weights; The semantic accuracy of the predicted key information for the sample; To ensure compliance with the length of key information for sample prediction; To ensure the semantic integrity of key information for sample prediction; Redundancy of key prediction information for the sample; To ensure the compliance of the format of key information for predicting samples.

[0110] In one embodiment, the initial values ​​of each weight in the above formulas (1-2) can be adjusted as needed. If the trimming result exceeds the limit or the format is seriously incorrect, an additional penalty value needs to be added. For example, the penalty value can be -2, -3, etc., which is not limited in this application.

[0111] In this embodiment, a multi-dimensional reward function is designed to provide reward feedback to the model from aspects such as semantic accuracy, length compliance, semantic integrity, redundancy, and format compliance. This guides the second pruning model to continuously optimize the pruning strategy during reinforcement learning, maintaining semantic quality while taking into account simplicity and standardization, thereby continuously improving the pruning effect and generalization ability of the second pruning model.

[0112] In some implementations, "determining a second reward value based on the length of the predicted key information of the sample" includes: if the length of the predicted key information of the sample is greater than a length threshold, using a preset value as the second reward value; if the length of the predicted key information of the sample is not greater than the length threshold, determining a target pruning ratio based on the length of the sample; and determining a second reward value based on the target pruning ratio and the length of the predicted key information of the sample.

[0113] Here, the length threshold can refer to the maximum window value of the second cropping model, or it can be any other threshold that is set.

[0114] The preset value can be 0 or other suitable values, which are not limited in this application. For example, to determine whether the length of the predicted optimal retained content exceeds the maximum window value of the second pruning model (i.e., the length threshold), if the length of the predicted optimal retained content is greater than the maximum window value of the second pruning model, the length compliance of the predicted key information (i.e., the second reward value) can be 0. If the length of the predicted optimal retained content is not greater than the maximum window value of the second pruning model, the length compliance of the predicted key information can be further determined based on the length of the sample and the length of the predicted key information of the sample.

[0115] The length of a sample can refer to the length of the original dialogue in the sample.

[0116] The target pruning ratio is used to measure whether the current sample has been reasonably compressed, and its value ranges from 0 to 1. The target pruning ratio can be determined in any suitable way. In some implementations, a correspondence between each pruning ratio and each length can be pre-established, and based on this correspondence, a target pruning ratio adapted to the sample length can be obtained. In some implementations, the sample length can be compared with a length threshold. If the sample length is not greater than the length threshold, the target pruning ratio can be a set pruning ratio (e.g., 0.8 or other suitable values); if the sample length is greater than the length threshold, the target pruning ratio can be determined based on the ratio between the length threshold and the sample length, for example, using this ratio or a weighted average of the ratio as the target pruning ratio.

[0117] The second reward value R2 can be determined in any suitable way.

[0118] In some implementations, a correspondence between each pruning ratio, each length, and each second reward value can be established in advance. Based on this correspondence, a second reward value R2 that is adapted to both the target pruning ratio and the length of the predicted key information of the sample can be obtained.

[0119] In some implementations, the second reward value can be determined based on the difference between a third preset value and a first value. For example, this difference, a weighted average of the differences, etc., can be used as the second reward value. The third preset value can be 1 or other suitable values. The first value is determined based on the second difference, which is the difference between the target pruning ratio and a fourth ratio, which is the ratio between the length of the predicted key information of the sample and the length of the sample. For example, if the absolute value of the second difference is not greater than the fourth preset value, then the absolute value of the second difference is used as the first value; if the absolute value of the second difference is greater than the fourth preset value, then the fourth preset value is used as the first value. The fourth preset value can be 1 or other suitable values.

[0120] In some implementations, the second reward value R2 can be determined using the following formulas (1-3): (1-3); in, To predict the length of the optimal content to retain, The length of the original dialogue in the current sample. τ represents the maximum window size, and τ represents the target pruning ratio.

[0121] In this embodiment, on the one hand, by determining whether the length of the predicted optimal retained content exceeds the maximum window value of the second pruning model, a hard constraint is imposed on the legality of the input, thereby reducing the possibility that the optimal retained content output by the pruning model will exceed the limit. On the other hand, by dynamically obtaining the target ratio value, an optimal balance between the pruning length and the optimal retained content is achieved, improving pruning efficiency and enhancing the robustness of the pruning model.

[0122] The technical solution of this application will be described in detail below.

[0123] Current large language models (such as qwen3) have a fixed context window limit in dialogue interaction. When the accumulated context length from multiple rounds of dialogue exceeds the window limit, context pruning is required to ensure the model functions correctly. Existing context pruning techniques are mainly divided into three categories: First, rule-based pruning: based on the "time decay" principle, the latest round of dialogue is retained, or the content at the end is truncated based on a fixed length, such as directly deleting the earliest N rounds of dialogue; Second, supervised fine-tuning of pruning: By manually labeling "original dialogue - pruned dialogue" sample pairs, a dedicated pruning model is trained, enabling the model to learn fixed pruning rules; Third, simple semantic trimming: a text summarization model is used to compress the context, extracting long texts into short texts before inputting them into the large model.

[0124] The above pruning techniques have the following problems: First, there is a contradiction between semantic preservation and length control: rule-based pruning is prone to deleting key historical information (such as constraints proposed by users in the early stage), resulting in semantic breaks; simple semantic pruning often loses details in order to compress the length, while supervised fine-tuning pruning is prone to the problem of the length still exceeding the limit after pruning if it pursues semantic integrity. Second, cumulative semantic loss over multiple rounds: If the mode of "using the pruned content as the input for the next round" is adopted, the core semantics will gradually decay due to repeated pruning after multiple rounds of interaction. This can lead to context collapse during multiple rounds, such as the "membership level" information mentioned by the user for the first time being mistakenly deleted after 3 rounds. Third, poor adaptability: The existing methods have fixed pruning rules (such as keeping the latest 5 rounds), which cannot be dynamically adjusted according to the length of the conversation and the scenario (customer service / daily chat), resulting in over-pruning of short conversations and incomplete pruning of long conversations; Fourth, insufficient redundancy control: Supervised fine-tuning and pruning relies on manually labeled sample distribution, which makes it difficult to cover all redundant scenarios (such as repeated questions and piled-up interjections), resulting in a large amount of invalid content still remaining after pruning.

[0125] To address the above-mentioned deficiencies, this application proposes a dialogue processing method that achieves a balance between "strict length control" and "accurate semantics" through "dynamic reward guidance + cache mechanism guarantee", while adapting to the dynamic requirements of multi-turn dialogue scenarios.

[0126] The purpose of this application is to solve problems such as "the contradiction between semantic retention and length control", "loss of multi-turn semantics", and "poor adaptability" in existing context pruning technologies, and to provide a pruning method that can dynamically adapt to dialogue scenarios, strictly comply with window limits, and accurately retain core semantics, playing an efficient compression sub-agent in agent development.

[0127] The core concept of this application is: centered around a reinforcement learning framework, construct an "agent-environment" interaction model, design a multi-dimensional dynamic reward function to guide the model to explore the optimal pruning strategy, and搭配"original input + pruning cache" mechanism to ensure multi-turn semantic integrity. The specific technical solutions include the following three parts: The pruning framework driven by reinforcement learning includes the following: Define the core elements of reinforcement learning to achieve a closed-loop of "pruning action - reward feedback - policy optimization": 1. Agent: A pruner built based on a lightweight large model (such as qwen3-4B), with the original dialogue text as the input, and the pruning result with retention tags (<retention>< / retention>) as the output. The action space is to "retain / discard" each semantic unit in the dialogue. 2. Environment: Includes a dialogue input module, a length verification module, and a semantic evaluation module, providing an interaction scenario of "original dialogue + window limit (configurable as 1M / 3M tokens)", and feedbacking the length compliance and semantic integrity of the pruning result. 3. State: The original dialogue text of the current round, the content of the historical pruning cache, the type of dialogue scenario (customer service / daily / work), and the ratio of the current dialogue length to the maximum window. 4. Reward function (core innovation): Design a scenario-adaptive weighted combined reward function that integrates 5 dimensions to achieve multi-objective balance. The total reward formula is: 5. Integrate 5 dimensions to achieve multi-objective balance. The total reward formula is as shown in the above formula (1-2).

[0128] Among them, the "original input + pruning cache" interaction mechanism includes: To solve the problem of multi-turn semantic loss, design a mechanism of "using the original for input and caching the core", and the steps are as follows: First, after receiving a new dialogue turn, concatenate the "historical original dialogue + content of the new turn" as the input to the pruner to ensure that the agent obtains the full context. Second, after the trimmer outputs the results with the retention mark, the retained content is extracted and stored in the "trimming cache" (only the core is stored, and the original redundancy is not stored). Third, cache length verification: If the cache length is ≤ Lmax, it is directly retained and waited for the next round; if it exceeds the limit, a second pruning is triggered (the core content in the cache is further simplified, and the latest semantics are retained first) to ensure that the cache is always compliant.

[0129] Compared with related technologies, this application has the following significant advantages: 1. Achieving dual compliance of length control and semantic preservation: By balancing hard constraint penalties and semantic rewards, a window compliance rate of 100% and a core semantic retention rate of ≥90% are achieved, resolving the core contradiction of existing technologies; 2. Multi-turn semantic decay-free: The original input ensures that the agent grasps the full context, and the cache is pruned to retain only the core meaning, avoiding semantic loss caused by repeated pruning; 3. Strong dynamic adaptability: The pruning strategy is adjusted based on the dialogue length and scenario to adapt to different window requirements, such as 8192 tokens to 1M tokens, and its generalization ability is better than that of fixed rules; 4. Precise redundancy control: Excessive retention penalty items ensure redundancy removal accuracy of ≥85%, while avoiding over-pruning and ensuring dialogue continuity; 5. High engineering feasibility: Based on lightweight models (no larger than 4B level models) and mature reinforcement learning frameworks (TRL / VERL), LoRa training can be completed on a single RTX4090. The interface is encapsulated during deployment to adapt to various dialogue systems.

[0130] The system in this application includes a hardware layer and a software layer, and the specific architecture is as follows: First, the hardware layer: the training end uses a single RTX 4090 graphics card (24GB VRAM) or a larger GPU and a cluster. Second, the software layer: This includes a data preparation module, a reinforcement learning training module, a pruning and execution module, a cache management module, and an interface module. The functions of each module are as follows: Data preparation module: After deploying the pre-built open-source model, it generates sample pairs of "original dialogue - optimal retained content - core summary", and after cleaning, it splits the training set / validation set in a 9:1 ratio; Third, reinforcement learning training module: based on the VERL framework, it implements agent training and includes two stages: supervised pre-training (SL) and reinforcement learning fine-tuning (RL). Fourth, the trimming execution module: receives the original dialogue, calls the trained trimmer to output the results, and performs format verification by extracting special tokens through regularization; Fifth, the cache management module: stores the pruned content, performs length verification and secondary pruning; Sixth, the interface module: It encapsulates the core HTTP interface through FastAPI, providing the ability to "input the original dialogue - return the trimmed result".

[0131] The implementation steps are as follows: Step 1: Dataset Construction First, call the optimized prompt words to generate samples: request the large model to generate multi-turn dialogues (≥3 turns, including redundancy) for various scenarios through prompt words, and output "original dialogue + labeled pruning results (i.e. the above-mentioned labeled optimal retention content) + core summary (i.e. the above-mentioned core semantic summary)"; Second, sample size: 1000 samples were generated, including 400 customer service scenarios, 300 daily chats, and 300 work communications. Third, data cleaning: 20% of the samples are manually sampled, and samples with incorrect labeling (such as missing core information) or unrealistic dialogue are deleted. After correction, the data is converted to JSONL format.

[0132] Step 2: Reinforcement Learning Training 1. Supervised training (SFT stage): Base model selection: Qwen3-1.7B (32K window).

[0133] 2. Training parameters: batch size = 4, learning rate = 2e-5, training epochs = 3, loss function is cross-entropy loss.

[0134] 3. Objective: To enable the model to initially grasp the basic rules of "retaining the core + adding labels", and to proceed to the next stage if the core retention accuracy on the validation set is ≥70%.

[0135] 4. Reinforcement Learning Fine-tuning (RL Phase): Framework and Policy: The VERL framework is used, and the training policy is GSPO.

[0136] 5. Reward Function Integration: The multi-dimensional reward function designed in this application is encapsulated as Reward_fn and passed into the training process.

[0137] 6. Training parameters: LoRA rank r=16, learning rate=1e-4, training epochs=5, evaluation using the validation set after each epoch.

[0138] 7. Stop conditions: Total verification reward value ≥ 0.8, window compliance rate = 100%, core semantic retention rate ≥ 90%.

[0139] Step 3: Pruning and Cache Management Taking a customer service dialogue scenario (maximum window Lmax = 32768 tokens) as an example, the execution flow is as follows: First, receive a new conversation: The user inputs "I previously subscribed to the monthly membership and want to upgrade to the annual membership. How much is the price difference? How can I pay?", and splice the historical original conversation ("User: What types of memberships are there? Customer service: There are two types, monthly at 29 yuan and annual at 199 yuan"); Second, Trimmer processing: Input the spliced original conversation, and the model outputs the marked result: "<Keep>User: What types of memberships are there? Customer service: There are two types, monthly at 29 yuan and annual at 199 yuan< / Keep>User: I previously subscribed to the monthly membership and want to upgrade to the annual membership. How much is the price difference? How can I pay?<Keep>Customer service: The price difference is 170 yuan. Pay in the "My - Membership Center" in the APP< / Keep>"; Third, extraction and caching: Extract the kept content and store it in the cache. The cache length is 320 tokens (≤8192), and no secondary trimming is required; Fourth, output the result: Return the trimmed text with marks, or return the pure core content without marks according to requirements.

[0140] If the subsequent conversation accumulates and causes the cache length to reach 8200 tokens, trigger secondary trimming: The Trimmer further streamlines the content in the cache, deletes the repeated expressions in "Customer service: There are two types, monthly at 29 yuan and annual at 199 yuan", and keeps "Customer service: Monthly at 29 yuan, annual at 199 yuan", reducing the cache length to 7800 tokens.

[0141] The execution process for training and validating the logical feasibility is as follows: 1. Sufficient dataset support: Construct a sample set of "original conversation - optimal kept content - core summary - exact match benchmark text" to provide a comprehensive data foundation for model learning; 2. Progressive training process: In the SFT stage, the model masters the basic rules. In the RL stage, the policy is optimized through a dynamic reward function. The set stop conditions (total reward ≥ 0.8, compliance rate 100%, etc.) can quantitatively evaluate the model effect to ensure that the model meets the expected performance after training; 3. Module coordination verification: The process connection between the character matching verification module and the trimming execution and cache management modules is clear. After passing the verification, cache storage is performed, which can prevent character deviation content from entering the subsequent process and logically ensure the output quality.

[0142] Based on the above embodiments, an embodiment of the present application provides a dialogue processing device, Figure 4 is a schematic diagram of the composition structure of a dialogue processing device provided by an embodiment of the present application, as Figure 4 shown. The dialogue processing device 40 includes a determination module 41 and a trimming module 42, where: The determination module 41 is used to determine the first information based on the current input information; The trimming module 42 is used to trim the first information using a trimming model to obtain the second information. The second information represents the key information of the first information. The second information is used as input to a preset dialogue model to generate the response information corresponding to the current input information. The trimming model is trained by reinforcement learning using a preset multi-dimensional reward function. The multi-dimensional reward function is used to update the parameters of the trimming model to be trained at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the trimmed key information. The second reward value represents the length compliance of the trimmed key information. The third reward value represents the redundancy of the trimmed key information. The fourth reward value represents the format compliance of the trimmed key information.

[0143] In some embodiments, the trimming module 42 is further configured to trim the first information using a trimming model to obtain the second information if the length of the first information is greater than a preset length threshold.

[0144] In some implementations, the determining module 41 is further configured to determine first information based on the current input information and the at least one round of historical dialogue information, if it is determined that at least one round of historical dialogue information exists.

[0145] In some embodiments, the dialogue processing device 40 further includes a generation module 43, which is used to generate new historical dialogue information based on the current input information, the response information corresponding to the current input information, and at least one round of historical dialogue information.

[0146] In some embodiments, the dialogue processing device 40 further includes a training module 44, which is used to train a first pruning model based on a training dataset to obtain a second pruning model; and to perform reinforcement learning training on the second pruning model based on a multi-dimensional reward function to obtain a pruning model.

[0147] In some implementations, the training module 44 is further configured to input samples into a second pruning model to obtain key prediction information of the samples; determine the reward value of the samples based on the multi-dimensional reward function and the key prediction information of the samples; and update the parameters of the second pruning model at least once based on the reward value of the samples to obtain the pruning model.

[0148] In some implementations, the training module 44 is further configured to: determine a first reward value based on the predicted key information of the sample and the key information of the sample; determine a second reward value based on the length of the predicted key information of the sample; determine a third reward value based on the length of the predicted key information of the sample and a preset minimum length; determine a fourth reward value based on the labeling information in the predicted key information of the sample; and determine the reward value of the sample based on the first reward value, the second reward value, the third reward value and the fourth reward value.

[0149] In some implementations, the training module 44 is further configured to: use a preset value as a second reward value when the length of the predicted key information of the sample is greater than a length threshold; determine a target pruning ratio based on the length of the sample when the length of the predicted key information of the sample is not greater than the length threshold; and determine a second reward value based on the target pruning ratio and the length of the predicted key information of the sample.

[0150] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0151] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0152] This application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements any of the methods described above. The computer-readable storage medium can be transient or non-transient.

[0153] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0154] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0157] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, each functional unit in the embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0159] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0160] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0161] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.< / keep> < / keep>

Claims

1. A dialogue processing method, comprising: Based on the current input information, determine the first piece of information; The first information is cropped using a cropping model to obtain the second information. The second information represents the key information of the first information. The second information is used as the input of a preset dialogue model to generate the response information corresponding to the current input information. The pruning model is trained using a preset multi-dimensional reward function through reinforcement learning. The multi-dimensional reward function is used to update the parameters of the pruning model at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the pruned key information, the second reward value represents the length compliance of the pruned key information, the third reward value represents the redundancy of the pruned key information, and the fourth reward value represents the format compliance of the pruned key information.

2. The method according to claim 1, wherein cropping the first information using a cropping model to obtain the second information comprises: If the length of the first information is greater than a preset length threshold, the first information is cropped using the cropping model to obtain the second information.

3. The method according to claim 1, wherein determining the first information based on the current input information includes: If it is determined that there is at least one round of historical dialogue information, the first information is determined based on the current input information and the at least one round of historical dialogue information.

4. The method according to claim 3, further comprising: Based on the current input information, the response information corresponding to the current input information, and the historical dialogue information of at least one round, new historical dialogue information is generated.

5. The method according to any one of claims 1 to 4, further comprising: Based on the training dataset, the first cropping model is trained to obtain the second cropping model; Based on the multi-dimensional reward function, the second pruning model is trained using reinforcement learning to obtain the pruning model.

6. The method according to claim 5, wherein training the second pruning model using reinforcement learning based on the multi-dimensional reward function to obtain the pruning model comprises: The sample is input into the second cropping model to obtain the key prediction information of the sample; Based on the multi-dimensional reward function and the key prediction information of the sample, the reward value of the sample is determined; Based on the reward value of the sample, the parameters of the second cropping model are updated at least once to obtain the cropping model.

7. The method according to claim 6, wherein determining the reward value of the sample based on the multi-dimensional reward function and the key prediction information of the sample comprises: Based on the prediction key information of the sample and the key information of the sample, the first reward value is determined; The second reward value is determined based on the length of the predicted key information of the sample; The third reward value is determined based on the length of the predicted key information of the sample and the preset minimum length; The fourth reward value is determined based on the labeling information in the prediction key information of the sample; The reward value of the sample is determined based on the first reward value, the second reward value, the third reward value, and the fourth reward value.

8. The method according to claim 7, wherein determining the second reward value based on the length of the predicted key information of the sample comprises: If the length of the predicted key information of the sample is greater than the length threshold, the preset value will be used as the second reward value. If the length of the predicted key information of the sample is not greater than the length threshold, the target pruning ratio is determined based on the length of the sample. The second reward value is determined based on the target pruning ratio and the length of the predicted key information of the sample.

9. A dialogue processing apparatus, comprising: The determination module is used to determine the first information based on the current input information; The trimming module is used to trim the first information using a trimming model to obtain second information. The second information represents the key information of the first information. The second information is used as input to a preset dialogue model to generate response information corresponding to the current input information. The pruning model is trained using a preset multi-dimensional reward function through reinforcement learning. The multi-dimensional reward function is used to update the parameters of the pruning model at least once. The multi-dimensional reward function includes at least two of the following: a first reward value, a second reward value, a third reward value, and a fourth reward value. The first reward value represents the semantic completeness and semantic accuracy of the pruned key information, the second reward value represents the length compliance of the pruned key information, the third reward value represents the redundancy of the pruned key information, and the fourth reward value represents the format compliance of the pruned key information.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 8.

Images