Large model intent understanding fine-tuning method and device based on reinforcement learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILEHANGZHOUINFORMATION TECH CO LTD
- Filing Date
- 2026-03-06
- Publication Date
- 2026-07-03
Smart Images
Figure CN121787595B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for fine-tuning large model intent understanding based on reinforcement learning. Background Technology
[0002] Traditional Natural Language Understanding (NLU) mainly relies on rule engines and statistical machine learning models to achieve intent recognition, and has achieved certain results in specific application scenarios (such as home robots, smart home control, etc.).
[0003] With the rapid development of Large Language Models (LLMs), supervised fine-tuning (SFT) has gradually replaced traditional natural language understanding techniques. This method fully leverages the superior semantic understanding capabilities of pre-trained language models, enabling large models to quickly adapt to domain-specific intent recognition tasks through fine-tuning on a small amount of high-quality labeled data. However, this method suffers a significant drop in recognition accuracy when dealing with complex semantics or intent expressions with insufficient training data, particularly in open scenarios and multi-turn dialogue environments. Furthermore, domain-specific fine-tuning may cause the model to forget general knowledge gained during pre-training, thus reducing its adaptability and robustness in other tasks.
[0004] Therefore, improving the accuracy of large-scale model execution intent understanding tasks has become a pressing technical problem for the industry. Summary of the Invention
[0005] This application provides a method and apparatus for fine-tuning large model intent understanding based on reinforcement learning, which addresses the technical problem of how to improve the accuracy of large models performing intent understanding tasks.
[0006] This application provides a method for fine-tuning large-scale model intent understanding based on reinforcement learning, including:
[0007] The current input is processed based on a large model to generate a local sample set containing multiple candidate responses;
[0008] The reward value for each candidate response is determined based on the intent matching reward value and slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response; the slot matching reward value is determined based on the importance level of each slot and the fill value matching degree of each slot.
[0009] The mean reward of the local sample group is determined based on the reward value of each candidate response, and the relative advantage value of each candidate response is determined based on the difference between the reward value of each candidate response and the mean reward.
[0010] The parameters of the large model are updated based on the relative advantage values of each candidate response.
[0011] In some embodiments, the slot matching reward value is determined based on the following steps:
[0012] The weight of each slot is determined based on its importance level.
[0013] The matching degree of the filling value of each slot is determined based on at least one of the exact matching strategy, the string similarity matching strategy, and the numerical fault-tolerant matching strategy.
[0014] Based on the weight of each slot and the matching degree of the fill value of each slot, the matching reward value of each slot is determined.
[0015] The matching reward value for each slot is determined based on the matching reward value for each slot.
[0016] In some embodiments, the method further includes:
[0017] If the predicted filling value for any slot is null and the actual filling value is not null, the matching reward value for any slot is deducted from the matching reward value for that slot.
[0018] If the predicted fill value for any slot is non-null and the actual fill value is null, determine the deduction ratio coefficient for any slot based on the importance level of the slot, determine the matching reward value deduction for any slot based on the deduction ratio coefficient and the matching reward value of the slot, and deduct the matching reward value deduction for any slot from the slot matching reward value.
[0019] If both the predicted and actual filling values for any slot are empty, the matching reward value for that slot will not be deducted from the slot matching reward value.
[0020] In some embodiments, the method further includes:
[0021] An initial corpus containing multi-turn dialogues is generated based on a pre-set large model;
[0022] The initial corpus is annotated with intent tags, slot fill values, and dialogue context to generate a training dataset;
[0023] The current input is determined based on the dialogue context in the training dataset, and the intent label and slot fill value in the dialogue context are determined as the true intent value and true fill value of the current input.
[0024] In some embodiments, updating the parameters of the large model based on the relative advantage values of each candidate response includes:
[0025] The policy gradient of each candidate response is updated based on the relative advantage value of each candidate response;
[0026] The parameters of the large model are updated based on the updated policy gradient.
[0027] In some embodiments, the candidate response includes a reasoning label for the reasoning process and an answer label for the output result.
[0028] In some embodiments, before determining the reward value of each candidate response based on the intent matching reward value and slot matching reward value of each candidate response, the method further includes:
[0029] The completeness of the thinking tags and answer tags for each candidate response is verified.
[0030] This application provides a large-scale model intent understanding fine-tuning device based on reinforcement learning, comprising:
[0031] The sample group generation module is used to process the current input based on the large model and generate local sample groups containing multiple candidate responses.
[0032] The reward value determination module is used to determine the reward value of each candidate response based on the intent matching reward value and the slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response; the slot matching reward value is determined based on the importance level of each slot and the fill value matching degree of each slot.
[0033] The advantage value determination module is used to determine the mean reward of the local sample group based on the reward value of each candidate response, and to determine the relative advantage value of each candidate response based on the difference between the reward value of each candidate response and the mean reward.
[0034] The parameter update module is used to update the parameters of the large model based on the relative advantage values of each candidate response.
[0035] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned reinforcement learning-based large model intent understanding fine-tuning method.
[0036] This application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned large model intent understanding fine-tuning method based on reinforcement learning.
[0037] The large-scale model intent understanding fine-tuning method and apparatus based on reinforcement learning provided in this application generate local sample groups containing multiple candidate responses. Within each local sample group, reward values are compared to determine the relative advantage value of each candidate response. The relative advantage value is used for adaptive policy gradient update, which effectively reduces the variance of the reward estimate. It does not rely on an independent value network that requires additional training, thus avoiding the instability problems that may be caused by building an independent value network. Combined with a refined reward function specifically designed for intent understanding tasks, which includes importance levels and multi-dimensional matching strategies, the large-scale model can more accurately identify intents and slots in multi-turn dialogues and open scenarios. This improves the accuracy and generalization ability of the large-scale model in performing intent understanding tasks, and fully meets the requirements of smart home and robot systems for rapid response and multi-task processing. Attached Figure Description
[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0039] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a flowchart illustrating the instruction-supervised fine-tuning training method provided in this application.
[0041] Figure 2 This is a flowchart illustrating the direct preference optimization training method provided in this application.
[0042] Figure 3 This is one of the flowcharts illustrating the large model intent understanding fine-tuning method based on reinforcement learning provided in this application.
[0043] Figure 4 This is the second flowchart of the large model intent understanding fine-tuning method based on reinforcement learning provided in this application.
[0044] Figure 5 This is a schematic diagram of the structure of the large model intent understanding fine-tuning device based on reinforcement learning provided in this application.
[0045] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0046] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0047] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.
[0048] Figure 1 This is a flowchart illustrating the instruction-supervised fine-tuning training method provided in this application, as shown below. Figure 1 As shown, the method includes:
[0049] Step 110: Construct single-sample pairs from the original intent data to form input-output pairs;
[0050] Step 120: Train the model using the cross-entropy loss function. The goal of the training is to maximize the log probability of the correct label.
[0051] Step 130: The trained model can effectively process the input data and output the expected intent recognition results.
[0052] In the fine-tuning of large models, reinforcement learning (RL) methods have been gradually introduced to make the output of large models closer to human expectations and to address the shortcomings of traditional supervised learning in terms of generalization and robustness. Direct Preference Optimization (DPO) has been proposed as an emerging reinforcement learning fine-tuning strategy. The DPO method optimizes the model directly by comparing the probability distributions of different candidate outputs under the same input, without the need to build and train a separate reward model, thus greatly simplifying the overall training process and reducing the complexity of hyperparameter tuning.
[0053] Figure 2 This is a flowchart illustrating the direct preference optimization training method provided in this application, as shown below. Figure 2 As shown, the method includes:
[0054] Step 210: Perform initial inference on the original intent data through the model, and use a confidence threshold screening mechanism (e.g., set the confidence threshold to 0.85) to separate high-confidence error samples from valid positive samples. The screening of hard negative samples further introduces semantic similarity calculation (e.g., set the cosine similarity to less than 0.6) to capture abnormal responses with semantic deviation.
[0055] Step 220: In the data construction phase, the system automatically constructs dynamic matching rules for positive and negative samples. For each input command, a manually labeled golden response is extracted from the positive sample pool as a benchmark. Simultaneously, 3 to 5 negative samples with high-confidence error features or slot filling bias are selected from the negative sample pool to generate a structured preference pair <input command, positive sample, negative sample>. An adversarial sample injection mechanism is specifically designed in this process, generating ambiguous commands by randomly masking 20% of the slot filling values to enhance the robustness of the model.
[0056] Step 230: In the model optimization stage, the reference model freezing technique is adopted, and the output model of the instruction-supervised fine-tuning stage is used as a fixed reference policy. The policy gradient is updated by maximizing the relative probability of the preference pair.
[0057] In order to extend and improve methods such as direct preference optimization, Figure 3 This is one of the flowcharts illustrating the large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application, such as... Figure 3 As shown, the method includes steps 310, 320, 330 and 340.
[0058] Step 310: Process the current input based on the large model to generate a local sample group containing multiple candidate responses.
[0059] Specifically, the execution subject of the large model intent understanding fine-tuning method provided in this application embodiment is a large model intent understanding fine-tuning device or system. This device can be implemented in software, such as a large model intent understanding fine-tuning program running in a computer; or it can be implemented in hardware, such as a smart home device (e.g., a home robot, smart speaker), personal computing device (e.g., a smartphone, personal computer), vehicle system, or cloud server that executes the large model intent understanding fine-tuning method.
[0060] In this application embodiment, the large model can be any type of large language model, such as a model with a small number of parameters designed to adapt to the deployment of edge devices, or a model with a large number of parameters deployed on a cloud server.
[0061] The current input refers to the text that requires the large model to understand the intent. This text can come from the user's voice command conversion, instant message input, etc.
[0062] The response refers to the output of the large model after processing the current input. When the large model receives the current input, it does not generate just one optimal response, but rather generates multiple different candidate responses with a certain degree of diversity through a specific decoding strategy. These multiple candidate responses form a local sample group corresponding to the current input.
[0063] Step 320: Determine the reward value of each candidate response based on the intent matching reward value and slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response; the slot matching reward value is determined based on the importance level of each slot and the matching degree of the fill value of each slot.
[0064] Specifically, after obtaining a local sample group, the quality of each candidate response within the group needs to be evaluated, and the evaluation result is the reward value. The reward value is a scalar value used to quantify how close the candidate response is to the ground truth. In the embodiments of this application, the reward value consists of two parts: intent matching reward value and slot matching reward value.
[0065] The intent matching reward is determined based on the intent matching degree of each candidate response. The intent matching degree measures the degree of match between the predicted intent value identified in the candidate response and the actual intent value corresponding to the current input.
[0066] In one specific embodiment, the intent matching degree can be binary, that is, the intent matching degree is 1 when the intent prediction value in the candidate response is completely consistent with the actual intent value corresponding to the current input, and 0 otherwise.
[0067] In another specific embodiment, a complete knowledge system containing 36 standard intents and over 40 semantic slots can be defined. A thesaurus of 127 natural language variants containing the 36 standard intents is constructed. Using the standard intents as the true intent values, the intent matching degree can be set to 1 when the predicted intent value perfectly matches the true intent value; it can be set to 0.8 when the predicted intent value matches a synonym in the thesaurus; and it can be set to 0 in all other cases.
[0068] In yet another specific embodiment, a continuous value between 0 and 1 can also be assigned based on the semantic similarity between the predicted intent value and the actual intent value.
[0069] The intent matching reward value can be the intent matching degree multiplied by a preset weighting coefficient.
[0070] The slot matching reward value reflects a refined evaluation of structured information extraction. A slot defines the key information the system needs to collect to recognize user intent, such as time, location, and event. Fill values refer to the information filled in to translate user intent into explicit instructions.
[0071] In this embodiment, not all slots are equally important. Slots can be divided into different importance levels according to business logic, such as core slots, auxiliary slots, and optional slots. When calculating the total slot matching reward value, different weights are assigned to slot matching scores of different levels, so that the model prioritizes learning to satisfy the extraction of core slots.
[0072] For each slot, a multi-level matching strategy can be used to determine the matching degree of the fill value. Multi-level matching strategies include exact matching strategies, string similarity matching strategies, and numerical error-tolerant matching strategies.
[0073] The slot matching reward value can be determined by combining the importance level of each slot and the matching degree of the fill value of each slot. For example, the weight of each slot is determined by the importance level, and the matching degree of the fill value of each slot is weighted and summed according to the weight to obtain the slot matching reward value.
[0074] Ultimately, the total reward value for a candidate response can be determined by summing the intent matching reward value and the slot matching reward value.
[0075] Step 330: Determine the mean reward of the local sample group based on the reward value of each candidate response, and determine the relative advantage value of each candidate response based on the difference between the reward value of each candidate response and the mean reward.
[0076] Specifically, the mean reward of a local sample group refers to the arithmetic mean of the reward values of all candidate responses within that local sample group. This mean can be regarded as a dynamic benchmark, reflecting the average performance level that the current large model can achieve when processing that specific input.
[0077] The difference between the reward value of each candidate response and the mean reward can be used as the relative advantage value of each candidate response. If the reward value of a candidate response is higher than the mean reward, its relative advantage value is positive, indicating that it is a good response that is better than average. If the reward value of a candidate response is lower than the mean reward, its relative advantage value is negative, indicating that it is a poor response that is worse than average.
[0078] In one specific embodiment, the relative advantage value can be expressed by the formula: .in, For the first The relative advantage value of each candidate response. For the first The reward value for each candidate response. This represents the number of candidate responses in the local sample group. For the average reward, The standard deviation is denoted as .
[0079] Step 340: Update the parameters of the large model based on the relative advantage values of each candidate response.
[0080] Specifically, for candidate responses with positive relative advantage values, the model parameters are adjusted to increase the probability that the large model will generate that response or a similar response in the future. For candidate responses with negative relative advantage values, the model parameters are adjusted to decrease the probability that the large model will generate that response or a similar response in the future.
[0081] In this way, large models are incentivized to explore and adopt policies that produce responses higher than their current average level, thereby achieving self-improvement and performance enhancement. In specific implementations, a clipping objective function similar to that in Proximal Policy Optimization (PPO) can be used, or a Kullback-Leibler Divergence (KL divergence) penalty term can be incorporated to constrain the step size of policy updates, preventing drastic changes in model parameters and ensuring the stability of the fine-tuning process.
[0082] The large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application generates local sample groups containing multiple candidate responses. Within each local sample group, reward values are compared to determine the relative advantage value of each candidate response. The relative advantage value is used for adaptive policy gradient update, which effectively reduces the variance of the reward estimate. It does not rely on an independent value network that requires additional training, thus avoiding the instability problems that may be caused by building an independent value network. Combined with a refined reward function specifically designed for intent understanding tasks, which includes importance levels and multi-dimensional matching strategies, the large model can more accurately identify intents and slots in multi-turn dialogues and open scenarios. This improves the accuracy and generalization ability of the large model in performing intent understanding tasks, fully meeting the requirements of smart home and robot systems for rapid response and multi-task processing.
[0083] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.
[0084] In some embodiments, the slot matching reward value is determined based on the following steps:
[0085] The weight of each slot is determined based on its importance level.
[0086] The matching degree of the filling value of each slot is determined based on at least one of the exact matching strategy, the string similarity matching strategy, and the numerical fault-tolerant matching strategy.
[0087] Based on the weight of each slot and the matching degree of the fill value of each slot, the matching reward value of each slot is determined.
[0088] The matching reward value for each slot is determined based on the matching reward value for each slot.
[0089] Specifically, a slot refers to a key information field in an intent template that needs to be filled in. In a user input, different slots have different levels of importance for the complete and accurate execution of the user intent.
[0090] In one specific embodiment, a slot system can be predefined, and each slot can be categorized into different importance levels. For example, slots can be divided into the following three levels:
[0091] (1) Core slots: Slots at this level are essential for the execution of the task; their absence or errors will cause the task to fail. For example, the "time" and "event" slots in an alert intent.
[0092] (2) Auxiliary slots: Slots at this level are not required, but they can provide richer contextual information to make task execution more complete. For example, reminding intents about "date" and "location".
[0093] (3) Optional slots: Slots at this level contain secondary supplementary information, and their presence or absence has little impact on the core functionality of the task. For example, the reminder object in the reminder intent.
[0094] After determining the importance level, a numerical weight is assigned to each slot at each level. Slots with higher importance levels have higher weights. For example, a core slot can be assigned a weight of 2.0, an auxiliary slot a weight of 1.5, and an optional slot a weight of 1.0. These weights are configurable parameters and can be dynamically adjusted according to specific business needs.
[0095] Imputed values refer to the specific numerical values or text generated by the large model for a particular slot in the candidate responses. Imputed ground truth values refer to the correct values corresponding to that slot in the manually labeled ground truth data.
[0096] The fill value matching score can be a score between 0 and 1, which quantifies how close the fill predicted value is to the fill true value.
[0097] To address the complexity of different types of slot values, this application's embodiments employ a flexible multi-level matching strategy, specifically including:
[0098] (1) Exact matching strategy: requires that the predicted padding value be completely consistent with the actual value at the string level;
[0099] (2) String similarity matching strategy: This strategy is used to handle situations where the padding value is free text and there may be multiple reasonable expressions. It does not require the strings to be completely equal, but rather calculates the semantic or literal similarity between the two strings. In one specific embodiment, the Jaro-Winkler distance algorithm can be used to calculate the similarity. The padding value matching degree can be expressed as... The matching bonus value for the corresponding slot is... .in, The weight of the slot. The string similarity between the fill value and the true value. Ensure the score does not exceed 1, and dynamically adjust the score based on similarity.
[0100] (3) Numerical Fault-Tolerant Matching Strategy: This strategy is used to handle slots containing numerical data such as time, date, amount, and quantity. In many real-world scenarios, minor deviations in these values are acceptable. This strategy achieves matching by setting a fault tolerance range. In a specific embodiment, the fill value matching degree can be expressed as... The matching bonus value for the corresponding slot is... .in, The weight of the slot. For predicted values, For the true value, Used to prevent the denominator from being zero.
[0101] By combining weights and matching degrees, the contribution of each evaluated slot to the total reward is calculated, which is the slot's matching reward value. Summing the matching reward values of all slots yields the slot matching reward value. In a specific embodiment, the slot matching reward value can be expressed by the formula: .in, Match reward values to slots; For the first The weight of each slot, For the first Predicted fill value for each slot. For the first The actual fill value for each slot. The number of slots; It is used to indicate the degree of matching between the filled predicted value and the filled actual value.
[0102] The large model intent understanding fine-tuning method based on reinforcement learning provided in this application refines the calculation process of slot matching reward values, enabling a more precise and reasonable evaluation of model output quality. By assigning weights to different slots according to their importance levels, the model can focus on learning key information, better meeting practical application needs. Furthermore, the comprehensive application of multi-level matching strategies ensures fair evaluation of various forms of correct or partially correct outputs. This provides high-quality, low-noise guidance signals for the entire reinforcement learning fine-tuning process, improving the large model intent understanding capability.
[0103] In some embodiments, the method further includes:
[0104] If the predicted filling value for any slot is null and the actual filling value is non-null, the matching reward value for any slot is deducted from the slot matching reward value.
[0105] If the predicted filling value for any slot is non-null and the actual filling value is null, determine the deduction ratio coefficient for any slot based on the importance level of any slot, and determine the matching reward value deduction for any slot based on the deduction ratio coefficient and the matching reward value of any slot; deduct the matching reward value deduction for any slot from the slot matching reward value.
[0106] If both the predicted and actual filling values for any slot are empty, the matching reward value for any slot will not be deducted from the slot matching reward value.
[0107] Specifically, this application embodiment has designed refined reward and punishment rules for various situations where the slot filling value is "null", which can be described in three cases.
[0108] In the first scenario, the predicted fill value for any slot is null, while the actual fill value is non-null. This indicates that the large model failed to identify a slot that should have been present in the current input, which may result in information loss and is considered a missed detection.
[0109] In this scenario, when a missed report is detected in a slot, the slot's contribution to the total reward should be negative. The matching reward value that the slot should receive under the assumption of a correct prediction (i.e., weight × 1.0, because ideally, the matching degree is 1.0 if the prediction is correct) is calculated as described in the previous embodiment. Then, this calculated ideal reward value is deducted as a penalty from the total slot matching reward value.
[0110] The second scenario involves a non-null predicted fill value for any slot, but a null actual fill value. This indicates that the model has "phantomized" slot information that does not exist in the current input. This is also an error that needs to be avoided because it introduces noise and interference into downstream business execution logic.
[0111] In this case, the embodiments of this application process the data according to the importance level of the slot. The main basis is that falsely reporting a high-importance core slot has a greater negative impact than falsely reporting a low-importance optional slot, and is therefore considered a false alarm.
[0112] First, a deduction ratio is determined based on the importance level of the slot being falsely flagged. This ratio is a value between 0 and 1. The higher the importance level of the slot, the larger the corresponding deduction ratio. For example, the deduction ratio for falsely flagged core slots can be set to 1.0, the deduction ratio for falsely flagged auxiliary slots to 0.6, and the deduction ratio for falsely flagged optional slots to 0.3.
[0113] Then, calculate the matching reward value (i.e., weight × 1.0) for that slot if it is assumed to exist and the prediction is correct. Multiply this value by the deduction ratio determined above to obtain the final matching reward value deduction.
[0114] Finally, subtract this calculated deduction from the total slot matching reward value.
[0115] The third scenario is where both the predicted and actual fill values for any slot are null. This indicates that the model accurately determined that a slot does not exist in the current input and therefore did not fill it. This is correct behavior and should not be penalized or rewarded.
[0116] The large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application has designed a comprehensive and refined set of reward and punishment rules for the empty value situation in slot filling, which greatly improves the accuracy and guidance ability of the reward function; the severe punishment for false negatives ensures the recall rate of the model, the differentiated punishment for false positives ensures the accuracy of the model, and the neutral treatment for correct rejections avoids the introduction of unnecessary noise; thus improving the accuracy of large models in performing intent understanding tasks.
[0117] In some embodiments, the method further includes:
[0118] An initial corpus containing multi-turn dialogues is generated based on a pre-set large model;
[0119] The initial corpus is labeled with intent tags, slot fill values, and dialogue context to generate a training dataset.
[0120] In the training dataset, the current input is determined based on the dialogue context, and the intent label and slot fill value in the dialogue context are determined as the true intent value and fill value of the current input.
[0121] Specifically, the pre-defined large model can be a high-performance, high-generation language model, such as a general-purpose large model with tens or hundreds of billions of parameters.
[0122] To establish a benchmark for model training and evaluation, an initial corpus containing multi-turn dialogues can be generated from a pre-set large model. Multi-turn dialogues are particularly important for intent understanding tasks because they can simulate complex real-world situations where context is required to accurately understand user intent.
[0123] The automatically generated initial corpus is in plain text format and cannot be directly used in the reinforcement learning fine-tuning process of this application's embodiments. Therefore, it needs to be structurally annotated to generate a training dataset containing supervisory signals. This step can be done manually, or a combination of model-assisted annotation and manual review can be used to improve efficiency. The annotation work includes:
[0124] (1) Intent labeling: For each user instruction in the dialogue that can trigger a clear task, label its corresponding intent label;
[0125] (2) Slot filling value labeling: After determining the intent label, extract all slots and their filling values related to the intent from the user command or its context;
[0126] (3) Dialogue context annotation: In order to handle the resolution of reference and information supplementation in multi-turn dialogue, it is necessary to annotate the context of each turn of dialogue, which usually means associating the current user command with its previous dialogue history.
[0127] After the above annotation process, the original plain text corpus is transformed into a structured training dataset. Each sample in the dataset typically includes: dialogue history (context), the current user command, and the standard intent label and slot fill value corresponding to the current command.
[0128] During fine-tuning, a sample is drawn from the training dataset. The "dialogue context" (which may contain multiple turns of historical dialogue) and the "current user instruction" in this sample together constitute the current input used to feed the model to be fine-tuned. This is done so that the model can learn to understand user intent in the presence of context.
[0129] Meanwhile, the intent labels and slot-filling values annotated in this sample are used as evaluation criteria, namely, the true intent value and the true slot-filling value. In the subsequent reward calculation step, the candidate responses generated by the large model will be compared with these true values to calculate their intent matching degree and slot-filling value matching degree, thereby obtaining the reward value.
[0130] The large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application utilizes a powerful pre-set large model to generate initial corpus, greatly reducing the cost and cycle of data acquisition; while the systematic multi-dimensional annotation process ensures the high quality and structure of training data, which can meet the requirements of reinforcement learning fine-tuning for accurate supervision signals; thus enabling the trained model to better handle complex dialogue scenarios in real applications, significantly improving the accuracy and robustness of intent understanding.
[0131] In some embodiments, the parameters of the large model are updated based on the relative advantage values of each candidate response, including:
[0132] The policy gradient of each candidate response is updated based on the relative advantage value of each candidate response;
[0133] The parameters of the large model are updated based on the updated policy gradient.
[0134] Specifically, the policy gradient can be understood as a vector that indicates in which direction the parameters of the large model should be adjusted to increase (or decrease) the probability of generating that particular candidate response.
[0135] In this embodiment, for any candidate response, the magnitude and direction of its policy gradient are functions of the gradient of its own generation probability and its relative advantage value. In a specific embodiment, the policy gradient update process is as follows:
[0136] First, for each candidate response in the local sample group, calculate the policy gradient of the log probability of the large model generating that response with respect to the model parameters. This gradient itself indicates the parameter update direction that best increases the probability of generating that response.
[0137] Then, the relative advantage value of each candidate response is multiplied by the policy gradient to obtain the updated policy gradient.
[0138] If the relative advantage value is positive, it will positively scale the original gradient. This means that the model parameters will be updated in a direction that increases the probability of generating this "good" response. And the larger the relative advantage value (i.e., the better the response), the stronger this push for update.
[0139] If the relative advantage value is negative, it scales the original gradient inversely. This means that the model parameters will be updated in a direction that reduces the probability of generating this "bad" response. Furthermore, the larger the absolute value of the relative advantage value (i.e., the worse the response), the stronger this "pull" or "suppression" of the update.
[0140] Based on the updated policy gradient, the parameters of the large model are updated. For example, the policy gradients of all candidate responses in the local sample group, weighted by relative advantage values, are first aggregated (e.g., by averaging) to obtain a final aggregated policy gradient representing the direction of this optimization. Then, an optimizer algorithm is used to update all or part of the parameters of the large model based on this aggregated policy gradient.
[0141] The large model intent understanding fine-tuning method based on reinforcement learning provided in this application calculates the policy gradient weighted by the relative advantage value, so that each step of the model optimization has a clear direction, which can significantly improve the accuracy of large models in complex tasks such as intent understanding.
[0142] In some embodiments, candidate responses include a reasoning label for the reasoning process and an answer label for the output result.
[0143] Specifically, in the embodiments of this application, the candidate response is no longer a single, direct answer output, but is designed as a composite structure, which includes at least two parts: a thinking label for the reasoning process. <think>and the answer labels of the output results <answer>.
[0144] The large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application designs the candidate response as a composite structure containing thinking labels and answer labels, which enhances the interpretability and debuggability of the model output and can significantly improve the accuracy of the model on complex intent understanding tasks.
[0145] In some embodiments, before determining the reward value of each candidate response based on the intent matching reward value and the slot matching reward value of each candidate response, the method further includes:
[0146] The completeness of the thinking tags and answer tags for each candidate response is verified.
[0147] Specifically, after receiving the current input, the large model generates multiple candidate responses, including structured responses (containing thought labels). <think>With answer tags <answer>) and unstructured response.
[0148] The method provided in this application can perform structured format verification on candidate responses. The structured format verification may include tag completeness verification, syntax tree analysis, and abnormal format penalties.
[0149] Tag completeness verification refers to verifying the existence of thought tags and answer tags using regular expressions and a syntax parser.
[0150] Syntax tree analysis refers to constructing an Abstract Syntax Tree (AST) to check the structural integrity of key-value pairs, especially verifying the node depth and data type of each field.
[0151] Abnormal format penalty refers to applying an exponentially decaying penalty term to unclosed labels.
[0152] The large model intent understanding fine-tuning method based on reinforcement learning provided in this application embodiment ensures the effectiveness of reinforcement learning feedback signals and avoids reward calculation errors or anomalies caused by chaotic model output format, thereby making the training process more stable.
[0153] Figure 4 This is the second flowchart of the large-scale model intent understanding fine-tuning method based on reinforcement learning provided in this application, as shown below. Figure 4 As shown, this method fine-tunes the parameters of a large model for the intent understanding task based on Generalized Reinforcement Preference Optimization (GRPO), specifically including:
[0154] Step 410: Receive input prompts.
[0155] Step 420: Generate a local sample group containing multiple candidate responses.
[0156] For each input prompt, the system generates multiple candidate responses, forming local sample groups to facilitate subsequent computation. In this process, each response contains not only the final... <answer>Output, also included <think>Reasoning information ensures that the data format is complete and structured.
[0157] Step 430: Calculate the reward value for each candidate response and the average reward within the group.
[0158] Step 440: Calculate the relative advantage value of each candidate response.
[0159] For each candidate response, the relative advantage value is calculated based on the difference between its reward score and the benchmark value. This relative advantage estimation helps reduce the variance of the reward estimate and improves the stability of gradient updates. This relative advantage estimation method effectively reduces the variance of the reward estimate and avoids the instability problems that may arise from constructing a value network separately in a high-dimensional state space.
[0160] Step 450: Update the policy gradient.
[0161] Step 460: Optimize the large model.
[0162] By using the calculated relative advantage value, the model parameters are adjusted through an adaptive policy gradient update algorithm, thereby significantly increasing the probability of the model generating a high-quality response.
[0163] Experimental data shows that small models with fewer than 7 billion parameters can gradually deduce user intent and corresponding context slots using the above method. On a small model with only 3 billion parameters, its performance is close to that of a 72-b model: intent recognition accuracy reaches 84%, and slot recognition accuracy is 76%. This breakthrough brings entirely new possibilities for edge deployments, especially in internet environments.
[0164] The apparatus provided in the embodiments of this application is described below. The apparatus described below can be referred to in correspondence with the method described above.
[0165] Figure 5 This is a schematic diagram of the structure of the large model intent understanding fine-tuning device based on reinforcement learning provided in this application, as shown below. Figure 5 As shown, the device includes:
[0166] The sample group generation module 510 is used to process the current input based on the large model and generate a local sample group containing multiple candidate responses.
[0167] The reward value determination module 520 is used to determine the reward value of each candidate response based on the intent matching reward value and the slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response; the slot matching reward value is determined based on the importance level of each slot and the matching degree of the fill value of each slot.
[0168] The advantage value determination module 530 is used to determine the mean reward of a local sample group based on the reward value of each candidate response, and to determine the relative advantage value of each candidate response based on the difference between the reward value of each candidate response and the mean reward.
[0169] The parameter update module 540 is used to update the parameters of the large model based on the relative advantage values of each candidate response.
[0170] The large-scale model intent understanding fine-tuning device based on reinforcement learning provided in this application generates local sample groups containing multiple candidate responses. Within each local sample group, reward values are compared to determine the relative advantage value of each candidate response. The relative advantage value is used for adaptive policy gradient update, effectively reducing the variance of the reward estimate. It does not rely on an independent value network that requires additional training, avoiding the instability problems that may be caused by building an independent value network. Combined with a refined reward function specifically designed for intent understanding tasks, which includes importance levels and multi-dimensional matching strategies, the large model can more accurately identify intents and slots in multi-turn dialogues and open scenarios. This improves the accuracy and generalization ability of the large model in performing intent understanding tasks, fully meeting the requirements of smart home and robot systems for rapid response and multi-task processing.
[0171] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communications bus 640. The processor 610 can call logical commands stored in the memory 630 to execute the methods described in the above embodiments, for example:
[0172] The large model processes the current input to generate a local sample group containing multiple candidate responses. Based on the intent matching reward value and slot matching reward value of each candidate response, the reward value for each candidate response is determined. The intent matching reward value is determined based on the intent matching degree of each candidate response. The slot matching reward value is determined based on the importance level of each slot and the fill value matching degree of each slot. The mean reward of the local sample group is determined based on the reward values of each candidate response, and the relative advantage value of each candidate response is determined based on the difference between the reward value of each candidate response and the mean reward value. The parameters of the large model are updated based on the relative advantage values of each candidate response.
[0173] Furthermore, the logical commands in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0174] The processor in the electronic device provided in this application embodiment can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effect, which will not be repeated here.
[0175] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.
[0176] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.
[0177] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / think> < / answer> < / answer> < / think> < / answer> < / think>
Claims
1. A large model intent understanding fine-tuning method based on reinforcement learning, characterized in that, include: The current input is processed based on a large model to generate a local sample set containing multiple candidate responses; The current input is the text that the large model needs to understand the intent of; The text is derived from the user's voice command conversion or real-time message input; The reward value for each candidate response is determined based on the intent matching reward value and slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response. The slot matching reward value is determined based on the importance level of each slot and the matching degree of the fill value of each slot. The mean reward of the local sample group is determined based on the reward value of each candidate response, and the relative advantage value of each candidate response is determined based on the difference between the reward value of each candidate response and the mean reward. The parameters of the large model are updated based on the relative advantage values of each candidate response. The method further includes: If the predicted filling value for any slot is null and the actual filling value is not null, the matching reward value for any slot is deducted from the matching reward value for that slot. If the predicted fill value for any slot is non-null and the actual fill value is null, determine the deduction ratio coefficient for any slot based on the importance level of the slot, determine the matching reward value deduction for any slot based on the deduction ratio coefficient and the matching reward value of the slot, and deduct the matching reward value deduction for any slot from the slot matching reward value. If both the predicted and actual filling values for any slot are empty, the matching reward value for that slot will not be deducted from the slot matching reward value.
2. The method for fine-tuning large model intent understanding based on reinforcement learning according to claim 1, characterized in that, The slot matching reward value is determined based on the following steps: The weight of each slot is determined based on its importance level. The matching degree of the filling value of each slot is determined based on at least one of the exact matching strategy, the string similarity matching strategy, and the numerical fault-tolerant matching strategy. Based on the weight of each slot and the matching degree of the fill value of each slot, the matching reward value of each slot is determined. The matching reward value for each slot is determined based on the matching reward value for each slot.
3. The method for fine-tuning large model intent understanding based on reinforcement learning according to claim 1, characterized in that, The method further includes: An initial corpus containing multi-turn dialogues is generated based on a pre-set large model; The initial corpus is annotated with intent tags, slot fill values, and dialogue context to generate a training dataset; The current input is determined based on the dialogue context in the training dataset, and the intent label and slot fill value in the dialogue context are determined as the true intent value and true fill value of the current input.
4. The method for fine-tuning large model intent understanding based on reinforcement learning according to claim 1, characterized in that, The updating of the parameters of the large model based on the relative advantage values of each candidate response includes: The policy gradient of each candidate response is updated based on the relative advantage value of each candidate response; The parameters of the large model are updated based on the updated policy gradient.
5. The method for fine-tuning large model intent understanding based on reinforcement learning according to any one of claims 1 to 4, characterized in that, The candidate responses include the reasoning labels for the reasoning process and the answer labels for the output results.
6. The method for fine-tuning large model intent understanding based on reinforcement learning according to claim 5, characterized in that, Before determining the reward value of each candidate response based on the intent matching reward value and slot matching reward value of each candidate response, the method further includes: The completeness of the thinking tags and answer tags for each candidate response is verified.
7. A fine-tuning device for understanding the intent of a large model based on reinforcement learning, characterized in that, include: The sample group generation module is used to process the current input based on the large model and generate local sample groups containing multiple candidate responses. The current input is text that the large model needs to understand; the text originates from the user's voice command conversion or instant message input. The reward value determination module is used to determine the reward value of each candidate response based on the intent matching reward value and slot matching reward value of each candidate response; the intent matching reward value is determined based on the intent matching degree of each candidate response. The slot matching reward value is determined based on the importance level of each slot and the matching degree of the fill value of each slot; The advantage value determination module is used to determine the mean reward of the local sample group based on the reward value of each candidate response, and to determine the relative advantage value of each candidate response based on the difference between the reward value of each candidate response and the mean reward. The parameter update module is used to update the parameters of the large model based on the relative advantage values of each candidate response. The device is also used for: If the predicted filling value for any slot is null and the actual filling value is not null, the matching reward value for any slot is deducted from the matching reward value for that slot. If the predicted fill value for any slot is non-null and the actual fill value is null, determine the deduction ratio coefficient for any slot based on the importance level of the slot, determine the matching reward value deduction for any slot based on the deduction ratio coefficient and the matching reward value of the slot, and deduct the matching reward value deduction for any slot from the slot matching reward value. If both the predicted and actual filling values for any slot are empty, the matching reward value for that slot will not be deducted from the slot matching reward value.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the large model intent understanding fine-tuning method based on reinforcement learning as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large model intent understanding fine-tuning method based on reinforcement learning as described in any one of claims 1 to 6.