Operating system controlled intent recognition model reinforcement learning training method and system

Abstract

The application discloses an operating system control intention recognition model reinforcement learning training method and system, the method comprises the following steps: constructing an intention and a slot label for a natural language command used for operating system control, and constructing a training data set according to the natural language command and the corresponding intention and slot label; using the LoRA technology to supervise and fine-tune a pre-trained intention recognition model based on the training data set, and obtaining a cold-start intention recognition model; using a reinforcement learning method introducing a group relative strategy optimization to perform reinforcement learning training on the cold-start intention recognition model, so as to obtain a final intention recognition model that can be used for end-side deployment. The application aims to overcome the problems of insufficient coverage of the intention and slot label system, high data quality requirements but difficult to obtain, high model customization cost and lack of user feedback driven optimization in the prior art, and improve the speed and accuracy of operating system intention recognition on the end-side device.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for operating systems, and specifically to a reinforcement learning training method and system for an intent recognition model of operating system control. Background Technology

[0002] As the functions of domestically developed operating systems become increasingly complex, users are placing higher demands on the interaction methods of system operation. Traditional human-computer interaction mainly relies on graphical interfaces and fixed commands. However, with the rapid development of Natural Language Processing (NLP) and Large Language Models (LLM), intelligent interaction based on intent recognition is gradually becoming a key capability of the next generation of operating systems. Through intent recognition, the operating system can understand the user's natural language input, parse their goals and operational needs, and thus directly trigger the corresponding system functions, such as opening applications, adjusting settings, or performing complex multi-step operations.

[0003] However, intent recognition in operating system scenarios presents significant challenges: (1) Scenario complexity: Operating systems involve numerous commands and functions, and intent recognition must not only handle direct explicit instructions but also understand ambiguous, context-dependent, or cross-modal user inputs. (2) Interaction robustness: User inputs are semantically diverse and highly arbitrary, and existing models are prone to recognition errors in boundary scenarios and under non-standard inputs. (3) Resource constraints: Device-side computing and storage resources are limited, and deploying large models requires efficient model compression and efficient parameter tuning methods.

[0004] To address the aforementioned issues, reinforcement learning (RL) methods have been introduced into the training process of intent recognition models, using user feedback as optimization signals to improve the model's performance in real-world environments. Simultaneously, efficient parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) can achieve rapid customization for specific tasks within the operating system without significantly increasing the number of model parameters. Group Relative Policy Optimization (GRPO) reinforcement learning methods can better align model output with user needs while maintaining model stability. Therefore, how to combine the characteristics of intent recognition tasks in domestic operating systems with efficient LoRA parameter fine-tuning and GRPO reinforcement learning training to achieve efficient, accurate, and deployable model optimization has become a pressing issue. Although existing intent recognition technologies have been widely applied in typical scenarios such as voice assistants, their adaptability in the control field of domestic operating systems remains insufficient. Rule-based or traditional machine learning methods rely on manually designed rule bases or shallow models (such as SVM and CRF), resulting in poor scalability, difficulty in covering complex and diverse system instructions, and low accuracy when facing fuzzy or complex intents. While deep learning-based end-to-end methods improve recognition accuracy, they heavily rely on large-scale labeled data, have limited generalization capabilities, and consume excessive computational resources when deployed on mobile devices. Transfer learning methods based on pre-trained large language models (such as BERT and GPT) offer improved generalization capabilities, but their customization process requires full parameter fine-tuning, resulting in high costs and large storage consumption, hindering rapid adaptation to operating system devices. Efficient parameter fine-tuning methods (such as LoRA) alleviate the fine-tuning overhead to some extent, but most existing methods are still limited to supervised training and lack effective utilization of real user preferences and interaction feedback. Overall, existing technologies have significant shortcomings in terms of intent and slot label system coverage, training efficiency, device adaptability, and alignment with user needs, making it difficult to fully support diverse operating system function calls. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a reinforcement learning training method and system for an operating system control intent recognition model, which addresses the above-mentioned problems in the prior art. The present invention aims to overcome the problems of insufficient coverage of intent and slot label system, high data quality requirements but difficulty in acquisition, high cost of model customization, and lack of user feedback-driven optimization in the prior art, thereby improving the speed and accuracy of operating system intent recognition on the edge device.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A reinforcement learning training method for an intent recognition model controlled by an operating system includes the following steps:

[0008] S101, construct intents and slot labels for natural language commands used for operating system control, wherein the intents and slot labels include intents and slot arrays used for operating system control, and the slot array contains control parameters for operating system control.

[0009] S102, construct a training dataset based on natural language commands and their corresponding intents and slot labels;

[0010] S103 uses LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model;

[0011] S104. A reinforcement learning method with relative policy optimization is used to train the cold-start intent recognition model, thereby obtaining the final intent recognition model that can be used for edge deployment.

[0012] Optionally, in step S101, when constructing intents and slot labels for natural language commands used for operating system control, the intents in the intents and slot labels include primary intents and secondary intents. The primary intents and secondary intents are obtained by semantic abstraction based on the functional modules of the operating system. The primary intents correspond to the primary functional modules of the operating system, and the secondary intents correspond to the functional modules under the primary functional modules of the operating system.

[0013] Optionally, in step S102, when constructing the training dataset based on natural language commands and their corresponding intents and slot labels, the training samples in the training dataset include manually labeled samples, semi-automatically labeled samples, and multi-round interactive enhancement samples, and the construction of the training dataset includes:

[0014] S201, construct artificially labeled samples consisting of natural language commands and their corresponding manually annotated intents and slot labels;

[0015] S202, based on template expansion and synonym replacement, natural language commands in manually labeled samples are processed to obtain semi-automatic labeled samples consisting of new natural language commands, their original intent and slot labels;

[0016] S203 consists of manually labeled samples and semi-automatically labeled samples. It is composed of multi-turn interaction enhancement samples by adding contextual dialogue, user follow-up questions, and intent superposition. The multi-turn interaction enhancement samples include multi-turn multi-intent interaction samples and multi-turn single-intent interaction samples. The multi-turn multi-intent interaction samples consist of natural language commands for multiple intents, their contextual dialogues, their original intents, and slot labels. The multi-turn single-intent interaction samples consist of natural language commands for a single intent, their contextual dialogues, user follow-up questions, their original intents, and slot labels.

[0017] Optionally, when using LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset in step S103, the output of the cold-start intent recognition model includes the intent, the slot array, and the follow-up output. The follow-up output is used to provide feedback to the user on the missing slot information of the slot array in order to re-acquire the natural language command input by the user.

[0018] Optionally, in step S104, when using a reinforcement learning method that incorporates group-relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows:

[0019] ;

[0020] in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The reward for correct follow-up questions is as follows: the format correctness reward includes whether the output format of the intent recognition model is correct, and the scores for whether the intent, slot array, and follow-up question output exist in the output of the intent recognition model; the intent accuracy reward is the score corresponding to the cosine similarity between the intent and intent label in the output of the intent recognition model; the slot accuracy reward includes the scores corresponding to whether the slot array is extracted, whether the intent contains slots, and whether the slot information in the slot array is correct; the follow-up question correctness reward is the score of the correlation between the follow-up question output and the missing slot information.

[0021] Optionally, when using a reinforcement learning method that introduces group relative policy optimization to train the cold-start intention recognition model in step S104, and updating the parameters of the intention recognition model using backpropagation of the gradient of the loss function during reinforcement learning training, the expression for the gradient of the loss function is as follows:

[0022] ;

[0023] in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... Let KL divergence be the KL divergence, and we have:

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] in, To obtain the minimum value, As an uncut alternative target, As a replacement target after cropping, Generate lexical units for the two rounds of intent recognition model The probability ratio, as a word element The advantage function represents the lexical generation by the intent recognition model. The reward obtained is based on the training sample. The difference between the reward and the average reward of all training samples; For the clipping function, For trimming parameters, Generate lexical units for the intent recognition model in the ideal model The probability, Generate lexical units for the intent recognition model under the current parameters. The probability of.

[0029] Optionally, the loss function is expressed as follows:

[0030] ;

[0031] ;

[0032] in, For loss function, For the mathematical expectation value, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, To cut off alternative targets, The penalty coefficient is... Let KL divergence be denoted as KL divergence.

[0033] The present invention also provides an operating system-controlled intention recognition model reinforcement learning training system, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method.

[0034] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute, via a processor, an intent recognition model reinforcement learning training method controlled by the operating system.

[0035] The present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute, via a processor, an intention recognition model reinforcement learning training method controlled by the operating system.

[0036] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The reinforcement learning training method for the intent recognition model of the operating system control of the present invention includes constructing intent and slot labels for natural language commands used for operating system control, constructing a training dataset based on the natural language commands and their corresponding intents and slot labels; using LoRA technology to perform supervised fine-tuning of the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model; and using a reinforcement learning method that introduces group relative policy optimization to perform reinforcement learning training on the cold-start intent recognition model, thereby obtaining the final intent recognition model that can be used for edge deployment. The present invention can overcome the problems of insufficient coverage of intent and slot label system, high data quality requirements but difficult acquisition, high model customization cost, and lack of user feedback-driven optimization in existing technologies, thereby improving the speed and accuracy of operating system intent recognition on edge devices. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0038] Figure 2 This is a flowchart illustrating the calculation process of the model output format reward in this embodiment of the invention.

[0039] Figure 3 This is a flowchart illustrating the calculation of the intention correctness reward in an embodiment of the present invention.

[0040] Figure 4 This is a flowchart illustrating the calculation of slot correctness rewards in an embodiment of the present invention.

[0041] Figure 5 This is a flowchart illustrating the calculation of the correctness reward in an embodiment of the present invention.

[0042] Figure 6 This is a flowchart illustrating the calculation of word-level rewards in an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0044] To overcome the problems in existing technologies, such as insufficient coverage of intent and slot labeling systems, high data quality requirements but difficulty in obtaining data, excessively high costs of model customization, and lack of user feedback-driven optimization, Figure 1 As shown, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment includes the following steps:

[0045] S101, construct intents and slot labels for natural language commands used for operating system control, wherein the intents and slot labels include intents and slot arrays used for operating system control, and the slot array contains control parameters for operating system control.

[0046] S102, construct a training dataset based on natural language commands and their corresponding intents and slot labels;

[0047] S103 uses LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model;

[0048] S104. A reinforcement learning method with relative policy optimization is used to train the cold-start intent recognition model, thereby obtaining the final intent recognition model that can be used for edge deployment.

[0049] In step S101, a systematic analysis of the operating system's functional modules is performed, abstracting common system operations into multi-level intents and designing corresponding slot label sets. This label system covers explicit instructions (such as "turn on Bluetooth") and ambiguous instructions (such as "light up"), thereby ensuring that the model can fully understand and parse diverse user inputs in operating system scenarios. Specifically, in step S101 of this embodiment, when constructing intents and slot labels for natural language commands used for operating system control, the intents in the intents and slot labels include primary intents and secondary intents. The primary intents and secondary intents are obtained by semantic abstraction based on the operating system's functional modules. Primary intents correspond to the primary functional modules of the operating system, and secondary intents correspond to the functional modules under the primary functional modules of the operating system. Specifically, in this embodiment, in the typical interaction scenario of the Galaxy Kylin operating system, the natural language commands input by users usually involve multiple types of operation intents and parameter information. To achieve systematic modeling, the functional modules of the operating system are first semantically abstracted, and the functional behaviors are divided into two levels of intent labels. The primary intents contain 8, and 87 secondary intent labels are derived from the primary intents, as shown in Table 1 for some examples.

[0050] Table 1: Examples of Intent and Slot Labels

[0051]

[0052] Taking the second intent and slot labels as an example, the first-level intent is "System Operation," and the second-level intent is "Switch Display Projection Mode." The slot array includes four slots: "Name," "Type," "Required," and "Description." Each slot in the slot array can be empty, and the number can be configured according to the intent's needs. For each second-level intent, this invention designs a set of slot labels to extract parameter information involved in the command, such as the target application, the operation object, and status values. The label system adopts a hierarchical design, enabling the model to adapt to various expression forms such as explicit commands, fuzzy commands, and compound commands, ensuring comprehensive coverage of operating system control tasks.

[0053] After establishing the intent and slot label system, this invention constructs a high-quality dataset based on actual operating system instruction scenarios. Building upon the intent and slot label system constructed in step S101, step S102 of this embodiment constructs a high-quality training dataset covering typical operating system instructions through a combination of manual annotation and semi-automatic generation. The dataset contains rich expression variations, contextual scenarios, and multi-turn interaction examples, ensuring the model's robustness in the face of diverse inputs. Simultaneously, this embodiment emphasizes the principle of "small but precise" data construction, using small batches of high-quality data instead of large-scale low-quality corpora to improve training efficiency and model generalization ability. Specifically, in step S102 of this embodiment, when constructing the training dataset based on natural language commands and their corresponding intents and slot labels, the training samples in the training dataset include manually labeled samples, semi-automatically labeled samples, and multi-turn interaction enhancement samples, and the construction of the training dataset includes:

[0054] S201, construct artificially labeled samples consisting of natural language commands and their corresponding manually annotated intents and slot labels;

[0055] S202, based on template expansion and synonym replacement, natural language commands in manually labeled samples are processed to obtain semi-automatic labeled samples consisting of new natural language commands, their original intent and slot labels;

[0056] S203 consists of manually labeled samples and semi-automatically labeled samples. Multi-turn interaction enhancement samples are constructed by adding contextual dialogues, user follow-up questions, and intent overlays. These multi-turn interaction enhancement samples include multi-turn multi-intent interaction samples and multi-turn single-intent interaction samples. The multi-turn multi-intent interaction samples consist of natural language commands for multiple intents, their contextual dialogues, their original intents, and slot labels. The multi-turn single-intent interaction samples consist of natural language commands for a single intent, their contextual dialogues, user follow-up questions, their original intents, and slot labels. The final dataset contains various representations and reaches thousands of data points, with a focus on ensuring quality and representativeness to provide efficient supervision signals for subsequent model training.

[0057] In step S103 of this embodiment, when the LoRA technique is used to supervise the fine-tuning of the pre-trained intent recognition model based on the training dataset, the output of the cold-start intent recognition model includes the intent, the slot array, and the follow-up output. The follow-up output is used to provide feedback to the user on the missing slot information of the slot array to re-acquire the natural language command input by the user. Addressing the issue of limited computing and storage resources on the operating system's edge devices, step S103 of this embodiment selects the pre-trained large language model Qwen3-1.7b, which possesses good language understanding capabilities, as the pre-trained large language model. Based on the pre-trained large language model, the Low-Rank Adaptation (LoRA) method is used for efficient parameter fine-tuning. Unlike full parameter fine-tuning, the LoRA method only needs to update some low-rank matrix parameters in the model, avoiding the storage and computing overhead caused by full parameter fine-tuning. This significantly reduces the number of trainable parameters and GPU memory usage, thereby quickly obtaining a cold-start model adapted to the operating system control scenario. In this way, the model can run efficiently on the edge, possessing fast inference speed and low storage usage, exhibiting good inference speed and low resource requirements on edge devices.

[0058] To further improve the model's performance in real user interactions, step S104 of this embodiment introduces the Group Relative Policy Optimization (GRPO) method based on the cold-start model. This method utilizes user feedback signals for reinforcement learning training. By constructing a reward function, it replaces the traditional reinforcement learning method's reliance on absolute reward signals provided by the environment with "relative merit within the group" to evaluate policy effectiveness. This guides the intent recognition model to generate intent and slot parsing results that better meet user needs under diverse user inputs. GRPO improves training stability during the optimization process, avoiding gradient oscillations or convergence difficulties common in traditional reinforcement learning methods, thereby further improving the model's accuracy and robustness in actual interactions. Specifically, in step S104 of this embodiment, when using the reinforcement learning method with introduced group relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows:

[0059] ;

[0060] in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The correctness reward for follow-up questions; wherein, the format correctness reward includes whether the output format of the intent recognition model is correct, and whether the output of the intent recognition model contains intent, slot array, and follow-up question output scores; for example, as an optional implementation, such as Figure 2 As shown, the calculation of the format correctness bonus includes:

[0061] Step 1: Determine if the output of the intent recognition model is in JSON format. If it is, increase the format correctness reward by 0.5 points and proceed to Step 2; otherwise, the format correctness reward is 0 points, and the process ends.

[0062] Step 2: Determine the number of intents in the output of the intent recognition model. If the number of intents is 1, the format correctness reward increases by 0.3 points, and proceed to Step 3; otherwise, the format correctness reward is 0 points, and the process ends.

[0063] Step 3: Determine the number of follow-up questions output by the intent recognition model. If the number of follow-up questions is 1, the format correctness reward increases by 0.3 points, and the process proceeds to Step 3; otherwise, the format correctness reward is 0 points, and the process ends.

[0064] Step 4: Determine the number of slot arrays in the intent recognition model. If the number of slot arrays is 1, the format correctness reward increases by 0.3 points, and the process jumps to step 3; otherwise, the format correctness reward is 0 points, and the process ends.

[0065] The intent accuracy reward is the score corresponding to the cosine similarity between the intent and the intent label in the output of the intent recognition model; for example, as an optional implementation, such as Figure 3 As shown, the calculation of the intent accuracy reward includes:

[0066] Step 1: Analyze the intent in the output of the intent recognition model;

[0067] Step 2: Calculate the cosine similarity between the intent and the labeled information;

[0068] Step 3: If the cosine similarity is greater than or equal to 0.9, the intention accuracy reward is 1; if the cosine similarity is greater than or equal to 0.8 and less than 0.9, the intention accuracy reward is 0.5; if the cosine similarity is less than 0.8, the intention accuracy reward is 0.

[0069] The slot accuracy reward includes scores for whether a slot array was extracted, whether the intent contains slots, and whether the slot information in the slot array correctly corresponds to the desired slot. For example, as an optional implementation, such as... Figure 4 As shown, the calculation of slot accuracy bonus includes:

[0070] Step 1: Determine the score of the intent accuracy reward. If the intent accuracy reward is 0, then the slot accuracy reward is 0, end and return; otherwise, proceed to step 2.

[0071] Step 2: Analyze the array size of slot bit groups in the output of the intent recognition model;

[0072] Step 3: Determine if the array size of the slot number groups is 0. If it is 0, proceed to step 4; otherwise, proceed to step 6.

[0073] Step 4: Analyze the intent type in the output of the intent recognition model. If the intent type is a slotless intent, the intent accuracy reward is 1, and the process ends and returns; otherwise, proceed to step 5.

[0074] Step 5: Determine if there is slot information in the input of the intent recognition model. If there is slot information, the intent accuracy reward is -1 as a penalty; otherwise, the intent accuracy reward is 1; end and return.

[0075] Step 6: Analyze the intent type of the intent in the output of the intent recognition model. If the intent type is a slotless intent, the intent accuracy reward is -0.2, and the process ends and returns; otherwise, proceed to step 7.

[0076] Step 7: Determine whether there is slot information in the input of the intent recognition model. If there is no slot information, the intent accuracy reward is -0.1, and the process ends and returns; otherwise, proceed to step 8.

[0077] Step 8: Determine if the slot array has been completely extracted. If the slot array has been completely extracted, the intent accuracy reward is 1; otherwise, the intent accuracy reward is 0.5. Perform a validity check on each slot in the slot array. If valid, update the slot accuracy reward according to the following formula:

[0078] ;

[0079] Otherwise, update the slot accuracy reward according to the following formula (no reward will be given if the slot value is unreasonable, and the reward score will be increased by 0):

[0080] ;

[0081] in," "An operator added to the original value, The size of the array of slots.

[0082] As an optional implementation, the correctness of the slot information in the slot array is determined by the inclusion of natural language commands and training samples. The intent label, the target slot name, the target slot requirements, and the values ​​of the slot information in the slot array are combined with the preset prompt words of the task instruction containing whether the slot information in the slot array is correct and input into the large language model to obtain the result.

[0083] The correctness reward for follow-up questions is a score reflecting the relevance of the follow-up question output to the missing slot information. As an optional implementation method, such as... Figure 5 As shown, the calculation of the reward for correct follow-up questions includes:

[0084] Step 1: parse the slot array in the output of the intent recognition model, and determine whether there are any required slots. If there are no required slots, the correctness reward for follow-up questions is 0, and the process ends and exits; otherwise, proceed to Step 2.

[0085] Step 2: Check the number of missing required slots in the slot array. If the number of missing required slots is 0, the correctness reward for follow-up questioning is 0, and the process ends and exits; otherwise, proceed to step 3.

[0086] Step 3: Analyze the follow-up question output in the intent recognition model. If the follow-up question output string is empty, the follow-up question correctness reward is -0.5, and the process ends and exits; otherwise, proceed to step 4.

[0087] Step 4: Evaluate the correctness of the follow-up question output using a large language model. If the evaluation result is correct, the follow-up question correctness reward is 0.5; otherwise, the follow-up question correctness reward is 0. In this embodiment, evaluating the correctness of the follow-up question output using a large language model includes processing the training samples... The intent label, the name of the missing slot, and the follow-up question output are combined with the task instruction prompt words that ask whether the missing slot information in the slot array can be filled. After inputting these into the large language model, the output is either "yes" or "no", indicating whether the evaluation result is correct.

[0088] like Figure 6 As shown, the word reward score calculation method of the intent recognition model in this embodiment is as follows: Initialize the reward scores of all words, and more accurately allocate the format reward score, intent reward score, slot reward score and follow-up question reward score to each word to obtain the score of each word in the output sequence. In this way, the abnormality of the score of one item in the sequence will not affect the score of other items. Each word has an independent score, which can realize the accurate transmission of gradient signals.

[0089] In step S104 of this embodiment, when the cold-start intention recognition model is trained using a reinforcement learning method that incorporates group relative policy optimization, the gradient of the loss function is used to backpropagate and update the parameters of the intention recognition model during reinforcement learning training. The expression for the gradient of the loss function is as follows:

[0090] ;

[0091] in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... The gradient described above, representing the KL divergence, consists of two parts: a reward-driven term to maximize the reward and a KL penalty term to limit deviation and ensure the stability of the model's training. The optimized model can more accurately identify user intent and maintain high recognition consistency across scenarios and multiple commands.

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] in, To take the minimum value means choosing a more conservative estimate. As an uncut alternative target, As a replacement target after cropping, For the current policy in state Select action The probability of the old strategy in the state Select action The ratio of the probabilities, The advantage function for each word represents the advantage of each word generated by the intent recognition model and evaluates the merits of the current action. For the clipping function, For trimming parameters, This represents the probability that the policy model (i.e., the trained model) generates a sequence of lexical terms before the policy is updated. The probability of the policy model generating a sequence of terms after the policy update can be obtained during the model's forward propagation. The advantage function for each term is expressed as follows:

[0097] ;

[0098] in, The reward for the word at position t. The average reward within the group. The within-group standard deviation To prevent decimals from being divided by zero.

[0099] In this embodiment, the function expression of the loss function is:

[0100] ;

[0101] ;

[0102] in, For loss function, For the mathematical expectation value, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, As a substitute target, The penalty coefficient is... The KL divergence is used. The completeness mask is the generated word mask used to distinguish which words need to be calculated for loss. The completeness mask is a 0-1 sequence corresponding to the word. 0 represents invalid words that are not included in the calculation, and 1 represents valid words that are included in the calculation. Through the above training process, the obtained intent recognition model has the following characteristics: (1) Lightweight: LoRA fine-tuning greatly reduces parameter storage and computation, making it suitable for deployment in terminal devices or operating system kernels; (2) High precision: Based on high-quality datasets and reinforcement learning optimization, the model performs stably in complex intent parsing tasks; (3) Strong scalability: The label system and training framework can be extended according to the new system functions, and can quickly adapt to different operating system versions and device types.

[0103] To verify the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment, relevant experiments were conducted. Experimental results show that, under the same hardware conditions, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment reduces model parameters by approximately 80%, reduces edge inference latency by approximately 40%, and improves accuracy by approximately 10%–15% on multi-scenario intent recognition tasks within the operating system. Therefore, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment effectively improves the natural language interaction capabilities and user experience of the operating system.

[0104] In summary, addressing the diverse, ambiguous, and cross-application characteristics of user natural language commands in operating system control scenarios, this embodiment's reinforcement learning training method for the intent recognition model in operating system control comprehensively utilizes key technologies such as label system construction, dataset construction, efficient parameter fine-tuning (LoRA), and group relative policy (GRPO) optimization. This method gradually overcomes the shortcomings of existing technologies in terms of coverage, training cost, adaptation efficiency, and alignment with user needs. It not only achieves efficient training under conditions of small batches of high-quality manually labeled data but also improves on-device adaptation efficiency through LoRA and effectively utilizes user feedback for optimization in conjunction with GRPO. This results in fast inference speed, high recognition accuracy, and strong robustness of intent recognition capabilities on on-device devices, significantly improving the natural language interaction experience of domestic operating systems. The intent recognition model obtained through the training path of "systematic intent labels + high-quality small datasets + LoRA fine-tuning + GRPO optimization" in this embodiment not only has fast inference speed and low resource consumption but also maintains high recognition accuracy and robustness under complex and varied user input, thereby significantly improving the natural language interaction experience of operating systems. This embodiment of the reinforcement learning training method for the intent recognition model under operating system control combines LoRA (Locally Aligned Parameters) with GRPO-optimized reinforcement learning in the context of operating system control on the edge side. This addresses the generalization, lightweighting, and interaction adaptability issues of the intent recognition model, achieving the following significant technical effects: (1) Significantly improves the coverage and accuracy of intent recognition under operating system control. This invention constructs a systematic intent and slot label system for operating system control scenarios, covering system-level operations, application-level control, and fuzzy intent scenarios, effectively improving the model's understanding of diverse system function calls. Compared to traditional voice assistant-style intent recognition systems, the recognition accuracy is improved by approximately 10% to 20%. (2) Reduces model fine-tuning costs and deployment resource overhead. By adopting LoRA, only a small number of low-rank weights need to be adjusted on the pre-trained model, significantly reducing the number of training parameters and storage overhead. Compared to full fine-tuning, the number of training parameters can be reduced by more than 90%, significantly improving the rapid adaptation and updating capabilities of the edge model. (3) Reinforcement learning-driven adaptive model optimization mechanism. This invention introduces a reinforcement learning method based on GRPO optimization, which uses the scoring mechanism of intent recognition results and interactive feedback information to dynamically optimize the model strategy, enabling the model to actively initiate follow-up questions or clarifications when faced with ambiguous, complex, or undefined intents, thereby improving user interaction experience and task completion rate. (4) High-quality training and generalization under small sample conditions. By combining manually labeled data with high-quality data augmentation strategies, the cold-start model can complete high-precision intent recognition with limited samples. GRPO training further enhances the robustness of the model under different user expression methods, solving the problem of traditional supervised learning's dependence on large-scale labeled data.

[0105] Furthermore, this embodiment also provides an operating system-controlled intention recognition model reinforcement learning training system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method via a processor.

[0106] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0107] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A reinforcement learning training method for an intent recognition model controlled by an operating system, characterized in that, Includes the following steps: S101, construct intents and slot labels for natural language commands used for operating system control, wherein the intents and slot labels include intents and slot arrays used for operating system control, and the slot array contains control parameters for operating system control. S102, construct a training dataset based on natural language commands and their corresponding intents and slot labels; S103 uses LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model; S104, The cold-start intent recognition model is trained by reinforcement learning method that introduces group relative policy optimization, so as to obtain the final intent recognition model that can be used for edge deployment. In step S103, when using LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset, the output of the cold-start intent recognition model includes the intent, slot array, and follow-up output. The follow-up output is used to provide feedback to the user on missing slot information of the slot array to re-acquire the user's input natural language command. In step S104, when using a reinforcement learning method that introduces group-relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows: ； in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The correctness reward for follow-up questions; the format correctness reward includes whether the output format of the intent recognition model is correct, and whether the output of the intent recognition model contains intent, slot array, and follow-up question output scores; The intent accuracy reward is the score corresponding to the cosine similarity between the intent and intent label in the output of the intent recognition model; the slot accuracy reward includes the scores for whether a slot array is extracted, whether the intent contains slots, and whether the slot information in the slot array is correctly matched; the follow-up question correctness reward is the score of the correlation between the follow-up question output and the missing slot information; in step S101, when constructing intents and slot labels for natural language commands used for operating system control, the intents in the intents and slot labels include primary intents and secondary intents. The primary intents and secondary intents are obtained by semantic abstraction based on the functional modules of the operating system. The primary intent corresponds to the primary functional module of the operating system, and the secondary intent corresponds to the functional module under the primary functional module of the operating system.

2. The reinforcement learning training method for the intent recognition model of operating system control according to claim 1, characterized in that, In step S102, when constructing the training dataset based on natural language commands and their corresponding intents and slot labels, the training samples in the training dataset include manually labeled samples, semi-automatically labeled samples, and multi-round interactive enhancement samples. The construction of the training dataset includes: S201, construct artificially labeled samples consisting of natural language commands and their corresponding manually annotated intents and slot labels; S202, based on template expansion and synonym replacement, natural language commands in manually labeled samples are processed to obtain semi-automatic labeled samples consisting of new natural language commands, their original intent and slot labels; S203 consists of manually labeled samples and semi-automatically labeled samples. It is composed of multi-turn interaction enhancement samples by adding contextual dialogue, user follow-up questions, and intent superposition. The multi-turn interaction enhancement samples include multi-turn multi-intent interaction samples and multi-turn single-intent interaction samples. The multi-turn multi-intent interaction samples consist of natural language commands for multiple intents, their contextual dialogues, their original intents, and slot labels. The multi-turn single-intent interaction samples consist of natural language commands for a single intent, their contextual dialogues, user follow-up questions, their original intents, and slot labels.

3. The reinforcement learning training method for the intent recognition model of operating system control according to claim 1, characterized in that, In step S104, when the cold-start intention recognition model is trained using a reinforcement learning method that incorporates group relative policy optimization, the gradient of the loss function is used for backpropagation to update the parameters of the intention recognition model during reinforcement learning training. The expression for the gradient of the loss function is as follows: ； in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... Let KL divergence be the KL divergence, and we have: ； ； ； ； in, To obtain the minimum value, As an uncut alternative target, As a replacement target after cropping, Generate tokens for the two rounds of the intent recognition model The probability ratio, as a word element The advantage function represents the lexical generation by the intent recognition model. The reward obtained is based on the training sample. The difference between the reward and the average reward of all training samples; For the clipping function, For trimming parameters, Generate lexical units for the intent recognition model in the ideal model The probability, Generate lexical units for the intent recognition model under the current parameters. The probability of.

4. The reinforcement learning training method for the intent recognition model of operating system control according to claim 3, characterized in that, The function expression of the loss function is: ； ； in, For loss function, For the mathematical expectation value, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, To cut off alternative targets, The penalty coefficient is... Let KL divergence be denoted as KL divergence.

5. An operating system-controlled reinforcement learning training system for an intent recognition model, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the intention recognition model reinforcement learning training method controlled by the operating system of any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the intention recognition model reinforcement learning training method controlled by the operating system as described in any one of claims 1 to 4.

7. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the intention recognition model reinforcement learning training method controlled by the operating system as described in any one of claims 1 to 4.

Images