Operating system controlled intention recognition model reinforcement learning training method and system

Abstract

The invention discloses an intention recognition model reinforcement learning training method and system controlled by an operating system, and the method comprises the steps: constructing intention and slot labels for natural language commands controlled by the operating system, and constructing a training data set according to the natural language commands and the corresponding intention and slot labels; a LoRA technology is adopted to supervise and finely adjust the pre-trained intention recognition model based on the training data set, and a cold start intention recognition model is obtained; and carrying out reinforcement learning training on the cold-start intention recognition model by adopting a reinforcement learning method introducing group relative strategy optimization, so as to obtain the intention recognition model which can be finally used for end-side deployment. The method aims at solving the problems that in the prior art, coverage of an intention and slot label system is insufficient, data quality requirements are high but acquisition is difficult, model customization cost is too high, and user feedback driving optimization is lacked, and the speed and precision of operating system intention recognition on end side equipment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control of operating systems, and in particular to an intent recognition model reinforcement learning training method and system for operating system control. BACKGROUND

[0002] With the increasing complexity of domestic operating systems, users have higher requirements for the interaction mode of system operation. Traditional human-computer interaction mainly relies on graphical interfaces and fixed commands, but with the rapid development of natural language processing (NLP) and large language models (LLM), intelligent interaction based on intent recognition has gradually become a key capability of the new generation of operating systems. Through intent recognition, the operating system can understand the user's natural language input, analyze its target and operation demand, and directly trigger the corresponding system function, such as opening an application, adjusting settings, or performing complex multi-step operations.

[0003] However, intent recognition in the operating system scenario presents significant challenges: (1) Scenario complexity: Operating systems involve a wide range of commands and functions, and intent recognition not only has to deal with direct explicit instructions, but also needs to understand ambiguous, context-dependent or cross-modal user inputs. (2) Interaction robustness: User input is highly diversified in semantics and expressive in randomness, and existing models are prone to recognition errors in boundary scenarios and non-standard inputs. (3) Resource constraints: Limited computing and storage resources on the device side require efficient model compression and parameter optimization methods for deploying large models.

[0004] To address the above problems, reinforcement learning (RL) methods are introduced into the training process of the intent recognition model to improve the model's performance in real-world environments using user feedback as an optimization signal. Meanwhile, parameter-efficient fine-tuning techniques such as LoRA (Low-Rank Adaptation) can quickly customize the model for specific operating system tasks without significantly increasing the model's parameter size. The reinforcement learning method of GRPO (Group Relative Policy Optimization) can better align the model output with user needs while maintaining model stability. Therefore, how to combine the characteristics of the intent recognition task of the domestic operating system, use LoRA parameter-efficient fine-tuning and GRPO reinforcement learning training to achieve efficient, accurate and deployable model optimization has become a pressing problem. Although existing intent recognition technologies have been widely used in typical scenarios such as voice assistants, their adaptability in the field of domestic operating system control is still insufficient. Rule-based or traditional machine learning methods rely on manually designed rule libraries or shallow models (such as SVM, CRF), have poor scalability, are difficult to cover complex and diverse system instructions, and have low accuracy when facing ambiguous or composite intents. End-to-end deep learning methods improve recognition accuracy, but they rely heavily on large-scale labeled data and have limited generalization ability, and consume too much computing resources when deployed on terminals. Transfer learning methods based on pre-trained large language models (such as BERT, GPT) have improved generalization ability, but the customization process requires full parameter fine-tuning, which is costly and storage-intensive, making it difficult to adapt quickly on the operating system side. Parameter-efficient fine-tuning methods (such as LoRA) have alleviated the fine-tuning overhead problem to some extent, but existing methods are still limited to supervised training and lack effective use of user real preferences and interaction feedback. Overall, existing technologies have obvious shortcomings in intent and slot label system coverage, training efficiency, end-side adaptability, and alignment with user needs, making it difficult to fully support diverse operating system function calls. SUMMARY

[0005] The technical problems solved by the present application: In view of the above problems in the prior art, the present application provides a reinforcement learning training method and system for an intent recognition model for operating system control. The present application aims to overcome the problems of insufficient coverage of intent and slot label system, high data quality requirements but difficult to obtain, high customization cost of the model, and lack of user feedback-driven optimization in the prior art, and to improve the speed and accuracy of intent recognition for operating systems on end-side devices.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: A system control intention recognition model reinforcement learning training method, comprising the following steps: S101, constructing an intention and slot label for a natural language command for system control, wherein the intention and slot label comprises an intention and a slot array for system control, and the slot array comprises a control parameter for system control; S102, constructing a training data set according to the natural language command and the corresponding intention and slot label; S103, using LoRA technology to supervise and fine-tune the pre-trained intention recognition model based on the training data set to obtain a cold-start intention recognition model; S104, using a reinforcement learning method introducing a group relative strategy optimization to perform reinforcement learning training on the cold-start intention recognition model, thereby obtaining a final intention recognition model that can be used for end-side deployment.

[0007] Optionally, when constructing the intention and slot label for the natural language command for system control in step S101, the intention in the intention and slot label comprises a primary intention and a secondary intention, which are obtained by performing semantic abstraction according to the function modules of the operating system, the primary intention corresponds to a primary function module of the operating system, and the secondary intention corresponds to a function module under the primary function module of the operating system.

[0008] Optionally, when constructing the training data set according to the natural language command and the corresponding intention and slot label in step S102, the training samples in the training data set comprise artificial fine-labeled samples, semi-automatic labeled samples, and multi-round interactive enhanced samples, and the construction of the training data set comprises: S201, constructing an artificial fine-labeled sample composed of a natural language command and an artificially labeled intention and slot label corresponding to the natural language command; S202, performing template expansion and synonym expression replacement based on the natural language command in the artificial fine-labeled sample, thereby obtaining a semi-automatic labeled sample composed of a new natural language command and an original intention and slot label of the new natural language command; S203, constructing a multi-round interactive enhanced sample by adding context dialogues, user follow-up examples, and intention superposition for the artificial fine-labeled sample and the semi-automatic labeled sample, wherein the multi-round interactive enhanced sample comprises a multi-round multi-intention interactive sample and a multi-round single-intention interactive sample, the multi-round multi-intention interactive sample is composed of natural language commands of multiple intentions, context dialogues thereof, and original intentions and slot labels thereof, and the multi-round single-intention interactive sample is composed of natural language commands of a single intention, context dialogues thereof, user follow-up examples, and original intentions and slot labels thereof.

[0009] Optionally, when using LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset in step S103, the output of the cold-start intent recognition model includes the intent, the slot array, and the follow-up output. The follow-up output is used to provide feedback to the user on the missing slot information of the slot array in order to re-acquire the natural language command input by the user.

[0010] Optionally, in step S104, when using a reinforcement learning method that incorporates group-relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows: ; in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The reward for correct follow-up questions is as follows: the format correctness reward includes whether the output format of the intent recognition model is correct, and the scores for whether the intent, slot array, and follow-up question output exist in the output of the intent recognition model; the intent accuracy reward is the score corresponding to the cosine similarity between the intent and intent label in the output of the intent recognition model; the slot accuracy reward includes the scores corresponding to whether the slot array is extracted, whether the intent contains slots, and whether the slot information in the slot array is correct; the follow-up question correctness reward is the score of the correlation between the follow-up question output and the missing slot information.

[0011] Optionally, when using a reinforcement learning method that introduces group relative policy optimization to train the cold-start intention recognition model in step S104, and updating the parameters of the intention recognition model using backpropagation of the gradient of the loss function during reinforcement learning training, the expression for the gradient of the loss function is as follows: ; in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... Let KL divergence be the KL divergence, and we have: ; ; ; ; in, To obtain the minimum value, As an uncut alternative target, As a replacement target after cropping, Generate lexical units for the two rounds of intent recognition model The probability ratio, as a word element The advantage function represents the lexical generation by the intent recognition model. The reward obtained is based on the training sample. The difference between the reward and the average reward of all training samples; For the clipping function, For trimming parameters, Generate lexical units for the intent recognition model in the ideal model The probability, Generate lexical units for the intent recognition model under the current parameters. The probability of.

[0012] Optionally, the loss function is expressed as follows: ; ; in, For loss function, For the mathematical expectation value, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, To cut off alternative targets, The penalty coefficient is... Let KL divergence be denoted as KL divergence.

[0013] The present invention also provides an operating system-controlled intention recognition model reinforcement learning training system, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method.

[0014] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute, via a processor, an intention recognition model reinforcement learning training method controlled by the operating system.

[0015] The application also provides a computer program product comprising a computer program or instructions programmed or configured to perform the operating system controlled intent recognition model reinforcement learning training method by a processor.

[0016] Compared with the prior art, the application mainly has the following beneficial effects: the operating system controlled intent recognition model reinforcement learning training method comprises constructing an intent and a slot label for a natural language command for operating system control, constructing a training data set according to the natural language command and the corresponding intent and slot label; using the LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training data set to obtain a cold-start intent recognition model; using a reinforcement learning method introducing a group relative strategy optimization to perform reinforcement learning training on the cold-start intent recognition model, thereby obtaining a final intent recognition model that can be used for end-side deployment. The application can overcome the problems of insufficient coverage of the intent and slot label system, high data quality requirements but difficult to obtain, high model customization cost and lack of user feedback driven optimization in the prior art, and improve the speed and accuracy of operating system intent recognition on end-side devices. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The basic flowchart of the method of the embodiment of the application is shown.

[0018] Figure 2 The calculation flowchart of the model output format reward in the embodiment of the application is shown.

[0019] Figure 3 The calculation flowchart of the intent correctness reward in the embodiment of the application is shown.

[0020] Figure 4 The calculation flowchart of the slot correctness reward in the embodiment of the application is shown.

[0021] Figure 5 The calculation flowchart of the follow-up question correctness reward in the embodiment of the application is shown.

[0022] Figure 6 The calculation flowchart of the word-level reward in the embodiment of the application is shown. DETAILED DESCRIPTION

[0023] In order to enable personnel in the technical field to better understand the technical solutions of the application, the technical solutions of the application will be further described in detail below in combination with the drawings in the embodiments of the application.

[0024] In order to overcome the problems of insufficient coverage of the intent and slot label system, high data quality requirements but difficult to obtain, high model customization cost and lack of user feedback driven optimization in the prior art, the application constructs an intent and a slot label for a natural language command for operating system control, constructs a training data set according to the natural language command and the corresponding intent and slot label, uses the LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training data set to obtain a cold-start intent recognition model, and uses a reinforcement learning method introducing a group relative strategy optimization to perform reinforcement learning training on the cold-start intent recognition model, thereby obtaining a final intent recognition model that can be used for end-side deployment.Figure 1 As shown, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment includes the following steps: S101, construct intents and slot labels for natural language commands used for operating system control, wherein the intents and slot labels include intents and slot arrays used for operating system control, and the slot array contains control parameters for operating system control. S102, construct a training dataset based on natural language commands and their corresponding intents and slot labels; S103 uses LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model; S104. A reinforcement learning method with relative policy optimization is used to train the cold-start intent recognition model, thereby obtaining the final intent recognition model that can be used for edge deployment.

[0025] In step S101, a systematic analysis of the operating system's functional modules is performed, abstracting common system operations into multi-level intents and designing corresponding slot label sets. This label system covers explicit instructions (such as "turn on Bluetooth") and ambiguous instructions (such as "light up"), thereby ensuring that the model can fully understand and parse diverse user inputs in operating system scenarios. Specifically, in step S101 of this embodiment, when constructing intents and slot labels for natural language commands used for operating system control, the intents in the intents and slot labels include primary intents and secondary intents. The primary intents and secondary intents are obtained by semantic abstraction based on the operating system's functional modules. Primary intents correspond to the primary functional modules of the operating system, and secondary intents correspond to the functional modules under the primary functional modules of the operating system. Specifically, in this embodiment, in the typical interaction scenario of the Galaxy Kylin operating system, the natural language commands input by users usually involve multiple types of operation intents and parameter information. To achieve systematic modeling, the functional modules of the operating system are first semantically abstracted, and the functional behaviors are divided into two levels of intent labels. The primary intents contain 8, and 87 secondary intent labels are derived from the primary intents, as shown in Table 1 for some examples.

[0026] Table 1: Examples of Intent and Slot Labels

[0027] Taking the second intent and slot labels as an example, the first-level intent is "System Operation," and the second-level intent is "Switch Display Projection Mode." The slot array includes four slots: "Name," "Type," "Required," and "Description." Each slot in the slot array can be empty, and the number can be configured according to the intent's needs. For each second-level intent, this invention designs a set of slot labels to extract parameter information involved in the command, such as the target application, the operation object, and status values. The label system adopts a hierarchical design, enabling the model to adapt to various expression forms such as explicit commands, fuzzy commands, and compound commands, ensuring comprehensive coverage of operating system control tasks.

[0028] After establishing the intent and slot label system, this invention constructs a high-quality dataset based on actual operating system instruction scenarios. Building upon the intent and slot label system constructed in step S101, step S102 of this embodiment constructs a high-quality training dataset covering typical operating system instructions through a combination of manual annotation and semi-automatic generation. The dataset contains rich expression variations, contextual scenarios, and multi-turn interaction examples, ensuring the model's robustness in the face of diverse inputs. Simultaneously, this embodiment emphasizes the principle of "small but precise" data construction, using small batches of high-quality data instead of large-scale low-quality corpora to improve training efficiency and model generalization ability. Specifically, in step S102 of this embodiment, when constructing the training dataset based on natural language commands and their corresponding intents and slot labels, the training samples in the training dataset include manually labeled samples, semi-automatically labeled samples, and multi-turn interaction enhancement samples, and the construction of the training dataset includes: S201, construct artificially labeled samples consisting of natural language commands and their corresponding manually annotated intents and slot labels; S202, based on template expansion and synonym replacement, natural language commands in manually labeled samples are used to obtain semi-automatic labeled samples consisting of new natural language commands, their original intent and slot labels; S203 consists of manually labeled samples and semi-automatically labeled samples. Multi-turn interaction enhancement samples are constructed by adding contextual dialogues, user follow-up questions, and intent overlays. These multi-turn interaction enhancement samples include multi-turn multi-intent interaction samples and multi-turn single-intent interaction samples. The multi-turn multi-intent interaction samples consist of natural language commands for multiple intents, their contextual dialogues, their original intents, and slot labels. The multi-turn single-intent interaction samples consist of natural language commands for a single intent, their contextual dialogues, user follow-up questions, their original intents, and slot labels. The final dataset contains various representations and reaches thousands of data points, with a focus on ensuring quality and representativeness to provide efficient supervision signals for subsequent model training.

[0029] In step S103 of this embodiment, when the LoRA technique is used to supervise the fine-tuning of the pre-trained intent recognition model based on the training dataset, the output of the cold-start intent recognition model includes the intent, the slot array, and the follow-up output. The follow-up output is used to provide feedback to the user on the missing slot information of the slot array to re-acquire the natural language command input by the user. Addressing the issue of limited computing and storage resources on the operating system's edge devices, step S103 of this embodiment selects the pre-trained large language model Qwen3-1.7b, which possesses good language understanding capabilities, as the pre-trained large language model. Based on the pre-trained large language model, the Low-Rank Adaptation (LoRA) method is used for efficient parameter fine-tuning. Unlike full parameter fine-tuning, the LoRA method only needs to update some low-rank matrix parameters in the model, avoiding the storage and computing overhead caused by full parameter fine-tuning. This significantly reduces the number of trainable parameters and GPU memory usage, thereby quickly obtaining a cold-start model adapted to the operating system control scenario. In this way, the model can run efficiently on the edge, possessing fast inference speed and low storage usage, exhibiting good inference speed and low resource requirements on edge devices.

[0030] To further improve the model's performance in real user interactions, step S104 of this embodiment introduces the Group Relative Policy Optimization (GRPO) method based on the cold-start model. This method utilizes user feedback signals for reinforcement learning training. By constructing a reward function, it replaces the traditional reinforcement learning method's reliance on absolute reward signals provided by the environment with "relative merit within the group" to evaluate policy effectiveness. This guides the intent recognition model to generate intent and slot parsing results that better meet user needs under diverse user inputs. GRPO improves training stability during the optimization process, avoiding gradient oscillations or convergence difficulties common in traditional reinforcement learning methods, thereby further improving the model's accuracy and robustness in actual interactions. Specifically, in step S104 of this embodiment, when using the reinforcement learning method with introduced group relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows: ; in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The correctness reward for follow-up questions; wherein, the format correctness reward includes whether the output format of the intent recognition model is correct, and whether the output of the intent recognition model contains intent, slot array, and follow-up question output scores; for example, as an optional implementation, such as Figure 2 As shown, the calculation of the format correctness bonus includes: Step 1: Determine if the output of the intent recognition model is in JSON format. If it is, increase the format correctness reward by 0.5 points and proceed to Step 2; otherwise, the format correctness reward is 0 points, and the process ends. Step 2: Determine the number of intents in the output of the intent recognition model. If the number of intents is 1, the format correctness reward increases by 0.3 points, and proceed to Step 3; otherwise, the format correctness reward is 0 points, and the process ends. Step 3: Determine the number of follow-up questions output by the intent recognition model. If the number of follow-up questions is 1, the format correctness reward increases by 0.3 points, and the process proceeds to Step 3; otherwise, the format correctness reward is 0 points, and the process ends. Step 4: Determine the number of slot arrays in the intent recognition model. If the number of slot arrays is 1, the format correctness reward increases by 0.3 points, and the process jumps to step 3; otherwise, the format correctness reward is 0 points, and the process ends.

[0031] The intent accuracy reward is the score corresponding to the cosine similarity between the intent and the intent label in the output of the intent recognition model; for example, as an optional implementation, such as Figure 3 As shown, the calculation of the intent accuracy reward includes: Step 1: Analyze the intent in the output of the intent recognition model; Step 2: Calculate the cosine similarity between the intent and the labeled information; Step 3: If the cosine similarity is greater than or equal to 0.9, the intention accuracy reward is 1; if the cosine similarity is greater than or equal to 0.8 and less than 0.9, the intention accuracy reward is 0.5; if the cosine similarity is less than 0.8, the intention accuracy reward is 0.

[0032] The slot accuracy reward includes scores for whether a slot array was extracted, whether the intent contains slots, and whether the slot information in the slot array correctly corresponds to the desired slot. For example, as an optional implementation, such as... Figure 4 As shown, the calculation of slot accuracy bonus includes: Step 1: Determine the score of the intent accuracy reward. If the intent accuracy reward is 0, then the slot accuracy reward is 0, end and return; otherwise, proceed to step 2. Step 2: Analyze the array size of slot bit groups in the output of the intent recognition model; Step 3: Determine if the array size of the slot number groups is 0. If it is 0, proceed to step 4; otherwise, proceed to step 6. Step 4: Analyze the intent type in the output of the intent recognition model. If the intent type is a slotless intent, the intent accuracy reward is 1, and the process ends and returns; otherwise, proceed to step 5. Step 5: Determine if there is slot information in the input of the intent recognition model. If there is slot information, the intent accuracy reward is -1 as a penalty; otherwise, the intent accuracy reward is 1; end and return. Step 6: Analyze the intent type of the intent in the output of the intent recognition model. If the intent type is a slotless intent, the intent accuracy reward is -0.2, and the process ends and returns; otherwise, proceed to step 7. Step 7: Determine whether there is slot information in the input of the intent recognition model. If there is no slot information, the intent accuracy reward is -0.1, and the process ends and returns; otherwise, proceed to step 8. Step 8: Determine if the slot array has been completely extracted. If the slot array has been completely extracted, the intent accuracy reward is 1; otherwise, the intent accuracy reward is 0.5. Perform a validity check on each slot in the slot array. If valid, update the slot accuracy reward according to the following formula: ; Otherwise, update the slot accuracy reward according to the following formula (no reward will be given if the slot value is unreasonable, and the reward score will be increased by 0): ; in," "An operator added to the original value, The size of the array of slots.

[0033] As an optional implementation, the correctness of the slot information in the slot array is determined by the inclusion of natural language commands and training samples. The intent label, the target slot name, the target slot requirements, and the values ​​of the slot information in the slot array are combined with the preset prompt words of the task instruction containing whether the slot information in the slot array is correct and input into the large language model to obtain the result.

[0034] The correctness reward for follow-up questions is a score reflecting the relevance of the follow-up question output to the missing slot information. As an optional implementation method, such as... Figure 5 As shown, the calculation of the reward for correct follow-up questions includes: Step 1: parse the slot array in the output of the intent recognition model, and determine whether there are any required slots. If there are no required slots, the correctness reward for follow-up questions is 0, and the process ends and exits; otherwise, proceed to Step 2. Step 2: Check the number of missing required slots in the slot array. If the number of missing required slots is 0, the correctness reward for follow-up questioning is 0, and the process ends and exits; otherwise, proceed to step 3. Step 3: Analyze the follow-up question output in the intent recognition model. If the follow-up question output string is empty, the follow-up question correctness reward is -0.5, and the process ends and exits; otherwise, proceed to step 4. Step 4: Evaluate the correctness of the follow-up question output using a large language model. If the evaluation result is correct, the follow-up question correctness reward is 0.5; otherwise, the follow-up question correctness reward is 0. In this embodiment, evaluating the correctness of the follow-up question output using a large language model includes processing the training samples... The intent label, the name of the missing slot, and the follow-up question output are combined with the task instruction prompt words that ask whether the missing slot information in the slot array can be filled. After inputting these into the large language model, the output is either "yes" or "no", indicating whether the evaluation result is correct.

[0035] like Figure 6 As shown, the word reward score calculation method of the intent recognition model in this embodiment is as follows: Initialize the reward scores of all words, and more accurately allocate the format reward score, intent reward score, slot reward score and follow-up question reward score to each word to obtain the score of each word in the output sequence. In this way, the abnormality of the score of one item in the sequence will not affect the score of other items. Each word has an independent score, which can realize the accurate transmission of gradient signals.

[0036] In step S104 of this embodiment, when the cold-start intention recognition model is trained using a reinforcement learning method that incorporates group relative policy optimization, the gradient of the loss function is used to backpropagate and update the parameters of the intention recognition model during reinforcement learning training. The expression for the gradient of the loss function is as follows: ; in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... The gradient described above, representing the KL divergence, consists of two parts: a reward-driven term to maximize the reward and a KL penalty term to limit deviation and ensure the stability of the model's training. The optimized model can more accurately identify user intent and maintain high recognition consistency across scenarios and multiple commands.

[0037] ; ; ; ; in, To take the minimum value means choosing a more conservative estimate. As an uncut alternative target, As a replacement target after cropping, For the current policy in state Select action The probability of the old strategy in the state Select action The ratio of the probabilities, The advantage function for each word represents the advantage of each word generated by the intent recognition model and evaluates the merits of the current action. For the clipping function, For trimming parameters, This represents the probability that the policy model (i.e., the trained model) generates a sequence of lexical terms before the policy is updated. The probability of the policy model generating a sequence of terms after the policy update can be obtained during the model's forward propagation. The advantage function for each term is expressed as follows: ; in, The reward for the word at position t. The average reward within the group. The within-group standard deviation To prevent decimals from being divided by zero.

[0038] In this embodiment, the function expression of the loss function is: ; ; in, For loss function, For the mathematical expectation value, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, As a substitute target, The penalty coefficient is... The KL divergence is used. The completeness mask is the generated word mask used to distinguish which words need to be calculated for loss. The completeness mask is a 0-1 sequence corresponding to the word. 0 represents invalid words that are not included in the calculation, and 1 represents valid words that are included in the calculation. Through the above training process, the obtained intent recognition model has the following characteristics: (1) Lightweight: LoRA fine-tuning greatly reduces parameter storage and computation, making it suitable for deployment in terminal devices or operating system kernels; (2) High precision: Based on high-quality datasets and reinforcement learning optimization, the model performs stably in complex intent parsing tasks; (3) Strong scalability: The label system and training framework can be extended according to the new system functions, and can quickly adapt to different operating system versions and device types.

[0039] To verify the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment, relevant experiments were conducted. Experimental results show that, under the same hardware conditions, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment reduces model parameters by approximately 80%, reduces edge inference latency by approximately 40%, and improves accuracy by approximately 10%–15% on multi-scenario intent recognition tasks within the operating system. Therefore, the reinforcement learning training method for the intent recognition model controlled by the operating system in this embodiment effectively improves the natural language interaction capabilities and user experience of the operating system.

[0040] In summary, addressing the diverse, ambiguous, and cross-application characteristics of user natural language commands in operating system control scenarios, this embodiment's reinforcement learning training method for the intent recognition model in operating system control comprehensively utilizes key technologies such as label system construction, dataset construction, efficient parameter fine-tuning (LoRA), and group relative policy (GRPO) optimization. This method gradually overcomes the shortcomings of existing technologies in terms of coverage, training cost, adaptation efficiency, and alignment with user needs. It not only achieves efficient training under conditions of small batches of high-quality manually labeled data but also improves on-device adaptation efficiency through LoRA and effectively utilizes user feedback for optimization in conjunction with GRPO. This results in fast inference speed, high recognition accuracy, and strong robustness of intent recognition capabilities on on-device devices, significantly improving the natural language interaction experience of domestic operating systems. The intent recognition model obtained through the training path of "systematic intent labels + high-quality small datasets + LoRA fine-tuning + GRPO optimization" in this embodiment not only has fast inference speed and low resource consumption but also maintains high recognition accuracy and robustness under complex and varied user input, thereby significantly improving the natural language interaction experience of operating systems. This embodiment of the reinforcement learning training method for the intent recognition model under operating system control combines LoRA (Locally Aligned Parameters) with GRPO-optimized reinforcement learning in the context of operating system control on the edge side. This addresses the generalization, lightweighting, and interaction adaptability issues of the intent recognition model, achieving the following significant technical effects: (1) Significantly improves the coverage and accuracy of intent recognition under operating system control. This invention constructs a systematic intent and slot label system for operating system control scenarios, covering system-level operations, application-level control, and fuzzy intent scenarios, effectively improving the model's understanding of diverse system function calls. Compared to traditional voice assistant-style intent recognition systems, the recognition accuracy is improved by approximately 10% to 20%. (2) Reduces model fine-tuning costs and deployment resource overhead. By adopting LoRA, only a small number of low-rank weights need to be adjusted on the pre-trained model, significantly reducing the number of training parameters and storage overhead. Compared to full fine-tuning, the number of training parameters can be reduced by more than 90%, significantly improving the rapid adaptation and updating capabilities of the edge model. (3) Reinforcement learning-driven adaptive model optimization mechanism. This invention introduces a reinforcement learning method based on GRPO optimization, which uses the scoring mechanism of intent recognition results and interactive feedback information to dynamically optimize the model strategy, enabling the model to actively initiate follow-up questions or clarifications when faced with ambiguous, complex, or undefined intents, thereby improving user interaction experience and task completion rate. (4) High-quality training and generalization under small sample conditions. By combining manually labeled data with high-quality data augmentation strategies, the cold-start model can complete high-precision intent recognition with limited samples. GRPO training further enhances the robustness of the model under different user expression methods, solving the problem of traditional supervised learning's dependence on large-scale labeled data.

[0041] Furthermore, this embodiment also provides an operating system-controlled intention recognition model reinforcement learning training system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the operating system-controlled intention recognition model reinforcement learning training method via a processor.

[0042] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0043] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A reinforcement learning training method for an intent recognition model controlled by an operating system, characterized in that, Includes the following steps: S101, construct intents and slot labels for natural language commands used for operating system control, wherein the intents and slot labels include intents and slot arrays used for operating system control, and the slot array contains control parameters for operating system control. S102, construct a training dataset based on natural language commands and their corresponding intents and slot labels; S103 uses LoRA technology to supervise and fine-tune the pre-trained intent recognition model based on the training dataset to obtain a cold-start intent recognition model; S104. A reinforcement learning method with relative policy optimization is used to train the cold-start intent recognition model, thereby obtaining the final intent recognition model that can be used for edge deployment.

2. The reinforcement learning training method for the intent recognition model of operating system control according to claim 1, characterized in that, In step S101, when constructing intents and slot labels for natural language commands used for operating system control, the intents in the intents and slot labels include primary intents and secondary intents. The primary intents and secondary intents are obtained by semantic abstraction based on the functional modules of the operating system. The primary intents correspond to the primary functional modules of the operating system, and the secondary intents correspond to the functional modules under the primary functional modules of the operating system.

3. The reinforcement learning training method for the intent recognition model of operating system control according to claim 1, characterized in that, In step S102, when constructing the training dataset based on natural language commands and their corresponding intents and slot labels, the training samples in the training dataset include manually labeled samples, semi-automatically labeled samples, and multi-round interactive enhancement samples. The construction of the training dataset includes: S201, construct artificially labeled samples consisting of natural language commands and their corresponding manually annotated intents and slot labels; S202, based on template expansion and synonym replacement, natural language commands in manually labeled samples are used to obtain semi-automatic labeled samples consisting of new natural language commands, their original intent and slot labels; S203 consists of manually labeled samples and semi-automatically labeled samples. It is composed of multi-turn interaction enhancement samples by adding contextual dialogue, user follow-up questions, and intent superposition. The multi-turn interaction enhancement samples include multi-turn multi-intent interaction samples and multi-turn single-intent interaction samples. The multi-turn multi-intent interaction samples consist of natural language commands for multiple intents, their contextual dialogues, their original intents, and slot labels. The multi-turn single-intent interaction samples consist of natural language commands for a single intent, their contextual dialogues, user follow-up questions, their original intents, and slot labels.

4. The reinforcement learning training method for the intent recognition model of operating system control according to claim 1, characterized in that, In step S103, when LoRA technology is used to supervise and fine-tune the pre-trained intent recognition model based on the training dataset, the output of the cold-start intent recognition model includes intent, slot array, and follow-up output. The follow-up output is used to provide feedback to the user on the missing slot information of the slot array in order to re-acquire the natural language command input by the user.

5. The reinforcement learning training method for the intent recognition model of operating system control according to claim 4, characterized in that, In step S104, when using a reinforcement learning method that incorporates group-relative policy optimization to train the cold-start intent recognition model, the function expression of the reward function is as follows: ； in, For training samples The reward For training samples Rewards for correct formatting For training samples Rewards based on the accuracy of intent. For training samples Slot accuracy reward For training samples The reward for correct follow-up questions is as follows: the format correctness reward includes whether the output format of the intent recognition model is correct, and the scores for whether the intent, slot array, and follow-up question output exist in the output of the intent recognition model; the intent accuracy reward is the score corresponding to the cosine similarity between the intent and intent label in the output of the intent recognition model; the slot accuracy reward includes the scores corresponding to whether the slot array is extracted, whether the intent contains slots, and whether the slot information in the slot array is correct; the follow-up question correctness reward is the score of the correlation between the follow-up question output and the missing slot information.

6. The reinforcement learning training method for the intent recognition model of operating system control according to claim 5, characterized in that, In step S104, when the cold-start intention recognition model is trained using a reinforcement learning method that incorporates group-relative policy optimization, the gradient of the loss function is used for backpropagation to update the parameters of the intention recognition model during reinforcement learning training. The expression for the gradient of the loss function is as follows: ； in, for about The partial derivatives, To cut off alternative targets, for about The partial derivatives, For loss function, For the parameters of the intent recognition model, The penalty coefficient is... Let KL divergence be the KL divergence, and we have: ； ； ； ； in, To obtain the minimum value, As an uncut alternative target, As a replacement target after cropping, Generate lexical units for the two rounds of intent recognition model The probability ratio, as a word element The advantage function represents the lexical generation by the intent recognition model. The reward obtained is based on the training sample. The difference between the reward and the average reward of all training samples; For the clipping function, For trimming parameters, Generate lexical units for the intent recognition model in the ideal model The probability, Generate lexical units for the intent recognition model under the current parameters. The probability of.

7. The reinforcement learning training method for the intent recognition model of operating system control according to claim 6, characterized in that, The function expression of the loss function is: ； ； in, For loss function, For mathematical expectation, As a word element, as a word element Gradient regularization loss, as a word element Valid word mask, To cut off alternative targets, The penalty coefficient is... Let KL divergence be denoted as KL divergence.

8. A reinforcement learning training system for an intent recognition model controlled by an operating system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the intention recognition model reinforcement learning training method controlled by the operating system of any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the intention recognition model reinforcement learning training method controlled by the operating system as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the intention recognition model reinforcement learning training method controlled by the operating system as described in any one of claims 1 to 7.

Images