Model training method and device based on reinforcement learning

Abstract

The embodiment of the invention provides a model training method and device based on reinforcement learning. The method comprises the steps that a user instruction and an equipment state are obtained, the user instruction is used for indicating equipment to execute a target task, the target task comprises a plurality of operation steps, and the equipment state is the current working state of the equipment; the user instruction and the equipment state are input into a strategy model, a plurality of output results are generated through the strategy model, and each output result comprises a prediction action instruction used for executing the same operation step; determining a group success rate based on the plurality of output results, the group success rate being used for representing a ratio of the output results for correctly executing the operation step among the plurality of output results; calculating a reward signal corresponding to the output result based on the group success rate; and updating parameters of the strategy model through a reinforcement learning algorithm based on the reward signal corresponding to each output result.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of machine learning technology, and more specifically, to a model training method and apparatus based on reinforcement learning. Background Technology

[0002] Reinforcement learning (RL), as an important branch of machine learning, has made significant progress in both theory and application in recent years. In RL, an agent interacts with the environment and learns optimal behavioral policies based on reward signals from the environment, aiming to maximize long-term cumulative rewards. However, currently widely used policy optimization algorithms, such as GRPO (Group Relative Policy Optimization) and PPO (Proximal Policy Optimization), are highly sensitive to the quality of reward signals. In high-dimensional, dynamic, and reward-sparse task environments, inappropriate reward signal design can easily lead to unstable model training.

[0003] Therefore, there is an urgent need for a high-quality reward signal design scheme to achieve more stable and efficient reinforcement learning training. Summary of the Invention

[0004] This specification provides a reinforcement learning-based model training scheme that calculates reward signals using the group success rate, thereby providing more stable reward signals and reducing uncertainty during model training.

[0005] In a first aspect, embodiments of this specification provide a model training method based on reinforcement learning, comprising: acquiring user instructions and device status, wherein the user instructions are used to instruct the device to perform a target task, the target task includes several operation steps, and the device status is the current working state of the device; inputting the user instructions and the device status into a policy model, wherein the policy model generates multiple output results, each output result including a predicted action instruction for performing the same operation step; determining a group composition power based on the multiple output results, wherein the group composition power is used to represent the ratio of the output results that correctly execute the operation step among the multiple output results; calculating a reward signal corresponding to the output result based on the group composition power; and updating the parameters of the policy model using a reinforcement learning algorithm based on the reward signal corresponding to each output result.

[0006] In some optional implementations, calculating the reward signal corresponding to the output result based on the group composition power includes: determining the task complexity of the operation step based on the group composition power, wherein the task complexity is negatively correlated with the group composition power; and calculating the reward signal corresponding to the output result based on the task complexity and the output result.

[0007] In some optional implementations, determining the task complexity of the operation step based on the group composition power includes: determining the complexity of the operation step as low complexity in response to the group composition power being higher than a preset threshold; and determining the complexity of the operation step as high complexity in response to the group composition power not being higher than the preset threshold.

[0008] In some optional implementations, the output result further includes the inference path from which the policy model derives the predicted action instruction. The step of calculating the reward signal corresponding to the output result based on the task complexity and the output result includes: determining the first reward signal corresponding to the output result using a first reward function based on the task complexity and the number of tokens in the inference path; wherein, when the task complexity is low, the number of tokens in the inference path is negatively correlated with the first reward signal, and when the task complexity is high, the number of tokens in the inference path is positively correlated with the first reward signal.

[0009] In some optional implementations, calculating the reward signal corresponding to the output result based on the task complexity and the output result includes: determining the decay rate of a second reward function based on the task complexity, wherein the decay rate is negatively correlated with the task complexity; calculating a second reward signal based on the difference between the predicted action instruction and the target action instruction for performing the operation step in each output result, wherein the second reward signal gradually decays according to the decay rate as the difference between the predicted action instruction and the target action instruction increases.

[0010] In some optional implementations, the second reward function is a Gaussian function, and the higher the task complexity, the greater the bandwidth of the Gaussian function.

[0011] In some optional implementations, updating the parameters of the policy model using a reinforcement learning algorithm based on the reward signals corresponding to each of the output results includes: inputting the user command and the device state into a baseline model, generating a baseline output result by the baseline model, wherein the baseline model is obtained by supervised fine-tuning of the policy model before training; determining a penalty term based on the difference between the baseline output result and the output result; and updating the parameters of the policy model using a reinforcement learning algorithm based on the reward signals corresponding to each of the output results and the penalty term.

[0012] In some optional implementations, determining the group composition power based on multiple output results includes: obtaining a target action instruction for performing the operation steps; and comparing each output result with the target action instruction to determine the group composition power.

[0013] In some optional implementations, calculating the reward signal corresponding to the output result based on the group composition power includes: determining a reward adjustment factor based on the group composition power; and calculating the reward signal corresponding to the output result based on the output result and the reward adjustment factor.

[0014] In some alternative implementations, the device has a graphical user interface, and the device state is the current display state of the graphical user interface.

[0015] Secondly, embodiments of this specification provide a model training apparatus based on reinforcement learning, comprising:

[0016] The acquisition module is used to acquire user instructions and device status. The user instructions are used to instruct the device to perform a target task, which includes several operation steps. The device status is the current working state of the device.

[0017] The inference module is used to input the user instructions and the device status into the policy model, and the policy model generates multiple output results, each output result including a predicted action instruction for performing the same operation step;

[0018] A comparison module is used to determine the group composition power based on multiple output results, wherein the group composition power represents the ratio of the output results that correctly execute the operation steps among the multiple output results;

[0019] The reward module is used to calculate the reward signal corresponding to the output result based on the group composition power.

[0020] The parameter tuning module is used to update the parameters of the policy model based on the reward signal corresponding to each of the output results using a reinforcement learning algorithm.

[0021] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0022] In the scheme provided by the above embodiments of this specification, during the training process of the policy model, after the user command is input into the policy model, the policy model will output multiple output results. These multiple output results can be regarded as a group of candidate schemes. The overall performance of this group of candidate schemes is measured by evaluating the group composition power of the multiple output results. Then, the reward signal is calculated by comprehensively considering the group composition power and multiple output results. This reward signal calculation method takes into account multiple output results and can effectively smooth the reward noise caused by the randomness of a single output result, thereby providing a more stable and reliable reward signal for the update of the policy model, reducing the uncertainty in the training process, and making the finally trained model policy more robust and generalizable. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the reinforcement learning process in the embodiments of this specification;

[0025] Figure 2 This is a flowchart illustrating a model training process in one embodiment of this specification.

[0026] Figure 3 This is a flowchart of a model training method based on reinforcement learning in the embodiments of this specification;

[0027] Figure 4 This is a schematic diagram of a second reward function in one of the embodiments of this specification;

[0028] Figure 5 This is a schematic diagram of the structure of a model training device based on reinforcement learning, as described in the embodiments of this specification. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0030] To address the training instability problem caused by improper reward signal design in the existing technology, this embodiment calculates the reward signal using the group composition power determined by multiple outputs of the policy model, and then uses this reward signal for reinforcement learning training of the model, thereby alleviating the training instability problem. To more clearly describe the reinforcement learning-based model training scheme in this embodiment, the relevant content of reinforcement learning is first introduced.

[0031] Reinforcement learning is a machine learning method that aims to optimize decision-making processes through the interaction between a model and its environment to maximize cumulative rewards. The process involves: an agent, a decision-making entity capable of perceiving the state of the environment, taking actions, and interacting with the environment to maximize cumulative rewards; a policy model (used to determine action choices) and a value function (used to evaluate the merits of states or actions), with the policy model being the core component and potentially a machine learning model; the environment, the external system in which the agent operates, which it learns through interaction; the state, the specific condition of the environment at a given moment, upon which the agent makes decisions; the action, the behavior the agent can choose in a specific state; and the reward, the feedback from the environment to the agent's action, typically a numerical value representing the merits of that action.

[0032] In the embodiments of this specification, the policy model can be a Large Language Model (LLM), also simply referred to as a large model. A large language model is a natural language processing model based on deep learning technology, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large language models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to achieve global modeling of sequential data, efficiently handling long-distance dependencies and thus performing excellently in natural language tasks. Large language models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of large language models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the large language model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0033] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0034] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0035] like Figure 1 In the reinforcement learning process shown, the agent obtains the current state s at the current time step t from the environment. t According to strategy π θ (a t |s t ) Decide on the action to be taken a t The action a t After acting on the environment, the environment transitions to the next state s. t+1 and return a reward signal r to the agent. t+1 This process is repeated continuously. Based on the trajectory collected during the agent's interaction with the environment (including the state, actions, and reward signals at each time step), the agent's parameters (mainly the model parameters of the policy model) are updated using reinforcement learning algorithms (such as PPO) to estimate the reward or advantage function, thereby improving the policy. By repeatedly observing the state, selecting actions, and receiving rewards, the agent continuously optimizes its policy, with the goal of learning a policy that maximizes the cumulative reward.

[0036] Currently, there are various reinforcement learning methods that can be used for agent policy learning and optimization, such as policy gradient-based methods, such as policy gradient methods based on reward increments, AC (Actor-Critic), PPO, Q-learning, and GRPO based on group relative advantage. Any reinforcement learning algorithm in the prior art can be used in the schemes of the embodiments of this specification, and there is no limitation thereto.

[0037] To facilitate understanding of the solutions provided in this manual, a typical reinforcement learning algorithm is briefly introduced below.

[0038] GRPO is a policy optimization method based on group sampling and relative advantage estimation, aiming to efficiently utilize human preference signals for large language model alignment. Its core idea is to generate multiple candidate outputs for the same input cue through a policy model, and construct a normalized advantage signal using the relative ranking of rewards within the group. This replaces the traditional Critic network (i.e., the value function), simplifying the training process and improving stability.

[0039] Specifically, for a given input hint q, from the old version of the current policy... We independently sample G complete candidate outputs (responses), denoted as... Subsequently, a reward model (RM) is used for each complete candidate output o. i Calculate a corresponding reward score r i =RM(q,o) i ).

[0040] To rely solely on relative merits within groups and reduce reliance on value function estimation, GRPO performs within-group normalization based on these G reward scores. GRPO's optimization objective combines a PPO-style pruning mechanism with a KL (Kullback-Leibler) divergence regularization term to prevent overly aggressive policy updates. Assume the current policy is π. θ The reference strategy is π. ref Then the objective function is:

[0041]

[0042] Where q is the current input. |o i | Output o for the i-th candidate i The number of elements (tokens) used for splitting the data. t represents the t-th element in the candidate output, or the t-th position (corresponding to the t-th element). π θ (O i,t |q,O i,<t ) represents the probability that the current agent (determined by parameter θ) predicts the t-th element based on the predicted elements before the t-th element (<t). This indicates the probability that the previous agent (e.g., the agent after the previous update) predicted the t-th element based on all predicted elements before the t-th element (represented by "<t" in the formula). `min` means taking the minimum value of each term within the parentheses. `clip` means truncating the first expression by the last two terms, such as taking... It sets the lower limit as 1-ε and the upper limit as 1+ε, where ε is the PPO clipping range (e.g., 0.2). In other words, Then take 1-ε, Then take 1+ε, otherwise take itself. It is a candidate output o i Within-group relative advantage estimation of the t-th element, advantage estimation One way to determine this is by: candidate output o i The reward r of the t-th element i,t Average reward of each element The difference divided by the standard deviation std i The quotient obtained can be denoted as: in, Another way to determine this is by defining r1, r2, r3...r respectively. G The estimated dominance values ​​of each group determined by within-group normalization Then the same candidate output o i Advantages Consider it as a unified advantage signal for all its tokens, i.e., for this candidate output o i Each position t = 1,…,∣o i | All use the same advantage Understandable It can be a positive or negative value. D KL (·||·) represents the KL divergence calculated on the output distribution at this location, and β is the KL regularization coefficient. It can be seen that when... When the optimization objective encourages the policy model to increase the probability on that output token; when Conversely, under the same input, this leads to a relative increase in high-reward outputs and a suppression of low-reward outputs. In actual training, the expectation is... GRPO approximates the value by averaging multiple inputs q and their corresponding G sampled candidate outputs in a batch. GRPO only needs to update the parameters of the policy model (such as a large language model) without training an additional value function.

[0043] As mentioned earlier, current mainstream policy optimization algorithms all require the use of reward functions to calculate reward signals. However, in current technologies, the calculation of reward signals relies on a uniform binary reward function (e.g., 1 for task success and 0 for failure). While this design is simple and easy to implement, it results in extremely sparse reward signals with a lack of discriminative power. This fails to provide the model with sufficient feedback to distinguish whether task success is due to random attempts or reasonable reasoning, leading to unstable training and limiting the effectiveness of policy optimization.

[0044] Based on this, the embodiments of this specification propose a model training method based on reinforcement learning, which calculates a more stable and reliable reward signal by the group success rate. Figure 2 This is a flowchart illustrating a reinforcement learning-based model training method according to one embodiment of this specification. This method can be executed by any device, platform, or cluster of devices with computing and processing capabilities, including steps 201-205 as shown below. It is understood that, as described above, the embodiments of this specification are applicable to any reinforcement learning algorithm.

[0045] like Figure 2 As shown, in step 201, user commands and device status are obtained.

[0046] The user instructions are used to direct the device to execute a target task, which includes several operational steps. The device status is the current operating state of the device. It is understandable that if the target task includes multiple operational steps, it will require multiple time steps to execute.

[0047] In the embodiments described in this specification, the intelligent agent can perform corresponding operation steps on the device according to user instructions to complete the target task. This embodiment does not limit the specific user instructions, devices, or target tasks. For example, user instructions can be instructions in different modalities such as text, voice, and images. The device can be various types of electronic devices such as mobile phones, computers, robots, and home appliances. The intelligent agent can be deployed locally on the device (e.g., integrated into its operating system or application), or it can be deployed on other devices or a remote server to work collaboratively with the device to achieve intelligent control of the device.

[0048] For example, user instruction 1 could be "Please set the living room temperature to 22 degrees Celsius," the device could be the living room air conditioner, the device status could be the current working status of the air conditioner, and the multiple operation steps of the target task could be: turn on the air conditioner and set the target temperature to 22 degrees Celsius; user instruction 2 could be "Please send Zhang Wei the beach photos I took last week, with the caption 'These are our photos from Sanya,'" the device could be the user's mobile phone, the device status could be the current display status of the mobile phone, and the multiple operation steps of the target task could be: open the photo album; search for photos from last week with the geotag "Sanya" or content identified as "beach"; open the messaging app, select the contact "Zhang Wei"; attach the text and photos and send.

[0049] This embodiment does not limit the method of obtaining user commands and device status. For example, in offline reinforcement learning training, training sample data can be obtained from a pre-built training set, and each training sample data includes user commands and device status at different time steps; or, in online interactive training, user commands and device status can be obtained in real time.

[0050] In one embodiment, the device (Graphical User Interface, or GUI) has a graphical user interface, and the device can be a mobile phone, watch, computer, television, or other device with a graphical user interface. The device state refers to the current display state of the graphical user interface; for example, it could be a screenshot of a mobile phone screen. Figure 3 The image shows the process by which a user issues user commands to an intelligent agent on their phone to achieve a target task. Figure 3 The process of obtaining user commands and device status is represented by a user and a mobile phone screen with a question mark. User commands could be, for example, "return to home screen" or "play a rock song".

[0051] In step 202, the user command and device status are input into the policy model, and the policy model generates multiple output results.

[0052] In this step, the policy model analyzes user commands and device status, generating multiple outputs for the current single time step. During the execution of the target task, at each time step, the policy model generates multiple outputs, each including a predicted action command for performing the same operation step. The predicted action command can be a structured sequence of action commands to invoke the device control interface.

[0053] For example, for the user instruction 1 mentioned above, after inputting user instruction 1 and the air conditioner status into the strategy model, one of the output results A output by the strategy model at the current time step may include: the predicted action instruction turn_on(device="air_conditioner",room="living_room") corresponding to operation step 1 turning on the air conditioner, and the other output result B output by the strategy model may include: another predicted action instruction turn_on(device="air_conditioner",room="bedroom") corresponding to operation step 1 turning on the air conditioner.

[0054] For example, for the user instruction 2 mentioned above, after inputting the user instruction 2 and the mobile phone display status (such as a screenshot) into the policy model, the output result C of the policy model at the current time step may include the predicted action instruction for opening the album: click the coordinates (x1, y1) on the screen, the output result D may include the predicted action instruction for opening the album: click the coordinates (x2, y2) on the screen, and the output result E may include the predicted action instruction for opening the album: click the coordinates (x3, y3) on the screen.

[0055] In this step, after receiving user instructions and device status, the policy model does not generate just one solution, but generates a group of G distinct output results O in parallel at each time step. i ,like Figure 3 The numbers shown are O1, O2, ..., O G This embodiment does not restrict the way the policy model generates multiple output results. For example, the policy model can first generate a score or probability distribution for all candidate action instructions, and then select the top K candidate action instructions based on the score or probability as multiple output results corresponding to that state. Alternatively, the policy model can generate a continuous action space, and then sample and sort within that space to obtain multiple output results. This group-generated output strategy can explore possible solutions more comprehensively.

[0056] It should be noted that when the policy model is a large language model, the output may also include the inference path from which the policy model derives the predicted action command. The inference path refers to the series of internal computational steps or logical unfoldings the model undergoes when processing input and generating output, including a step-by-step derivation process, reflecting the model's thought process from input to predicted action command. Each output result O i Both can include inference paths. Early graphical user interface agents primarily adopted an end-to-end model, directly mapping screen pixel information and user commands to specific actions (such as clicks and input). This approach lacks explicit intermediate inference processes, resulting in poor interpretability of decisions and difficulty in effective attribution analysis when tasks fail. In this embodiment, the policy model outputs inference paths along with specific predicted action commands, enhancing the interpretability of the model's output and facilitating attribution analysis.

[0057] In one embodiment, the reasoning path may refer to a chain of thought, where each output at a single time step includes a chain of thought and a predicted action instruction, the chain of thought being used to reason out the predicted action instruction.

[0058] In step 203, the group composition power is determined based on multiple output results.

[0059] Here, the group success rate represents the ratio of outputs that correctly execute the operation steps among multiple outputs. This embodiment not only evaluates whether a single output O is successful, but also evaluates the entire group {O1, O2, ..., O} from a macroscopic perspective. G The overall performance of the strategy model is as follows. It's understandable that not every predicted action instruction in the output will correctly execute the corresponding operation step. For example, after inputting user instruction 1 and the air conditioner status into the strategy model, the model outputs outputs A and B. Output A predicts the action instruction to turn on the living room air conditioner, while output B predicts the action instruction to turn on the bedroom air conditioner. Output A correctly executes operation step 1, while output B does not. Assuming the strategy model outputs two results at the current time step, namely output A and output B, then the success rate of the group at the current time step is 50%.

[0060] In this step, the group composition power at the current time step is obtained by calculating how many out of G output results can successfully complete the corresponding operation steps. This embodiment does not limit the specific method of determining the group composition power based on multiple output results. For example, a professional can compare and statistically analyze each output result and then input the determined group composition power; another example is that multiple output results, user commands, and device status can be input into a pre-trained model, which can then determine the group composition power; yet another example is that the target action command for executing the operation steps can be obtained, and then each output result and the target action command can be compared to determine the group composition power. This target action command can be set by a person skilled in the art according to the target task. By comparing the key fields of the predicted action command and the target action command in the output results through relevant algorithms or models, it can be determined whether the output result can correctly execute the operation steps, thereby determining the group composition power.

[0061] In step 204, the reward signal corresponding to the output result is calculated based on the group composition power.

[0062] In this step, the group composition power and specific output results can be combined to calculate different reward signals corresponding to different output results at the same time step.

[0063] Understandably, compared to relying on the success or failure of a single attempt (which may be accidental), the swarm success rate better reflects the overall ability of the policy model to solve the current task in the current state. A high swarm success rate indicates that most outputs for this type of task are successful, and the model has mastered this type of task well. Conversely, a low swarm success rate indicates that the model has not yet mastered this type of task. Therefore, calculating the reward signal based on the swarm success rate can effectively provide feedback to the policy function on the execution status of the current task. It provides a smoother and more reliable feedback signal, effectively avoiding excessive influence on policy updates due to a single "bad sample," thereby stabilizing the entire training process. This embodiment does not restrict the specific method for calculating the reward signal corresponding to the output result based on the swarm success rate.

[0064] In one embodiment, a reward adjustment factor can be determined based on the group composition power, and then the reward signal corresponding to the output result can be calculated based on the output result and the reward adjustment factor. The reward adjustment factor can be a weight or an additional term. This embodiment does not limit the method of determining the reward adjustment factor based on the group composition power. For example, the group composition power can be directly determined as the reward adjustment factor, or the reward adjustment factor can be obtained by transforming the group composition power through a pre-set function. For instance, after determining the reward factor, a preliminary reward signal is calculated from the output result using a pre-set reward function or environmental feedback. Then, the reward adjustment factor can be directly added to the preliminary reward signal as an additional term, or multiplied by the preliminary reward signal as a weight to obtain the reward signal corresponding to the output result.

[0065] In another embodiment, considering that the swarm composition power actually reflects the operational difficulty of the current task for the current policy model, the task complexity of the operation step can be determined based on the swarm composition power, where task complexity is negatively correlated with swarm composition power. Then, based on the task complexity and the output result, the reward signal corresponding to the output result is calculated. The higher the swarm composition power, the simpler the current task and the lower the task complexity of the current operation step; conversely, the lower the swarm composition power, the more difficult the current task and the higher the task complexity of the current operation step.

[0066] This embodiment does not limit the method for determining task complexity. For example, a preset threshold can be set, such as 70%. If the swarm composition power is higher than the preset threshold, the complexity of the operation step is determined to be low; if the swarm composition power is not higher than the preset threshold, the complexity of the operation step is determined to be high. In other examples, task complexity can be divided into more levels. For example, when the swarm composition power is below 10%, the task complexity of the operation step is determined to be level A, indicating very high task complexity; when the swarm composition power is greater than or equal to 10% and less than 40%, the task complexity of the operation step is determined to be level B, indicating relatively high task complexity; when the swarm composition power is greater than or equal to 40% and less than 80%, the task complexity of the operation step is determined to be level C, indicating medium task complexity; and when the swarm composition power is greater than or equal to 80%, the task complexity of the operation step is determined to be level D, indicating relatively low task complexity.

[0067] For operation steps with varying task complexities, different methods can be used to calculate reward signals. That is, the calculation criteria and form of the reward can be dynamically adjusted based on the real-time complexity of the operation step at each time step, so that the reward signal can provide more information for the reinforcement learning of the policy model. For example, for high-complexity tasks, the penalty for task failure in the reward signal can be reduced, while for low-complexity tasks, the penalty for task failure can be increased. Conversely, for high-complexity tasks, the reward for task success can be increased, while for low-complexity tasks, the reward for task success can be reduced.

[0068] Considering that policy optimization methods such as GRPO primarily focus on the correctness of the final action but neglect modeling the efficiency of the inference process itself, this often leads to unnecessary redundant thinking in simple tasks, affecting overall execution efficiency and planning capabilities. Therefore, this specification presents an adaptive reward signal calculation method in its embodiments. In one embodiment, each output result at a single time step includes the inference path from which the policy model derives the predicted action instruction and the predicted action instruction. In the above embodiment, when calculating the reward signal corresponding to the output result based on task complexity, the first reward signal corresponding to the output result can be determined based on the task complexity and the number of tokens in the inference path through a first reward function. Specifically, in cases of low task complexity, the number of tokens in the inference path is negatively correlated with the first reward signal; in cases of high task complexity, the number of tokens in the inference path is positively correlated with the first reward signal. The number of tokens in the inference path can be the number of tokens in the thought chain.

[0069] In this embodiment, the calculation of the reward signal is not static but dynamically adjusted based on the task's complexity, regulating the reward and penalty for thought length according to the task's difficulty. For simple tasks, the reward signal value decreases as thought length increases, encouraging the model to use concise and efficient thinking to quickly solve problems and avoid wasting computational resources by "overthinking" simple steps. For complex tasks, the reward signal value increases as thought length increases. This encourages the model to engage in deeper and more detailed thinking to ensure the correct handling of complex logic and operations, thereby improving the success rate. In this way, the model is forced to learn to intelligently allocate cognitive resources, achieving highly efficient reasoning by thinking less about simple tasks and more about difficult ones.

[0070] This embodiment does not impose restrictions on the setting of the first reward function, as long as it conforms to the relationship between the number of tokens in the above inference path and the first reward signal. For example, the first reward function R... thought It can be as follows:

[0071]

[0072] Where l represents the task complexity, t can be the number of tokens in the thought chain. T is a preset value, and min and max are the minimum and maximum values, respectively. The specific values ​​mentioned above can be set according to actual needs.

[0073] In some alternative implementations, reward signals can be designed with more granular task complexity to provide refined feedback to the policy model, distinguishing between "lucky success" and "precise success." When calculating the reward signal corresponding to the output result based on task complexity and output result, the decay rate of the second reward function can be determined based on task complexity, with the decay rate negatively correlated with task complexity. Then, based on the difference between the predicted action instruction and the target action instruction used to execute the operation step in each output result, the second reward signal is calculated. The second reward signal gradually decays according to the decay rate as the difference between the predicted action instruction and the target action instruction increases. In this way, the model can be encouraged to execute precisely on simple tasks, while on complex tasks, it can be encouraged to make bolder attempts, thereby effectively improving the model's success rate and generalization ability in diverse scenarios.

[0074] This embodiment does not impose restrictions on the construction of the second reward function. For example, an exponential decay function, an inverse proportional function, or a Gaussian function can be used to model the relationship that the larger the error, the lower the reward. The difference between the predicted action instruction and the target action instruction used to execute the operation step can be measured by similarity, L1 distance, and semantic distance.

[0075] For example, the second reward function can be a Gaussian function, such as... Figure 4 As shown, the higher the task complexity, the greater the bandwidth of the Gaussian function. For example, when the device is a mobile phone, the target action instruction used to perform the operation steps can be represented as μ. gt =(x gt ,y gt ), meaning the screen coordinates (x) were clicked. gt ,y gt The predicted action instruction can be represented as p = (x pred ,y pred ), meaning the screen coordinates (x) were clicked. pred ,y pred The second reward function can be represented by the following formula:

[0076]

[0077] Where l represents the task complexity. N thrThis can be a preset threshold. The value of σ (sigma) is related to the task complexity. For operations with low task complexity, a narrow-bandwidth (small σ) Gaussian function can be used, requiring extremely high accuracy to achieve a high score. For operations with high task complexity, a wide-bandwidth (large σ) function is used, allowing exploration within a certain deviation range and awarding rewards. For example, for simple tasks with low task complexity, σ can be set to 0.3; for medium-complexity tasks, σ can be set to 0.5; and for complex tasks with high task complexity, σ can be set to 0.7. Figure 4 As shown, the vertical axis represents the value of the second reward signal, and the horizontal axis represents the difference between the predicted action command and the target action command. The green center can be regarded as the target action command. The closer the curve of the second reward function is to the green center, the smaller the difference between the predicted action command and the target action command, and the larger the value of the second reward signal.

[0078] The aforementioned refined feedback mechanism enables the model to learn strategies for different scenarios more quickly, thereby accelerating the convergence speed and improving the final action accuracy.

[0079] In other embodiments, considering that the system may give greater exploration rewards if the success rate of swarm formation is very low, and encourage the model to try more diverse solutions, policy entropy can also be added as a regularization term to the subsequent objective function to encourage randomness, so that the lower the success rate, the greater the weight of policy entropy in the objective function, and the stronger the policy randomness of the model.

[0080] It should be noted that it can be as follows: Figure 3 As shown, for each output result at a single time step, the corresponding reward signal R can be calculated. j Specifically, R1, R2, ... R G In other embodiments, multiple output results can be calculated to obtain a single reward signal corresponding to the multiple output results, that is, a reward signal corresponding to the current time step.

[0081] In step 205, the parameters of the policy model are updated using a reinforcement learning algorithm based on the reward signals corresponding to each output result.

[0082] It is understandable that steps 201-204 above execute the operation steps within a single time step. For example, in the first time step, the user command and the device state at the first time step are input into the policy model. The policy model generates multiple output results, including the predicted action command for operation step 1 in the first time step. After calculating the reward signals corresponding to the multiple output results, the operation steps corresponding to the target action command (or a certain predicted action command) of the current time step can be executed on the device to change the device state. Then, steps 201-204 are iteratively executed to execute the operation steps of the next time step. For example, the device state of the second time step is input into the policy model. The policy model generates multiple output results, including the predicted action command for operation step 2 in the second time step, and then the reward signals corresponding to the multiple output results are calculated... until all the operation steps included in the target task have been executed. The entire task constitutes a trajectory, containing a series of states, actions, and reward signals. Figure 3 As shown, the parameters of the policy model can then be updated using a reinforcement learning algorithm based on the reward signals corresponding to the output results at all time steps in the trajectory.

[0083] This embodiment does not restrict the specific method of updating the parameters of the policy model based on the reward signal using a reinforcement learning algorithm. For example, policy optimization methods in PPO and GRPO can be used to normalize, sort, or perform weighted averaging of these reward signals for group calculation and then update the parameters.

[0084] In one embodiment, the dominance function can be calculated directly based on the reward signal, similar to the dominance function calculation method in GRPO. For example, for the G reward signals corresponding to multiple output results at each time step, the output result O at a certain time step t can be calculated using the following formula. i Corresponding advantage function:

[0085]

[0086] Here, mean represents the average value, and std represents the standard deviation. This advantage function can be substituted into the objective function of formula (1) above to derive the gradient signal for updating the parameters of the policy model. For example... Figure 3 As shown, the arrows connect the multiple advantage functions generated at the end of the process (assuming there are n advantage functions across all time steps) to the policy model at the beginning, forming a complete closed loop. This process repeats continuously. In each iteration, the model is fine-tuned based on the objective function of the previous iteration, with the goal of generating thought and action sequences that yield higher rewards (i.e., more efficient and accurate).

[0087] In other embodiments, a reward estimation / value function can be constructed based on the state, action, and reward signal at each time step, and then the advantage function can be estimated. The objective function is then optimized to update the parameters of the policy model. Furthermore, the parameters of the value function can also be updated. Taking PPO as an example, the agent includes a policy model and a value function Vφ(s). First, trajectory data of the state, reward signal, and action at all time steps can be collected according to the execution process of the aforementioned objective task. Then, for each time step t in the collected trajectory data, an estimate of the advantage function is calculated. This embodiment does not restrict the method of calculating the advantage function; for example, generalized advantage estimation (GAE) can be used. Then, the policy loss and value function loss are calculated separately, and the parameters of the policy model and value function are adjusted by minimizing these losses.

[0088] Based on the above embodiments, in order to prevent the strategy model from deviating too far from a normal and reasonable language distribution during the optimization process, such as Figure 3 As shown, a baseline model can also be set up. User commands and device status can be input into the baseline model, which will generate baseline output results. The baseline model can be obtained by supervising and fine-tuning the policy model before training. Its role is to serve as a rational reference. Then, based on the difference between the baseline output results and each output result, a penalty term is determined. The penalty term can be KL divergence, JS (Jensen-Shannon) divergence, or Hellinger distance, etc. Finally, based on the reward signal and penalty term corresponding to each output result, the parameters of the policy model are updated through reinforcement learning algorithms. For example, the penalty term can be added to the objective function. For example, it can be used as the KL divergence penalty term in the objective function of formula (1). In this way, by using the baseline model as a reference, the stability and quality of the content generated by the policy model can be guaranteed.

[0089] Figure 5 This is a schematic diagram of the structure of a reinforcement learning-based model training device as described in the embodiments of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0090] The acquisition module 51 is used to acquire user instructions and device status. The user instructions are used to instruct the device to perform a target task, which includes several operation steps. The device status is the current working state of the device.

[0091] The inference module 52 is used to input user commands and device status into the policy model, and the policy model generates multiple output results, each of which includes a predicted action command for performing the same operation step.

[0092] The comparison module 53 is used to determine the group composition power based on multiple output results. The group composition power is used to represent the ratio of the output results that correctly execute the operation steps among the multiple output results.

[0093] The reward module 54 is used to calculate the reward signal corresponding to the output result based on the group composition power.

[0094] The parameter tuning module 55 is used to update the parameters of the policy model based on the reward signals corresponding to each output result through a reinforcement learning algorithm.

[0095] In some optional implementations, the reward module 54 is specifically used to determine the task complexity of the operation steps based on the group composition power, wherein the task complexity is negatively correlated with the group composition power; and to calculate the reward signal corresponding to the output result based on the task complexity and the output result.

[0096] In some optional implementations, the reward module 54 is specifically used to determine the complexity of the operation step as low complexity in response to the group composition power being higher than a preset threshold, and to determine the complexity of the operation step as high complexity in response to the group composition power not being higher than the preset threshold.

[0097] In some optional implementations, the output also includes an inference path for generating predicted action instructions by deriving the policy model. The reward module 54 is specifically used to determine the first reward signal corresponding to the output based on the task complexity and the number of tokens in the inference path through a first reward function. Specifically, when the task complexity is low, the number of tokens in the inference path is negatively correlated with the first reward signal, and when the task complexity is high, the number of tokens in the inference path is positively correlated with the first reward signal.

[0098] In some optional implementations, the reward module 54 is specifically used to determine the decay rate of the second reward function based on the task complexity, the decay rate being negatively correlated with the task complexity; and to calculate a second reward signal based on the difference between the predicted action instruction and the target action instruction for performing the operation step in each output result, the second reward signal gradually decaying according to the decay rate as the difference between the predicted action instruction and the target action instruction increases.

[0099] In some alternative implementations, the second reward function is a Gaussian function, and the higher the task complexity, the greater the bandwidth of the Gaussian function.

[0100] In some optional implementations, the parameter tuning module 55 is specifically used to input user commands and device status into the benchmark model, generate benchmark output results from the benchmark model, and obtain the benchmark model by supervised fine-tuning of the policy model before training; determine the penalty term based on the difference between the benchmark output result and the output result; and update the parameters of the policy model through a reinforcement learning algorithm based on the reward signal and penalty term corresponding to each output result.

[0101] In some optional implementations, the comparison module 53 is specifically used to obtain the target action instruction for performing the operation steps; compare each output result with the target action instruction to determine the group composition power.

[0102] In some optional implementations, the reward module 54 is specifically used to determine the reward adjustment factor based on the group composition power; and to calculate the reward signal corresponding to the output result based on the output result and the reward adjustment factor.

[0103] In some alternative implementations, the device has a graphical user interface, and the device state is the current display state of the graphical user interface.

[0104] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0105] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0106] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0107] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0108] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A model training method based on reinforcement learning, the method comprising: The system acquires user instructions and device status. The user instructions are used to instruct the device to perform a target task, which includes several operation steps. The device status is the current working state of the device. The user command and the device status are input into the strategy model, and the strategy model generates multiple output results. Each output result includes a predicted action command for performing the same operation step. Based on the multiple output results, a group composition power is determined, which represents the ratio of the output results that correctly execute the operation steps among the multiple output results; Based on the group composition power, the reward signal corresponding to the output result is calculated; Based on the reward signals corresponding to each of the output results, the parameters of the policy model are updated using a reinforcement learning algorithm.

2. The method according to claim 1, wherein, The calculation of the reward signal corresponding to the output result based on the group composition power includes: Based on the group composition power, the task complexity of the operation step is determined, and the task complexity is negatively correlated with the group composition power; Based on the task complexity and the output result, the reward signal corresponding to the output result is calculated.

3. The method according to claim 2, wherein, Determining the task complexity of the operation steps based on the group composition power includes: In response to the group composition power being higher than a preset threshold, the complexity of the operation step is determined to be low complexity; In response to the group composition power not being higher than the preset threshold, the complexity of the operation step is determined to be high complexity.

4. The method according to claim 2, wherein, The output also includes the inference path derived by the policy model to generate the predicted action instruction, and the calculation of the reward signal corresponding to the output based on the task complexity and the output includes: Based on the task complexity and the number of tokens in the inference path, the first reward signal corresponding to the output result is determined by the first reward function; Specifically, when the task complexity is low, the number of tokens in the inference path is negatively correlated with the first reward signal; when the task complexity is high, the number of tokens in the inference path is positively correlated with the first reward signal.

5. The method according to claim 2, wherein, The calculation of the reward signal corresponding to the output result based on the task complexity and the output result includes: Based on the task complexity, the decay rate of the second reward function is determined, and the decay rate is negatively correlated with the task complexity. A second reward signal is calculated based on the difference between the predicted action instruction and the target action instruction for performing the operation step in each of the output results. The second reward signal gradually decays according to the decay rate as the difference between the predicted action instruction and the target action instruction increases.

6. The method according to claim 5, wherein, The second reward function is a Gaussian function, and the higher the task complexity, the greater the bandwidth of the Gaussian function.

7. The method according to claim 2, wherein, The step of updating the parameters of the policy model using a reinforcement learning algorithm based on the reward signals corresponding to each of the output results includes: The user instructions and the device status are input into the benchmark model, and the benchmark model generates a benchmark output result. The benchmark model is obtained by supervised fine-tuning of the policy model before training. Based on the difference between the baseline output and the output, a penalty term is determined; Based on the reward signal and penalty term corresponding to each output result, the parameters of the policy model are updated using a reinforcement learning algorithm.

8. The method according to claim 1, wherein, The determination of group composition power based on multiple output results includes: Obtain the target action instruction for performing the operation steps; The group composition power is determined by comparing each output result with the target action command.

9. The method according to claim 1, wherein, The calculation of the reward signal corresponding to the output result based on the group composition power includes: Based on the group composition power, determine the reward adjustment factor; Based on the output result and the reward adjustment factor, the reward signal corresponding to the output result is calculated.

10. The method according to claim 1, wherein, The device has a graphical user interface, and the device state is the current display state of the graphical user interface.

11. A model training device based on reinforcement learning, the device comprising: The acquisition module is used to acquire user instructions and device status. The user instructions are used to instruct the device to perform a target task, which includes several operation steps. The device status is the current working state of the device. The inference module is used to input the user instructions and the device status into the policy model, and the policy model generates multiple output results, each output result including a predicted action instruction for performing the same operation step; A comparison module is used to determine the group composition power based on multiple output results, wherein the group composition power represents the ratio of the output results that correctly execute the operation steps among the multiple output results; The reward module is used to calculate the reward signal corresponding to the output result based on the group composition power. The parameter tuning module is used to update the parameters of the policy model based on the reward signal corresponding to each output result using a reinforcement learning algorithm.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-10.

Images