Online integrated speculative decoding method for mobile terminal graphical user interface operation agent
By integrating an online learning mechanism and using base learners with differentiated learning rates, the draft model is adaptively optimized, solving the problem of poor adaptability of speculative decoding in mobile terminal GUI operating environments, and achieving a reduction in inference latency and an improvement in task fluency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2026-02-13
- Publication Date
- 2026-06-05
AI Technical Summary
Existing speculative decoding methods rely on fixed draft models trained offline, which cannot adapt to the differences and changes in the mobile terminal GUI operating environment, resulting in inference latency and poor task smoothness.
An online ensemble learning mechanism is adopted. By constructing multiple base learners and meta-learners with differentiated learning rates, and utilizing the feedback signals from the target model validation process, the draft model is dynamically adjusted to adapt to operating environments with different rates of change, thereby achieving adaptive optimization of the draft model.
It significantly improved the acceptance rate of draft models, reduced the multi-step operation inference latency of mobile agents, and improved the smoothness of task execution and user experience.
Smart Images
Figure CN122152178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an online integrated speculative decoding method for operating intelligent agents in a mobile terminal graphical user interface, belonging to the technical field of large language model inference acceleration and automated operation of mobile intelligent agents. Background Technology
[0002] With the continuous breakthroughs in the capabilities of large language models in natural language understanding and generation, intelligent agents for mobile terminal graphical user interfaces (GUIs) based on large language models have become an emerging research and application direction. These agents automatically generate sequences of operation instructions such as clicking, swiping, text input, and application switching by sensing the current screen state of the mobile terminal. This allows them to perform complex multi-step tasks such as cross-application information transfer, batch message replies, and automatic form filling, demonstrating broad application prospects in scenarios such as personal affairs automation assistants, accessibility assistance, and enterprise mobile office automation. However, the inherent sequential dependency of autoregressive decoding in large language models—each character can only be generated after the previous character—leads to significant inference delays in multi-step operation reasoning processes, severely restricting the smoothness of task execution and user experience of mobile intelligent agents.
[0003] Speculative decoding, a widely adopted paradigm for accelerating inference with large language models, effectively reduces inference latency by introducing a lightweight draft model to quickly generate candidate character sequences, which are then verified in parallel by a large target model. This approach maintains consistent output quality. When applied to mobile GUI agents, the draft model rapidly generates candidate screen operation command sequences, while the target model verifies the consistency of the interface state and the correctness of the operation logic in parallel. The acceleration effect of this method directly depends on the similarity between the output distributions of the draft model and the target model: a higher acceptance rate means more operation commands are accepted in a single verification, resulting in lower inference latency.
[0004] However, most existing speculative decoding methods rely on fixed-draft models trained offline, which face significant challenges in the actual deployment of mobile GUI-based intelligent agents. On one hand, the system UI styles of different brands and models of mobile phones vary significantly. The interface layout, control naming, and interaction logic of the same application may differ on different terminals, making it impossible for offline-trained draft models to cover the operational distribution across all terminal environments. On the other hand, mobile applications experience frequent changes in interface layout due to version iterations. The addition of new features, rearrangement, or deletion of interface elements causes the operational patterns learned during offline training to quickly become invalid after deployment. Furthermore, users' application ecosystems are highly personalized; different users have different combinations of installed applications, usage frequencies, and operating habits, making it difficult for fixed-draft models to adapt to these significant individual differences. These factors collectively lead to a significant decrease in the acceptance rate of fixed-draft models in actual deployment, and the cumulative latency of multi-step operation inference severely impacts task fluency. Therefore, there is an urgent need for an online learning method that can fully utilize the feedback signals naturally generated during the speculative decoding and verification process, while adaptively responding to the complex dynamic changes in the mobile terminal GUI operating environment where gradual and sudden changes coexist, fundamentally reducing the latency of multi-step operation inference, so as to improve the task execution efficiency and user experience of mobile intelligent agents in actual deployment scenarios. Summary of the Invention
[0005] Purpose of the Invention: To address the challenges of existing mobile terminal graphical user interface (GUI) operators based on large language models in practical deployments, such as the inability of offline-trained fixed draft models to adapt to UI differences across different terminal systems, frequent interface layout changes due to application version iterations, and significant differences in user terminal application ecosystems, and the difficulty of existing online update methods using a single fixed learning rate to simultaneously handle complex dynamic changes in the operating environment where gradual and abrupt changes coexist, this invention proposes an online ensemble speculative decoding method for mobile terminal GUI operators. This method fully utilizes the feedback signals naturally generated during the target model verification process in speculative decoding to construct an online ensemble learning mechanism, significantly improving the acceptance rate of the draft model, reducing the inference latency of multi-step operations for mobile agents, and enhancing task execution smoothness and user experience.
[0006] Technical Solution: An online integrated speculative decoding method for mobile terminal graphical user interface (GUI) operation agents is proposed to adaptively address the dynamic impact of factors such as UI differences between different mobile phone brands, frequent changes in interface layout due to application version iterations, and significant differences in user terminal application ecosystems on the acceptance rate of draft models. This method is used to accelerate inference optimization for mobile terminal GUI operation agents performing multi-step operation tasks such as cross-application information transfer, batch message reply, and automatic form filling. Specifically, in the offline initialization phase, historical operation data of the mobile terminal GUI is collected to train the initial parameters of the target model and the draft model. Multiple base learners with differentiated learning rates are then constructed based on the initial draft model. Next, in the online adaptation phase, a speculative decoding architecture is used to perform multi-step operation inference: the draft model quickly generates candidate screen operation instruction sequences based on the current screen state, the target model verifies the consistency of the interface state and the correctness of the operation logic in parallel, the length of the accepted operation instruction is determined based on the likelihood ratio of the output distributions of the target model and the draft model, and a feedback signal is generated based on the verification results. Then, an online ensemble learning mechanism is constructed using the feedback signals naturally generated during the target model validation process: multiple base learners with differentiated learning rates are maintained. Each base learner independently performs online gradient descent updates according to its own learning rate. Base learners with smaller learning rates are adapted to gradual scenarios with slow iterations of the application interface, while base learners with larger learning rates are adapted to abrupt scenarios with rapid switching across application task types. "Smaller" means less than a user-defined threshold, and "larger" means greater than or equal to the user-defined threshold. Finally, the meta-learner adopts an exponential weighting strategy, dynamically adjusting the fusion weights based on the cumulative loss of each base learner in the validation feedback. The outputs of each base learner are weighted and fused to obtain the final draft strategy. Finally, the fused draft model is applied to the multi-step operation inference of the mobile terminal graphical user interface operator, generating operation instruction sequences for screen clicks, swipes, text input, and application switching, achieving adaptive optimization of the multi-step operation inference latency of the mobile terminal operator. Adaptively tracking the optimal draft model for the current operating environment significantly improves the acceptance rate of the draft model and reduces the multi-step operation inference latency of the mobile terminal operator.
[0007] To ensure the draft model has good initial performance and to construct a diverse pool of base learners, the specific steps of the offline initialization phase are as follows: Step 100: Offline collection of historical data of mobile terminal graphical user interface operations. ,in This represents the total number of samples in the offline dataset. This represents a feature vector of the interface state, composed of screenshots, a tree of interface elements, and a sequence of historical operations. This indicates the corresponding target operation instruction label; Step 101, Select the target model ,in , Represents the parameter space of the target model. This represents the output distribution of the target model. This represents the input interface state sequence. The target model is a large language model with a large number of parameters, used to accurately predict and verify screen operation commands. Step 102, select the draft model ,in , Represents the parameter space of the draft model. This represents the output distribution of the draft model. This represents the input interface state sequence. The draft model is a lightweight model and satisfies... This is used to quickly generate candidate operation instruction sequences; Step 103, Select the loss function This is used to measure the difference between the distribution of operation instructions predicted by the draft model and the distribution of output of the target model, where the input... express 3D real vector space, Indicates the output space; Step 104: On the historical dataset collected in step 100, using the draft model selected in step 102 and the loss function selected in step 103, obtain the offline initial parameters of the draft model by minimizing the loss function: ,in To initialize the model; Step 105, based on the initial draft model parameters obtained in step 104 Copy build Individual learners ,in For the first Each base learner is assigned a set of differentiated learning rates. ,in For the first The learning rate of each learner, and the initial weights of the meta-learner. For uniform distribution , .
[0008] In order to perform speculative decoding inference at each time step and generate verification feedback signals for online updates, the specific steps of the speculative decoding architecture in performing multi-step operational inference and generating verification feedback are as follows: Step 200, at each time step Obtain the current interface status information of the mobile terminal. This includes screenshots, a tree of interface elements, and historical operation context; Step 201, from the current draft model Based on the current interface state, autoregressively generate a length of... Candidate operation instruction sequence Each operation instruction From conditional distribution Generated by sampling, where Indicates time step Output distribution of the draft model This indicates the current interface state and the prefix of the generated operation instructions; Step 202, Target Model Candidate operation instruction sequences are verified in parallel, and the target distribution at all positions is calculated in one forward propagation. ,in This represents the output distribution of the target model; Step 203, for each position Independently sampled uniform random numbers Calculate the likelihood ratio between the target model and the draft model at this location. ,in For the token at position j, Given the token preceding j, determine the length of the accepted operation instruction. ,in The interval is Uniform distribution; Step 204, if Then in the first The positions are from the corrected distribution. Mid-sample generates corrected operation instructions; if Then from the target distribution The next operation instruction is generated by sampling; the received operation instruction sequence is appended to the current operation sequence. Step 205: Generate feedback signals based on the validation results and construct the loss function. ,in Represents the parameter space of the draft model. This represents the output distribution of the draft model. This represents the sequence of input interface states. This represents the output distribution of the target model. This refers to the cross-entropy loss between the target model's output distribution and the draft model's predicted distribution. This feedback signal is generated naturally during the validation process and requires no additional computational overhead.
[0009] To enable multiple draft model base learners to adapt to operating environments with different rates of change, the specific steps for constructing and updating these multiple draft model base learners with differentiated learning rates are as follows: Step 300, Maintenance Individual learners ,in It is the first A draft model at time step The weights are set at different times, and each base learner is configured with a different learning rate. The base learner with a smaller learning rate is suitable for the gradual change scenario where the application interface is slowly iterated, while the base learner with a larger learning rate is suitable for the abrupt change scenario where the application task type is quickly switched. Step 301, at each time step Each base learner independently receives the validation feedback loss function generated in step 205. Calculate the gradient at each of their respective parameters. ; Step 302: Each base learner independently performs online gradient descent updates. ,in Represents the parameter constraint space The projection operator, For the first The learning rate of each base learner For the first The gradient of each base learner at the current parameters; Step 303, Updated parameters of each base learner It is passed to the meta-learner module for weighted fusion at the next time step.
[0010] To adaptively track the optimal draft strategy in the current operating environment, the meta-learner dynamically adjusts the fusion weights based on the cumulative loss of the validation feedback. The specific steps are as follows: Step 400: Initialize the fusion weights of the meta-learner to be uniformly distributed. , ,in The total number of base learners, Let i be the weight; Step 401, at each time step The meta-learner records the loss value of each base learner on the validation feedback loss function in the current round. , And update the cumulative loss of each base learner. ; Step 402: The meta-learner updates the fusion weights of each base learner using an exponential weighting strategy. ,in This is a sensitivity control parameter used to adjust the degree of response of the weights to the cumulative loss difference, which, after normalization, satisfies... ; Step 403: The meta-learner weights and fuses the outputs of each base learner to obtain the final draft model at the current time step. ,in For the first Each base learner at time step The fusion weight, For the first Each base learner at time step Model parameters.
[0011] The draft models available in step 102 include: a lightweight language model based on a Transformer decoder, a draft prediction head network attached to the hidden layers of the target model, and a draft model based on a feature retrieval mechanism.
[0012] The loss functions available in step 103 include: cross-entropy loss function, KL divergence loss function, mean squared error loss function, and DPO preference optimization loss function.
[0013] The differential learning rate set in step 105 The geometrically proportional interval setting is adopted, that is , ,in Minimum learning rate, For common ratio, The total number of base learners; a smaller learning rate corresponds to slow adaptation in a stable environment, while a larger learning rate corresponds to fast response in a non-stationary environment. Geometric equal intervals ensure that the set of learning rates has uniform logarithmic scale coverage of environments with different rates of change.
[0014] The methods for obtaining the current screen status information of the mobile terminal in step 200 include: collecting interface element tree information through the Android accessibility service interface, obtaining current screen image information through the screenshot interface, obtaining application package name and activity page name information through the system log interface, and obtaining operation instruction context information through operation history cache.
[0015] Sensitivity control parameters in step 402 With the length of the candidate operation instruction sequence The parameters are set based on the statistical characteristics of the offline collected mobile terminal graphical user interface operation history dataset. The statistical characteristics include the average length of the operation command sequence, the frequency of cross-application switching, and the rate of change of interface elements.
[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the online integrated speculative decoding method for operating intelligent agents with a graphical user interface for mobile terminals as described above.
[0017] A computer-readable storage medium storing a computer program that performs the online integrated speculative decoding method for operating an intelligent agent with a graphical user interface for a mobile terminal as described above.
[0018] Beneficial effects: Compared with existing methods, this invention maintains multiple draft models that adapt to different rates of change through an online ensemble learning mechanism, and the meta-learner dynamically selects the optimal combination. It can adaptively cope with the complex dynamic changes in the mobile GUI operating environment where gradual and sudden changes coexist without the need to predict the environmental change pattern. In mobile intelligent agent deployment scenarios with dynamic changes in application environments such as personal affairs automation assistants, accessibility assistance operations, and enterprise mobile office automation, it can improve the inference efficiency and task execution smoothness of the system in real time online. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the overall process of the online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to an embodiment of the present invention. Detailed Implementation
[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0021] like Figure 1 As shown, an online integrated speculative decoding method for operating intelligent agents in a mobile terminal graphical user interface first collects historical operation data of the mobile terminal graphical user interface during the offline initialization phase, and trains the initial parameters of the target model and the draft model. The initialization process includes historical operational data collection, selection of target and draft model structures, selection of loss functions, and initial parameter training. Subsequently, a model is constructed based on the parameters of the initial draft model. Individual learners And assign a set of differential learning rates with geometrically equal intervals to each base learner. The initialization weights of the meta-learner are uniformly distributed.
[0022] During the online adaptation phase, the speculative decoding inference and verification process, as well as the online ensemble learning process, are as follows: At each time step, the screen state awareness module first acquires state information such as the current screen screenshot, interface element tree, and historical operation context of the mobile terminal; subsequently, the currently fused draft model autoregressively generates a length of... The candidate operation instruction sequence is used, and the target model performs parallel verification on this sequence. The likelihood ratio of the output distribution of the target model and the draft model is calculated position by position and compared with uniformly sampled random numbers to determine the length of the accepted operation instruction. For rejected positions, corrected operation instructions are generated by sampling from the corrected distribution to ensure that the final output is consistent with the distribution of the target model. The cross-entropy loss naturally generated during the validation process serves as a feedback signal, requiring no additional computational overhead. After obtaining validation feedback, the online ensemble learning update process begins: each base learner independently receives the feedback loss function, calculates the gradient at its respective parameters, and performs online gradient descent updates according to its configured differentiated learning rate.
[0023] Finally, the fused draft model is applied to the multi-step operation reasoning of the mobile terminal graphical user interface operation agent to generate operation instruction sequences such as screen click, swipe, text input, and application switching, and continues to execute the above process in the next iteration, realizing online adaptive optimization of multi-step operation reasoning latency.
[0024] The specific steps of the offline initialization phase are as follows: Step 101, Select the target model ,in Represents the parameter space of the target model. This represents the output distribution of the target model. This represents the input interface state sequence. The target model is a large language model with a large number of parameters, used to accurately predict and verify screen operation commands.
[0025] Step 102, select the draft model ,in Represents the parameter space of the draft model. This represents the output distribution of the draft model. This represents the input interface state sequence. The draft model is a lightweight model and satisfies... It is used to quickly generate candidate operation instruction sequences.
[0026] Step 103, Select the loss function This is used to measure the difference between the distribution of operation instructions predicted by the draft model and the distribution of output of the target model, where the input... express 3D real vector space, Indicates the output space.
[0027] Step 104: On the historical dataset collected in step 100, using the draft model selected in step 102 and the loss function selected in step 103, obtain the offline initial parameters of the draft model by minimizing the loss function: ,in To initialize the model.
[0028] Step 105, based on the initial draft model parameters obtained in step 104 Copy build Individual learners And assign a differentiated set of learning rates to each base learner. ,in For the first The learning rate of each learner, and the initial weights of the meta-learner. For uniform distribution , .
[0029] The specific steps by which the speculative decoding architecture performs multi-step operational inference and generates verification feedback are as follows: Step 200, at each time step Obtain the current screen status information of the mobile terminal. This includes screenshots, a tree of interface elements, and historical operation context.
[0030] Step 201, from the current draft model Based on the current screen state, autoregressively generate a length of... Candidate operation instruction sequence Each operation instruction From conditional distribution Generated by sampling, where Indicates time step Output distribution of the draft model This indicates the current interface state and the prefix of the generated operation instructions.
[0031] Step 202, Target Model Candidate operation instruction sequences are verified in parallel, and the target distribution at all positions is calculated in one forward propagation. ,in This represents the output distribution of the target model.
[0032] Step 203, for each position Independently sampled uniform random numbers Calculate the likelihood ratio between the target model and the draft model at this location. The length of the received operation instruction is determined to be... ,in The interval is The uniform distribution.
[0033] Step 204, if Then in the first The positions are from the corrected distribution. Mid-sample generates corrected operation instructions; if Then from the target distribution The next operation instruction is generated by sampling; the received operation instruction sequence is appended to the current operation sequence.
[0034] Step 205: Generate feedback signals based on the validation results and construct the loss function. ,in Represents the parameter space of the draft model. This represents the output distribution of the draft model. This represents the sequence of input interface states. This represents the output distribution of the target model. This refers to the cross-entropy loss between the target model's output distribution and the draft model's predicted distribution. This feedback signal is generated naturally during the validation process and requires no additional computational overhead.
[0035] The specific steps for constructing and updating multiple draft model base learners with differentiated learning rates are as follows: Step 300, Maintenance A draft model base learner ,in It is the first A draft model at time step The weights are set at different times, and each base learner is configured with a different learning rate. The base learner with a smaller learning rate is suitable for gradual scenarios where the application interface iterates slowly, while the base learner with a larger learning rate is suitable for abrupt scenarios where the application task type changes rapidly.
[0036] Step 301, at each time step Each base learner independently receives the validation feedback loss function generated in step 205. Calculate the gradient at each of their respective parameters. .
[0037] Step 302: Each base learner independently performs online gradient descent updates. ,in Represents the parameter constraint space The projection operator, For the first The learning rate of each base learner For the first The gradient of each base learner at the current parameters.
[0038] Step 303, Updated parameters of each base learner It is passed to the meta-learner module for weighted fusion at the next time step.
[0039] The specific steps of the meta-learner dynamically adjusting the fusion weights based on the cumulative loss of validation feedback are as follows: Step 400: Initialize the fusion weights of the meta-learner to be uniformly distributed. , ,in The total number of base learners.
[0040] Step 401, at each time step The meta-learner records the loss value of each base learner on the validation feedback loss function in the current round. , And update the cumulative loss of each base learner. .
[0041] Step 402: The meta-learner updates the fusion weights of each base learner using an exponential weighting strategy. ,in This is a sensitivity control parameter used to adjust the degree of response of the weights to the cumulative loss difference, which, after normalization, satisfies... .
[0042] Step 403: The meta-learner weights and fuses the outputs of each base learner to obtain the final draft model at the current time step. ,in For the first Each base learner at time step The fusion weight, For the first Each base learner at time step Model parameters.
[0043] Obviously, those skilled in the art should understand that the steps of the online integrated speculative decoding method for operating intelligent agents of mobile terminal graphical user interfaces described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
Claims
1. An online integrated speculative decoding method for operating intelligent agents in a mobile terminal graphical user interface, characterized in that, The process includes an offline initialization phase and an online adaptation phase. First, in the offline initialization phase, historical operation data of the mobile terminal's graphical user interface is collected to train the initial parameters of the target model and the draft model. Then, multiple base learners with differentiated learning rates are constructed based on the initial draft model. Next, in the online adaptation phase, a speculative decoding architecture is used to perform multi-step operation inference: the draft model generates candidate screen operation instruction sequences based on the current screen state, the target model verifies the consistency of the interface state and the correctness of the operation logic in parallel, and the length of the accepted operation instruction is determined based on the likelihood ratio of the output distributions of the target model and the draft model. Then, an online ensemble learning mechanism is constructed using the feedback signals naturally generated during the target model validation process: multiple draft model base learners with differentiated learning rates are maintained, and each base learner independently performs online gradient descent updates according to its own learning rate; the meta-learner adopts an exponential weighting strategy, dynamically adjusting the fusion weights based on the cumulative loss of each base learner in the validation feedback, and weighting and fusing the outputs of each base learner to obtain the final draft strategy. Finally, the fused draft model is applied to the multi-step operation reasoning of the mobile terminal graphical user interface operation agent to generate operation instruction sequences such as screen click, swipe, text input, and application switching, thereby achieving adaptive optimization of the multi-step operation reasoning latency of the mobile terminal agent.
2. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The specific steps of the offline initialization phase are as follows: Step 100: Offline collection of historical operation datasets from the mobile terminal's graphical user interface. ,in This represents the total number of samples in the historical operation dataset. This represents a feature vector of the interface state, composed of screenshots, a tree of interface elements, and a sequence of historical operations. This indicates the corresponding target operation instruction label; Step 101, Select the target model ,in , Represents the parameter space of the target model. This represents the output distribution of the target model. This represents the input interface state sequence. The target model is a large language model, used to accurately predict and verify screen operation commands. Step 102, select the draft model ,in , Represents the parameter space of the draft model. This represents the output distribution of the draft model. The draft model represents the input interface state sequence and satisfies... , used to generate candidate operation instruction sequences; Step 103, Select the loss function This is used to measure the difference between the distribution of operation instructions predicted by the draft model and the distribution of output of the target model, where the input... express 3D real vector space, Indicates the output space; Step 104: On the historical dataset collected in step 100, using the draft model selected in step 102 and the loss function selected in step 103, obtain the offline initial parameters of the draft model by minimizing the loss function: ,in To initialize the model; Step 105, based on the initial draft model parameters obtained in step 104 Copy build Individual learners ,in For the first Each base learner is assigned a set of differentiated learning rates. ,in For the first The learning rate of each learner, and the initial weights of the meta-learner. For uniform distribution , .
3. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The specific steps by which the speculative decoding architecture performs multi-step operational reasoning and generates verification feedback are as follows: Step 200, at each time step Obtain the current interface status information of the mobile terminal. This includes screenshots, a tree of interface elements, and historical operation context; Step 201, from the current draft model Based on the current interface state, autoregressively and quickly generate a length of... Candidate operation instruction sequence Each operation instruction From conditional distribution Generated by sampling, where Indicates time step Output distribution of the draft model This indicates the current interface state and the prefix of the generated operation instructions; Step 202, Target Model Candidate operation instruction sequences are verified in parallel, and the target distribution at all positions is calculated in one forward propagation. ,in This represents the output distribution of the target model; Step 203, for each position Independently sampled uniform random numbers Calculate the likelihood ratio between the target model and the draft model at this location. ,in For the token at position j, Given the token preceding j, determine the length of the accepted operation instruction. ,in The interval is Uniform distribution; Step 204, if Then in the first The positions are from the corrected distribution. Mid-sample generates corrected operation instructions; if Then from the target distribution The next operation instruction is generated by sampling; the received operation instruction sequence is appended to the current operation sequence. Step 205: Generate feedback signals based on the validation results and construct the loss function. ,in Represents the parameter space of the draft model. This represents the output distribution of the draft model. This represents the sequence of input interface states. This represents the output distribution of the target model. That is, the cross-entropy loss between the target model output distribution and the draft model prediction distribution, which is generated naturally during the validation process.
4. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The specific steps for constructing and updating the multiple draft model base learners with differentiated learning rates are as follows: Step 300, Maintenance Individual learners ,in It is the first A draft model at time step The weights are set at different times, and each base learner is configured with a different learning rate. The base learner with a smaller learning rate is suitable for the gradual change scenario where the application interface is slowly iterated, while the base learner with a larger learning rate is suitable for the abrupt change scenario where the application task type is quickly switched. Step 301, at each time step Each base learner independently receives the validation feedback loss function. Calculate the gradient at each of their respective parameters. ; Step 302: Each base learner independently performs online gradient descent updates. ,in Represents the parameter constraint space The projection operator, For the first The learning rate of each base learner For the first The gradient of each base learner at the current parameters; Step 303, Updated parameters of each base learner It is passed to the meta-learner module for weighted fusion at the next time step.
5. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The specific steps for the meta-learner to dynamically adjust the fusion weights based on the cumulative loss of the verification feedback are as follows: Step 400: Initialize the fusion weights of the meta-learner to be uniformly distributed. , ,in The total number of base learners, Let i be the weight; Step 401, at each time step The meta-learner records the loss value of each base learner on the validation feedback loss function in the current round. , And update the cumulative loss of each base learner. ; Step 402: The meta-learner updates the fusion weights of each base learner using an exponential weighting strategy. ,in This is a sensitivity control parameter used to adjust the degree of response of the weights to the cumulative loss difference, which, after normalization, satisfies... ; Step 403: The meta-learner weights and fuses the outputs of each base learner to obtain the final draft model at the current time step. ,in For the first Each base learner at time step The fusion weight, For the first Each base learner at time step Model parameters.
6. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The alternative draft models include: a lightweight language model based on a Transformer decoder, a draft prediction head network attached to the hidden layers of the target model, and a draft model based on a feature retrieval mechanism.
7. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The available loss functions include: cross-entropy loss function, KL divergence loss function, mean squared error loss function, and DPO preference optimization loss function.
8. The online integrated speculative decoding method for mobile terminal graphical user interface operating intelligent agents according to claim 1, characterized in that, The methods for obtaining the current interface status information of a mobile terminal include: collecting interface element tree information through the Android Accessibility Service interface, obtaining the current screen image information through the screenshot interface, obtaining application package name and activity page name information through the system log interface, and obtaining operation instruction context information through operation history cache.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the online integrated speculative decoding method for operating intelligent agents with graphical user interfaces for mobile terminals as described in any one of claims 1-8.
10. A computer-readable storage medium storing a computer program that performs the online integrated speculative decoding method for operating an intelligent agent with a graphical user interface for a mobile terminal as described in any one of claims 1-8.