Mobile agent-oriented difficulty adaptive reinforcement learning method, device and equipment

Abstract

The invention particularly relates to a difficulty adaptive reinforcement learning method, device and equipment for a mobile agent, and the method comprises the steps: determining a sampling range in an interaction environment, and carrying out the multi-round sampling in the interaction environment based on the sampling range, and obtaining a multi-round sampling track; on the basis of a difficulty self-adaptive positive sample playback mechanism, calculating a dominant score of each round of sampling trajectory, sorting multiple rounds of sampling trajectories according to the dominant scores, screening trajectories corresponding to the dominant scores meeting a preset buffer condition from a sorting result to obtain multiple successful trajectories, and forming a high-quality playback pool on the basis of the multiple successful trajectories; and extracting a key track in the high-quality playback pool, and updating a strategy network of the mobile agent by utilizing the key track to mix with the online data according to a preset proportion, so as to optimize a pre-established reasoning warm start model according to the updated strategy network. Therefore, the problems of low exploration efficiency, poor generalization and the like of the mobile intelligent agent in an interaction environment are solved, and efficient generalization is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a difficulty-adaptive reinforcement learning method, apparatus, and device for mobile intelligent agents. Background Technology

[0002] In recent years, with the continuous enhancement of the capabilities of Large Language Models (LLMs) in multimodal perception, natural language understanding, and task reasoning, researchers have gradually constructed intelligent agent systems with autonomously controllable graphical interfaces. These systems can directly perform graphical interface operation tasks on mobile devices (such as the Android system) based on natural language instructions. These intelligent agents are usually called Mobile GUI Agents, and their core goal is to achieve multi-turn interaction, rational planning, and accurate execution of operation instructions on user interfaces with dynamic state changes and complex task dependencies.

[0003] Most methods in related technologies are built upon robust language models, generating action sequences such as taps, swipes, and text input by perceiving screenshots or interface structures (e.g., XML trees). To enhance the interactive capabilities of these agents, various system frameworks and algorithms have been proposed. Simultaneously, these technologies also attempt to train agents using behavior cloning or reinforcement learning. The community is actively promoting the construction of evaluation benchmarks for Android GUI agents to facilitate objective comparison and development of model capabilities. Recently emerging environments not only support progressive action execution and state feedback but also place higher demands on the model's perception capabilities, task planning abilities, and fault tolerance.

[0004] However, most methods in related technologies remain limited to offline training paradigms, relying solely on static, manually generated demonstration data. They lack online interaction and proactive exploration capabilities, making it difficult to adapt to interface changes and task diversity. Many works employ single-round interactive modeling, where each sample contains only one action, failing to capture multi-step reasoning and long-term planning capabilities. Although current benchmarks cover a wide range of application scenarios and configuration commands, the rapid updates to mobile app interface designs and the continuous changes in task objectives mean that building a comprehensive, stable, and representative benchmark system remains a long-standing and unresolved issue. Summary of the Invention

[0005] This application provides a difficulty-adaptive reinforcement learning method, apparatus, and device for mobile intelligent agents to address the problems of low exploration efficiency and poor generalization of mobile intelligent agents in interactive environments.

[0006] The first aspect of this application provides a difficulty-adaptive reinforcement learning method for mobile intelligent agents, including the following steps: Determine the sampling range in the interactive environment, and based on the sampling range, perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories; Based on the difficulty-adaptive positive sample replay mechanism, the advantage score of the sampling trajectory in each round is calculated. The sampling trajectories in multiple rounds are sorted according to the advantage score. Trajectories with advantage scores that meet the preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories. A high-quality replay pool is formed based on the multiple successful trajectories. Key trajectories are extracted from the high-quality replay pool, and the policy network of the mobile agent is updated using the key trajectories mixed with online data at a preset ratio, so as to optimize the pre-established inference hot-start model based on the updated policy network.

[0007] Optionally, before optimizing the pre-established inference hot-start model based on the updated policy network, the method further includes: Based on the preset action trajectory dataset and pre-trained inference model, multiple inference-action pairs are generated, and the final action sequence corresponding to each inference-action pair is determined. Based on the preset expert action sequence, inference-action pairs that meet the preset matching conditions are selected from the final action sequences corresponding to all inference-action pairs to obtain the initial augmented dataset. The initial inference policy model is obtained by supervised fine-tuning training on the initial augmented dataset; Based on preset screening conditions, target inference explanations are selected from the initial augmented dataset, and the initial augmented dataset is updated based on the target inference explanations until the output of the initial inference strategy model and the expert behavior matching rate in the preset action trajectory dataset reach a preset stable condition, thus obtaining the pre-established inference hot start model.

[0008] Optionally, the step of performing multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories includes: Multiple Android virtual devices are encapsulated using Docker containers; A centralized controller based on the gRPC protocol uniformly schedules and manages the concurrent sampling tasks of the multiple Android virtual devices, enabling the multiple Android virtual devices to run independently, perform parallel data acquisition, and obtain the current round of sampling trajectory; An asynchronous sampling-training decoupling mechanism is adopted to upload the current round of sampling trajectory to a shared data buffer, read data in batches from the shared data buffer and asynchronously send it to the sampling end for a new round of sampling, and obtain the results of the multiple rounds of sampling.

[0009] Optionally, when updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the method further includes: Based on a preset failed task course filtering mechanism, the number of consecutive failures of each task in the current training task pool is counted when the policy network is updated. Identify the first task whose consecutive failure count reaches a first threshold, and dynamically reduce the probability of the first task being sampled based on a preset downsampling strategy; Identify a second task whose consecutive failure count reaches a second threshold, and remove the second task from the current training task pool, wherein the second threshold is greater than the first threshold.

[0010] Optionally, when updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the method further includes: Based on the key trajectory, all successful paths in the current training task pool are determined, and the shortest path in the current training task pool is selected from all successful paths. According to the preset successful path shortening incentive mechanism, the final reward of each successful path is normalized according to the relative difference between the length of the corresponding successful path and the length of the shortest path to obtain the final reward of each successful path. The policy network of the mobile agent is updated based on the final reward for all successful paths.

[0011] A second aspect of this application provides a difficulty-adaptive reinforcement learning device for mobile intelligent agents, comprising: A sampling module is used to determine the sampling range in the interactive environment, and based on the sampling range, to perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories; The filtering module is used to calculate the advantage score of the sampling trajectory in each round based on the difficulty adaptive positive sample replay mechanism, sort the sampling trajectories in multiple rounds according to the advantage score, and filter the trajectory corresponding to the advantage score that meets the preset buffer condition from the sorting results to obtain multiple successful trajectories, and form a high-quality replay pool based on the multiple successful trajectories. An optimization module is used to extract key trajectories from the high-quality playback pool and use the key trajectories to update the policy network of the mobile agent after mixing with online data at a preset ratio, so as to optimize the pre-established inference hot-start model according to the updated policy network.

[0012] Optionally, before optimizing the pre-established inference hot-start model according to the updated policy network, the optimization module is further configured to: Based on the preset action trajectory dataset and pre-trained inference model, multiple inference-action pairs are generated, and the final action sequence corresponding to each inference-action pair is determined. Based on the preset expert action sequence, inference-action pairs that meet the preset matching conditions are selected from the final action sequences corresponding to all inference-action pairs to obtain the initial augmented dataset. The initial inference policy model is obtained by supervised fine-tuning training on the initial augmented dataset; Based on preset screening conditions, target inference explanations are selected from the initial augmented dataset, and the initial augmented dataset is updated based on the target inference explanations until the output of the initial inference strategy model and the expert behavior matching rate in the preset action trajectory dataset reach a preset stable condition, thus obtaining the pre-established inference hot start model.

[0013] Optionally, the sampling module is specifically used for: Multiple Android virtual devices are encapsulated using Docker containers; A centralized controller based on the gRPC protocol uniformly schedules and manages the concurrent sampling tasks of the multiple Android virtual devices, enabling the multiple Android virtual devices to run independently, perform parallel data acquisition, and obtain the current round of sampling trajectory; An asynchronous sampling-training decoupling mechanism is adopted to upload the current round of sampling trajectory to a shared data buffer, read data in batches from the shared data buffer and asynchronously send it to the sampling end for a new round of sampling, and obtain the results of the multiple rounds of sampling.

[0014] Optionally, when updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the optimization module is further configured to: Based on a preset failed task course filtering mechanism, the number of consecutive failures of each task in the current training task pool is counted when the policy network is updated. Identify the first task whose consecutive failure count reaches a first threshold, and dynamically reduce the probability of the first task being sampled based on a preset downsampling strategy; Identify a second task whose consecutive failure count reaches a second threshold, and remove the second task from the current training task pool, wherein the second threshold is greater than the first threshold.

[0015] Optionally, when updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the optimization module is further configured to: Based on the key trajectory, all successful paths in the current training task pool are determined, and the shortest path in the current training task pool is selected from all successful paths. According to the preset successful path shortening incentive mechanism, the final reward of each successful path is normalized according to the relative difference between the length of the corresponding successful path and the length of the shortest path to obtain the final reward of each successful path. The policy network of the mobile agent is updated based on the final reward for all successful paths.

[0016] A third aspect of this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform a difficulty-adaptive reinforcement learning method for mobile intelligent agents as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the difficulty-adaptive reinforcement learning method for mobile intelligent agents as described in the above embodiments.

[0018] A fifth aspect of this application provides a computer program product storing a computer program that, when executed by a processor, implements the difficulty-adaptive reinforcement learning method for mobile intelligent agents as described in the above embodiments.

[0019] Therefore, this application embodiment determines the sampling range in the interactive environment and, based on the sampling range, performs multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories. Based on the difficulty-adaptive positive sample replay mechanism, the advantage score of each round of sampling trajectory is calculated. The multiple rounds of sampling trajectories are sorted according to the advantage scores, and the trajectories corresponding to the advantage scores that meet the preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories. A high-quality replay pool is formed based on multiple successful trajectories. Key trajectories are extracted from the high-quality replay pool, and the policy network of the mobile agent is updated using the key trajectories mixed with online data at a preset ratio. The pre-established inference hot-start model is then optimized based on the updated policy network. This solves the problems of low exploration efficiency and poor generalization of mobile agents in interactive environments, achieving efficient generalization.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of a difficulty-adaptive reinforcement learning method for mobile intelligent agents according to an embodiment of this application; Figure 2 This is a schematic diagram of the AndroidRL algorithm flow for a difficulty-adaptive reinforcement learning method for mobile intelligent agents according to an embodiment of this application; Figure 3 This is a schematic diagram of a difficulty-adaptive reinforcement learning device for mobile intelligent agents provided according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0023] The following describes a difficulty-adaptive reinforcement learning method, apparatus, and device for mobile intelligent agents according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems of low exploration efficiency and poor generalization of mobile intelligent agents in interactive environments mentioned in the background art, this application provides a difficulty-adaptive reinforcement learning method for mobile intelligent agents. In this method, embodiments of this application determine the sampling range in the interactive environment and, based on the sampling range, perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories. Based on a difficulty-adaptive positive sample replay mechanism, calculate the advantage score of each round of sampling trajectory, sort the multiple rounds of sampling trajectories according to the advantage scores, and select trajectories corresponding to advantage scores that meet preset buffer conditions from the sorting results to obtain multiple successful trajectories. A high-quality replay pool is constructed based on these multiple successful trajectories. Key trajectories are extracted from the high-quality replay pool, and the policy network of the mobile intelligent agent is updated using the key trajectories mixed with online data at a preset ratio. The pre-established inference hot-start model is then optimized based on the updated policy network. This solves the problems of low exploration efficiency and poor generalization of mobile intelligent agents in interactive environments. Specifically, Figure 1 This is a flowchart illustrating a difficulty-adaptive reinforcement learning method for mobile intelligent agents provided in an embodiment of this application.

[0024] like Figure 1 As shown, this difficulty-adaptive reinforcement learning method for mobile agents includes the following steps: In step S101, the sampling range in the interactive environment is determined, and based on the sampling range, multiple rounds of sampling are performed in the interactive environment to obtain the multiple rounds of sampling trajectory. Specifically, in this embodiment of the application, a sampling range is determined in a test field consisting of a large-scale parallel Android simulation environment, and multiple samplings are performed within this range to form an interaction trajectory, resulting in a multi-round sampling trajectory; wherein, the sampling trajectory is a complete interaction record, which typically includes information such as task instructions, multi-round state sequences, multi-round action sequences, and reward signals.

[0025] Optionally, in some embodiments, multiple rounds of sampling are performed in an interactive environment to obtain multiple rounds of sampling trajectories, including: encapsulating multiple Android virtual devices using Docker containers; using a centralized controller based on the gRPC protocol to uniformly schedule and manage the concurrent sampling tasks of multiple Android virtual devices, enabling multiple Android virtual devices to run independently and perform parallel data acquisition to obtain the current round of sampling trajectory; and using an asynchronous sampling-training decoupling mechanism to upload the current round of sampling trajectory to a shared data buffer, read data in batches from the shared data buffer and asynchronously send it to the sampling end for a new round of sampling to obtain multiple rounds of sampling results.

[0026] Understandably, this invention designs a high-concurrency, high-throughput Android GUI interactive sampling system to address the bottlenecks of slow environment operation and poor parallelism in traditional reinforcement learning training. The system encapsulates Android Virtual Devices (AVDs) in Docker containers, with each container independently running a complete Android emulator instance and providing graphical interaction capabilities. All containers are scheduled and managed by a centralized gRPC controller, which coordinates task distribution, trajectory sampling, and state collection processes. The system supports multi-machine deployment, with each host capable of running multiple parallel AVD containers, and the overall system can be scaled to support over 500 concurrent simulation environments. Compared to traditional solutions based on physical devices or serial emulators, this embodiment significantly improves the reinforcement learning sample generation rate, ensuring sufficient and diverse data support during policy updates. To address the resource waste caused by differences in environment response speed during reinforcement learning training, this invention employs an asynchronous sampling-training decoupling mechanism; the training system consists of multiple independent components, including a sampling engine, a data queue, a model trainer, and a policy inference engine. The sampling process interacts with the environment in real time, uploading generated trajectories to a shared data buffer. The trainer independently reads trajectories in batches and optimizes policies, without waiting for all sampling tasks to complete before processing. The inference engine supports asynchronous model updates, avoiding policy shifts caused by old policies remaining for too long or updating too frequently. This asynchronous architecture maximizes the utilization of GPU computing and CPU parallel capabilities, significantly improving overall system throughput and training stability, and is particularly suitable for scenarios in reinforcement learning where large-scale environment sampling and complex model training are conducted in parallel.

[0027] In step S102, based on the difficulty adaptive positive sample replay mechanism, the advantage score of each round of sampling trajectory is calculated, the multiple rounds of sampling trajectories are sorted according to the advantage score, and the trajectories corresponding to the advantage scores that meet the preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories, and a high-quality replay pool is formed based on the multiple successful trajectories.

[0028] Specifically, in this embodiment of the application, a Difficulty-Adaptive Positive Replay (DAPR) mechanism is used to globally score the sampling trajectory in each round to obtain the advantage score of the sampling trajectory in each round. After each training cycle, the sampling trajectories are sorted according to the advantage function score at the trajectory level. Then, the top 25% of high-advantage successful trajectories are selected to form a high-quality replay pool.

[0029] In step S103, key trajectories are extracted from the high-quality playback pool, and the policy network of the mobile agent is updated by mixing the key trajectories with online data at a preset ratio, so as to optimize the pre-established inference hot-start model according to the updated policy network.

[0030] Specifically, unlike traditional reinforcement learning methods that rely solely on online samples, this application employs a hybrid online-offline sampling strategy during training. The top 25% of high-advantage, high-difficulty trajectories are retained in the replay buffer, and online data is mixed in at a rate of 25% during training. Key trajectories and online trajectories are used together to update the policy network, and the pre-built inference hot-start model is optimized based on the updated policy network. This ensures that rare but valuable trajectories in high-difficulty tasks are not forgotten, while simultaneously increasing the proportion of high-quality samples during training, thereby improving the model's stability and accuracy in complex environments. It should be noted that each sampled trajectory is retained for a maximum of four training epochs to avoid overfitting to historical data.

[0031] Optionally, in some embodiments, when updating the policy network of the mobile agent using key trajectories mixed with online data at a preset ratio, the method further includes: based on a preset failed task course filtering mechanism, counting the number of consecutive failures of each task in the current training task pool when updating the policy network; identifying a first task whose number of consecutive failures reaches a first threshold, and dynamically reducing the probability of the first task being sampled based on a preset downsampling strategy; identifying a second task whose number of consecutive failures reaches a second threshold, and removing the second task from the current training task pool, wherein the second threshold is greater than the first threshold.

[0032] The first threshold and the second threshold can be thresholds preset by the user, thresholds obtained through a limited number of experiments, or thresholds obtained through a limited number of computer simulations; no specific limitations are imposed here.

[0033] Understandably, to avoid training resources being consumed by tasks with long-term failures, this invention proposes a Failure Curriculum Filtering (FCF) mechanism. This mechanism dynamically adjusts the probability of a task participating in training by counting the number of consecutive failures for each task, based on a preset downsampling strategy. For example, if a task fails completely in two consecutive training cycles (the first threshold), the system marks it as a "cooling-off task," and its sampling probability is gradually reduced exponentially over the next three cycles. If a task accumulates more than three failures during training (the second threshold), it will be permanently removed from the training task pool. FCF effectively shields tasks that the model cannot yet solve in the short term, preventing them from interfering with the main task's learning trajectory, thereby improving overall training efficiency and sample quality.

[0034] Optionally, in some embodiments, when updating the policy network of the mobile agent using key trajectories mixed with online data at a preset ratio, the method further includes: determining all successful paths in the current training task pool based on the key trajectories, and selecting the shortest path in the current training task pool from all successful paths; normalizing the final reward of each successful path according to the relative difference between the length of the corresponding successful path and the length of the shortest path according to a preset successful path shortening incentive mechanism, to obtain the final reward of each successful path; and updating the policy network of the mobile agent based on the final rewards of all successful paths.

[0035] Understandably, traditional reinforcement learning assigns the same reward to all successful paths, which can lead to longer paths contributing more gradient updates due to their higher number of steps, causing the model to tend towards verbose strategies. To address this, this invention introduces a reward adjustment mechanism based on path length, namely the Success Pathway Adjustment (SPA). The final reward for all successful paths is discounted based on the relative difference between the path length and the shortest path length in the batch. The reward for each successful path is then adjusted using length normalization to obtain the final reward for each successful path. The calculation formula is as follows: ; in, The shortest success path length within the current batch. The recommended default value is 1.0. This normalization process ensures that the reinforcement learning in the embodiments of this application tends to select more efficient solution paths, improve policy quality and generalization ability, and update the policy network of the mobile agent.

[0036] Optionally, in some embodiments, before optimizing the pre-established inference hot-start model according to the updated policy network, the method further includes: generating multiple inference-action pairs based on a preset action trajectory dataset and a pre-trained inference model, determining the final action sequence corresponding to each inference-action pair, and selecting inference-action pairs that meet preset matching conditions from the final action sequences corresponding to all inference-action pairs based on preset expert action sequences to obtain an initial augmented dataset; performing supervised fine-tuning training on the initial augmented dataset to obtain an initial inference policy model; selecting target inference explanations from the initial augmented dataset based on preset selection conditions, and updating the initial augmented dataset based on the target inference explanations until the output of the initial inference policy model and the expert behavior matching rate in the preset action trajectory dataset reach a preset stable condition, thereby obtaining the pre-established inference hot-start model.

[0037] Understandably, traditional supervised fine-tuning based on expert data typically only includes the final behavior sequence and lacks intermediate reasoning processes, resulting in the agent's policy lacking structured task understanding and difficulty in generalizing to new environments. Furthermore, the quality of expert data varies, and some information may be missing. This application addresses the problem of expert data only containing the final behavior and lacking intermediate reasoning steps by utilizing a pre-trained reasoning model to generate reasoning-behavior pairs and iteratively applying them.

[0038] Specifically, this embodiment of the application, given task instances and expert behaviors (pre-defined expert action sequences) from an expert dataset, employs a pre-trained inference model and uses different decoding strategies to generate multiple inference-action pair candidates; retains inference-action pairs that perfectly match the expert actions, forming an initial augmentation dataset. The initial inference model is then supervised and fine-tuned using the initial augmentation dataset to learn preliminary inference augmentation strategies. In each iteration, multiple inference-action pair candidates are generated for all tasks using the current model. The best inference explanation is selected based on the degree of matching with expert actions, the simplicity of the inference steps, and probability scores (pre-defined selection criteria). The initial augmentation dataset is updated, and the initial inference strategy model is further fine-tuned based on this; the above iterations are repeated until the output of the initial inference strategy model and the expert behavior matching rate stabilize, resulting in the final inference hot-start model (a pre-established inference hot-start model).

[0039] Therefore, this application's embodiments iteratively enhance the expert dataset through an iterative inference-enhanced hot-start strategy and multi-round self-supervised inference behavior; optimize the relative strategy of difficulty-adaptive grouping, including difficulty-adaptive positive sample replay and failed task course filtering, finely adjusting the reinforcement learning process according to task difficulty; and utilize an efficient and scalable distributed environment sampling system based on Docker containerized Android virtual devices, employing gRPC centralized management to achieve parallel large-scale sampling. Compared to related technologies, this invention significantly improves the generalization and stability of GUI agents in real mobile application interaction tasks, outperforming closed-source and open-source methods of related technologies, and is suitable for agent reinforcement learning training in various complex interaction scenarios.

[0040] To facilitate a better understanding of the difficulty-adaptive reinforcement learning method for mobile intelligent agents in the embodiments of this application, the following is combined with... Figure 2 The embodiments shown will be described in detail.

[0041] Specifically, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the AndroidRL algorithm flow for a difficulty-adaptive reinforcement learning method for mobile agents, according to one embodiment of this application. This invention proposes a reinforcement learning framework called AndroidRL to improve the exploration ability, generalization performance, and training efficiency of large language model agents in multi-turn interactive tasks using graphical user interfaces. This invention systematically integrates three key technical modules: iterative inference-enhanced training, difficulty-adaptive grouping relative policy optimization (DA-GRPO), and distributed environment sampling architecture.

[0042] Furthermore, initially utilizing the final action trajectory dataset provided by human experts, a pre-trained Instruct inference model is used to automatically generate inference-action pairs containing intermediate inference steps and behaviors, forming a preliminary augmented dataset. Supervised fine-tuning training is then performed using this augmented dataset to obtain the initial inference policy model. Multiple iterations are then performed using the inference policy model, with the optimal data selected in each round based on the matching degree with expert behavior and the simplicity of the inference steps, gradually improving the model's inference quality and policy generalization ability.

[0043] Furthermore, the difficulty-adaptive grouping relative strategy improves upon the traditional GRPO algorithm by proposing a difficulty-adaptive training mechanism tailored to the complexity of GUI interaction tasks. In the difficulty-adaptive positive sample replay (DAPR) module, after each round of sampling, a trajectory advantage score is calculated. High-quality but difficult successful trajectories are stored in a replay buffer and mixed with online sampled data in subsequent training to improve the model's learning performance on high-difficulty tasks. Failed task course filtering tracks the success rate of each task, gradually reducing the sampling probability of tasks with continuous failures, ultimately removing them from training to effectively concentrate resources on trainable tasks. Success path adjustment normalizes the reward based on the length of successful trajectories, significantly encouraging the model to generate more efficient and shorter successful paths, thereby improving the efficiency and robustness of the final model strategy.

[0044] Furthermore, this embodiment of the application uses Docker containers to encapsulate multiple Android virtual devices, each running independently to achieve parallel data acquisition. A centralized controller is built using the gRPC protocol to uniformly schedule and manage the concurrent sampling tasks of a large number of virtual devices. The sampling system supports the parallel operation of more than 500 virtual environments, greatly improving the sampling efficiency of RL training data and the system scalability.

[0045] Therefore, the embodiments of this application, through the cooperation of the above modules, constitute an efficient and robust Android RL reinforcement learning framework, breaking through the performance bottleneck of traditional reinforcement learning in complex GUI interaction environments.

[0046] Furthermore, to verify the effectiveness of the AndroidRL framework proposed in this invention in real-world Android application scenarios, this invention conducted a comprehensive experimental evaluation based on two mainstream interactive graphical user interface (GUI) intelligent agent evaluation benchmarks: AndroidLab and AndroidWorld. These two benchmarks cover a total of 254 real-world tasks, encompassing 20 common mobile applications (such as Settings, Dialer, Messages, Photos, File Manager, etc.). All test tasks are based on natural language commands and require the intelligent agent to complete multiple rounds of clicks, swipes, and inputs during the interaction to achieve the target behavior.

[0047] Specifically, in the experiments, the method of this invention was compared with the previous best results on these two benchmarks. All models ran in a unified graphical user interface interaction space, with specific actions including tap, long-tap, swipe, enter-text, and back. Each action had specific parameters (such as position coordinates, text content, etc.) provided by a standardized environment interface.

[0048] Table 1

[0049] As shown in Table 1, the experimental results of the embodiments of this application demonstrate that the complete training scheme of AndroidRL achieves significant performance improvements on both benchmarks. In the AndroidLab environment, the final success rate of the proposed method reaches 44.5%, which is 6.2 percentage points higher than the current best open-source agent V-Droid's 38.3%, representing an increase of 23.6%. In the more challenging AndroidWorld environment, AndroidRL achieves a success rate of 72.9%, significantly surpassing its own hot-start model (56.8%) and improving upon the closed-source system UI-Tars-1.5's 64.2% by 8.7 percentage points, representing a relative increase of 28.3%, demonstrating superior task generalization and multi-turn interaction capabilities.

[0050] It is worth noting that even without the introduction of online reinforcement learning, the "inference enhancement training" proposed in this invention, namely the AndroidRL-Warm up model in the table, has achieved performance close to or better than previous models in both environments, proving the effectiveness of inference trajectories in the early stages of agent training. Combining the complete DA-GRPO policy optimization algorithm with a large-scale distributed concurrent environment sampling system further enhances the model's performance, validating the powerful practical value and technological advancement of this invention in real-world mobile interaction tasks.

[0051] According to the difficulty-adaptive reinforcement learning method for mobile agents proposed in this application, the sampling range in the interactive environment is determined, and multiple rounds of sampling are performed in the interactive environment based on the sampling range to obtain multiple rounds of sampling trajectories. Based on the difficulty-adaptive positive sample replay mechanism, the advantage score of each round of sampling trajectory is calculated. The multiple rounds of sampling trajectories are sorted according to the advantage scores, and the trajectories corresponding to the advantage scores that meet the preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories. A high-quality replay pool is formed based on multiple successful trajectories. Key trajectories are extracted from the high-quality replay pool, and the policy network of the mobile agent is updated using the key trajectories mixed with online data at a preset ratio. The pre-established inference hot-start model is then optimized based on the updated policy network. This solves the problems of low exploration efficiency and poor generalization of mobile agents in interactive environments. Next, with reference to the accompanying drawings, a difficulty-adaptive reinforcement learning device for mobile intelligent agents proposed according to embodiments of this application is described.

[0052] Figure 3 This is a block diagram of a difficulty-adaptive reinforcement learning device for mobile intelligent agents according to an embodiment of this application.

[0053] like Figure 3 As shown, the difficulty-adaptive reinforcement learning device 10 for mobile intelligent agents includes: a sampling module 100, a filtering module 200, and an optimization module 300.

[0054] The sampling module 100 is used to determine the sampling range in the interactive environment and, based on the sampling range, perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories. The filtering module 200 is used to calculate the advantage score of each round of sampling trajectory based on the difficulty adaptive positive sample replay mechanism, sort the multiple rounds of sampling trajectory according to the advantage score, and filter the trajectory corresponding to the advantage score that meets the preset buffer conditions from the sorting results to obtain multiple successful trajectories, and form a high-quality replay pool based on multiple successful trajectories. The optimization module 300 is used to extract key trajectories from the high-quality playback pool and use the key trajectories to update the policy network of the mobile agent after mixing with online data at a preset ratio, so as to optimize the pre-established inference hot-start model based on the updated policy network.

[0055] Optionally, before optimizing the pre-established inference hot-start model according to the updated policy network, the optimization module 300 is further configured to: generate multiple inference-action pairs based on the preset action trajectory dataset and the pre-trained inference model, determine the final action sequence corresponding to each inference-action pair, and based on the preset expert action sequence, select inference-action pairs that meet the preset matching conditions from the final action sequences corresponding to all inference-action pairs to obtain an initial augmented dataset; perform supervised fine-tuning training on the initial augmented dataset to obtain an initial inference policy model; select target inference explanations from the initial augmented dataset based on the preset selection conditions, and update the initial augmented dataset based on the target inference explanations until the output of the initial inference policy model and the expert behavior matching rate in the preset action trajectory dataset reach the preset stable condition, thereby obtaining the pre-established inference hot-start model.

[0056] Optionally, the sampling module 100 is specifically used for: encapsulating multiple Android virtual devices using Docker containers; using a centralized controller based on the gRPC protocol to uniformly schedule and manage the concurrent sampling tasks of multiple Android virtual devices, enabling multiple Android virtual devices to run independently and perform parallel data acquisition to obtain the current round of sampling trajectory; and using an asynchronous sampling-training decoupling mechanism to upload the current round of sampling trajectory to a shared data buffer, read data in batches from the shared data buffer and asynchronously send it to the sampling end for a new round of sampling to obtain multiple rounds of sampling results.

[0057] Optionally, when updating the policy network of the mobile agent using key trajectories mixed with online data at a preset ratio, the optimization module 300 is further configured to: based on a preset failed task course filtering mechanism, count the number of consecutive failures of each task in the current training task pool when updating the policy network; identify the first task whose number of consecutive failures reaches a first threshold, and dynamically reduce the probability of the first task being sampled based on a preset downsampling strategy; identify the second task whose number of consecutive failures reaches a second threshold, and remove the second task from the current training task pool, wherein the second threshold is greater than the first threshold.

[0058] Optionally, when updating the policy network of the mobile agent using key trajectories mixed with online data at a preset ratio, the optimization module 300 is further configured to: determine all successful paths in the current training task pool based on the key trajectories, and select the shortest path in the current training task pool from all successful paths; normalize the final reward of each successful path according to the relative difference between the length of the corresponding successful path and the length of the shortest path according to a preset successful path shortening incentive mechanism, and obtain the final reward of each successful path; update the policy network of the mobile agent based on the final rewards of all successful paths.

[0059] It should be noted that the foregoing explanation of the difficulty-adaptive reinforcement learning method embodiment for mobile intelligent agents also applies to the difficulty-adaptive reinforcement learning device for mobile intelligent agents in this embodiment, and will not be repeated here.

[0060] According to the difficulty-adaptive reinforcement learning device for mobile agents proposed in this application, the following steps are taken: First, a sampling range is determined in the interactive environment. Based on this range, multiple rounds of sampling are performed in the interactive environment to obtain multiple sampling trajectories. Second, based on a difficulty-adaptive positive sample replay mechanism, the advantage score of each round of sampling trajectory is calculated. Third, the multiple rounds of sampling trajectories are sorted according to their advantage scores, and trajectories with advantage scores that meet preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories. These successful trajectories form a high-quality replay pool. Fourth, key trajectories are extracted from the high-quality replay pool, and these key trajectories are mixed with online data at a preset ratio to update the mobile agent's policy network. Finally, the updated policy network is used to optimize a pre-established inference hot-start model. This solves the problems of low exploration efficiency and poor generalization of mobile agents in interactive environments. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0061] When the processor 402 executes the program, it implements the difficulty-adaptive reinforcement learning method for mobile intelligent agents provided in the above embodiments.

[0062] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0063] The memory 401 is used to store computer programs that can run on the processor 402.

[0064] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0065] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0066] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0067] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0068] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described difficulty-adaptive reinforcement learning method for mobile intelligent agents.

[0069] This application also provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described difficulty-adaptive reinforcement learning method for mobile intelligent agents.

[0070] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0071] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0072] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0073] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0074] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

Claims

1. A difficulty-adaptive reinforcement learning method for mobile intelligent agents, characterized in that, Includes the following steps: Determine the sampling range in the interactive environment, and based on the sampling range, perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories; Based on the difficulty-adaptive positive sample replay mechanism, the advantage score of the sampling trajectory in each round is calculated. The sampling trajectories in multiple rounds are sorted according to the advantage score. Trajectories with advantage scores that meet the preset buffer conditions are selected from the sorting results to obtain multiple successful trajectories. A high-quality replay pool is formed based on the multiple successful trajectories. Key trajectories are extracted from the high-quality replay pool, and the policy network of the mobile agent is updated using the key trajectories mixed with online data at a preset ratio, so as to optimize the pre-established inference hot-start model based on the updated policy network.

2. The method according to claim 1, characterized in that, Before optimizing the pre-built inference hot-start model based on the updated policy network, the following steps are also included: Based on the preset action trajectory dataset and pre-trained inference model, multiple inference-action pairs are generated, and the final action sequence corresponding to each inference-action pair is determined. Based on the preset expert action sequence, inference-action pairs that meet the preset matching conditions are selected from the final action sequences corresponding to all inference-action pairs to obtain the initial augmented dataset. The initial inference policy model is obtained by supervised fine-tuning training on the initial augmented dataset; Based on preset screening conditions, target inference explanations are selected from the initial augmented dataset, and the initial augmented dataset is updated based on the target inference explanations until the output of the initial inference strategy model and the expert behavior matching rate in the preset action trajectory dataset reach a preset stable condition, thus obtaining the pre-established inference hot start model.

3. The method according to claim 1, characterized in that, The process of performing multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories includes: Multiple Android virtual devices are encapsulated using Docker containers; A centralized controller based on the gRPC protocol uniformly schedules and manages the concurrent sampling tasks of the multiple Android virtual devices, enabling the multiple Android virtual devices to run independently, perform parallel data acquisition, and obtain the current round of sampling trajectory; An asynchronous sampling-training decoupling mechanism is adopted to upload the current round of sampling trajectory to a shared data buffer, read data in batches from the shared data buffer and asynchronously send it to the sampling end for a new round of sampling, and obtain the results of the multiple rounds of sampling.

4. The method according to claim 1, characterized in that, When updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the method further includes: Based on a preset failed task course filtering mechanism, the number of consecutive failures of each task in the current training task pool is counted when the policy network is updated. Identify the first task whose consecutive failure count reaches a first threshold, and dynamically reduce the probability of the first task being sampled based on a preset downsampling strategy; Identify a second task whose consecutive failure count reaches a second threshold, and remove the second task from the current training task pool, wherein the second threshold is greater than the first threshold.

5. The method according to claim 1, characterized in that, When updating the policy network of the mobile agent using the key trajectory mixed with online data at a preset ratio, the method further includes: Based on the key trajectory, all successful paths in the current training task pool are determined, and the shortest path in the current training task pool is selected from all successful paths. According to the preset successful path shortening incentive mechanism, the final reward of each successful path is normalized according to the relative difference between the length of the corresponding successful path and the length of the shortest path to obtain the final reward of each successful path. The policy network of the mobile agent is updated based on the final reward for all successful paths.

6. A difficulty-adaptive reinforcement learning device for mobile intelligent agents, characterized in that, include: A sampling module is used to determine the sampling range in the interactive environment, and based on the sampling range, to perform multiple rounds of sampling in the interactive environment to obtain multiple rounds of sampling trajectories; The filtering module is used to calculate the advantage score of the sampling trajectory in each round based on the difficulty adaptive positive sample replay mechanism, sort the sampling trajectories in multiple rounds according to the advantage score, and filter the trajectory corresponding to the advantage score that meets the preset buffer condition from the sorting results to obtain multiple successful trajectories, and form a high-quality replay pool based on the multiple successful trajectories. An optimization module is used to extract key trajectories from the high-quality playback pool and use the key trajectories to update the policy network of the mobile agent after mixing with online data at a preset ratio, so as to optimize the pre-established inference hot-start model according to the updated policy network.

7. The apparatus according to claim 6, characterized in that, Before optimizing the pre-established inference hot-start model according to the updated policy network, the optimization module is further configured to: Based on the preset action trajectory dataset and pre-trained inference model, multiple inference-action pairs are generated, and the final action sequence corresponding to each inference-action pair is determined. Based on the preset expert action sequence, inference-action pairs that meet the preset matching conditions are selected from the final action sequences corresponding to all inference-action pairs to obtain the initial augmented dataset. The initial inference policy model is obtained by supervised fine-tuning training on the initial augmented dataset; Based on preset screening conditions, target inference explanations are selected from the initial augmented dataset, and the initial augmented dataset is updated based on the target inference explanations until the output of the initial inference strategy model and the expert behavior matching rate in the preset action trajectory dataset reach a preset stable condition, thus obtaining the pre-established inference hot start model.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the difficulty-adaptive reinforcement learning method for mobile agents as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the difficulty-adaptive reinforcement learning method for mobile agents as described in any one of claims 1-5.

10. A computer program product, said computer program product storing a computer program, characterized in that, When the program is executed by the processor, it implements the difficulty-adaptive reinforcement learning method for mobile agents as described in any one of claims 1-5.

Images