Strategy learning method and system for Android agent beyond behavior cloning

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a strategy learning method and system for Android agent beyond behavior cloning, and the method comprises the steps: 1) carrying out the behavior cloning of an Android agent base model according to expert demonstration data; and 2) constructing a reward function on the basis of the Android agent base model after behavior cloning, and performing off-line reinforcement learning RWR and AWR training on the reward function according to expert demonstration data to obtain a final Android agent. According to the method, a strategy learning method exceeding traditional behavior cloning is designed by combining the advantages of behavior cloning and offline reinforcement learning, compared with a traditional behavior cloning method, the method can learn a better strategy, and compared with implicit behavior cloning, generative adversarial imitation learning and other methods, the method is more suitable for the field of Android agents; moreover, the method occupies less resources, is simple in training mode, does not need to train a discriminator network compared with an offline reinforcement learning method, and is also effective under the condition of sparse rewards.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to a strategy learning method and system for an Android agent, and in particular to a strategy learning method and system for an Android agent that transcends behavior cloning. Background Art

[0002] Smart terminals such as mobile phones are excellent carriers for the implementation of big model technology. The combination of smart terminals and big models can bring users subversive product experience and highly intelligent services. Currently, the smart assistants and RPA tools on the market still have multiple limitations in assisting users to complete operations. The scope of application of their functions is extremely limited, and they are not suitable for highly variable tasks. The initial deployment and maintenance costs are high, and there are also many difficulties in understanding users' complex language instructions and context associations.

[0003] Android Agent is a software system based on artificial intelligence technology. It can be designed to autonomously perform specific tasks or solve problems. It can run in a specific environment and has certain autonomous decision-making and task planning capabilities. The agent analyzes tasks based on a "brain" composed of a large model, designs and plans problem solutions and paths, and has the ability to call tools and perform operations. As a new technical solution, it can be used to solve many problems faced by mobile phone intelligent control tasks. However, the Android agent must learn strategies before it can perform tasks.

[0004] Behavior cloning is a traditional strategy learning method that requires a large amount of expert data and uses mean square error loss or cross entropy loss to perform gradient descent to learn expert strategies. Since behavior cloning gives each sample an equal loss weight, that is, each sample is learned equally, it has a significant effect in the early stages of model training and can rapidly improve the performance of Android agents. However, as the model fits the expert data better and better, the loss of equal weight makes it impossible for the model to further fit and learn poorly learned samples, but for samples that have already been well-fitted and learned, multiple learning results in the inability to further improve model performance and may even cause overfitting. Conversely, insufficient training will cause underfitting problems. Both overfitting and underfitting will aggravate the compound error problem of behavior cloning.

[0005] Reinforcement learning is a common method to solve the problem of behavioral cloning and compounding. Reinforcement learning is divided into online reinforcement learning and offline reinforcement learning based on whether it needs to interact with the environment. The data efficiency of offline reinforcement learning is higher than that of online reinforcement learning. However, since the Android agent cannot interact with the environment, the target strategy and the behavior strategy cannot be too far apart, otherwise it is easy to cause an overestimation of the value of out-of-distribution actions, which limits the exploration and strategy optimization of offline reinforcement learning. Based on the above theory, although the lower limit of the strategy ultimately learned by offline reinforcement learning is difficult to guarantee, it has a greater impact on the behavior strategy. If the behavior strategy is better, the strategy obtained by offline reinforcement learning will generally be better.

[0006] In order to improve the model's ability to learn expert data, Google converted behavioral cloning from explicit to implicit. Some researchers also used the method of constructing a generative adversarial network for imitation learning, while improving data utilization. However, these methods have low feasibility in complex state and action spaces. Others divide different expert demonstration data into non-overlapping subsets, train a model for each subset of data, and use the sampling probability of the model for the next subset of data as the loss weight for training the next subset model. This method performs better than behavioral cloning in noisy data. The disadvantage is that a model must be trained for each subset, which consumes too many resources.

[0007] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new strategy learning method and system for Android agents. Summary of the invention

[0008] In order to overcome the defects of the prior art, the present invention proposes a strategy learning method and system for an Android agent, which combines the advantages of behavior cloning and offline reinforcement learning to design a strategy learning method that surpasses traditional behavior cloning. Compared with traditional behavior cloning methods, this method can learn better strategies, and is more suitable for the field of Android agents than implicit behavior cloning, generative adversarial imitation learning and other methods, and occupies fewer resources and has a simple training mode. Compared with offline reinforcement learning methods, this method does not require training of the discriminator network and is also effective when the rewards are sparse.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A strategy learning method for an Android agent that transcends behavior cloning, characterized in that it comprises the following steps:

[0011] 1) Using the large language model as the Android agent base model, and cloning the behavior of the Android agent base model according to the expert demonstration data;

[0012] 2) Construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model according to expert demonstration data to obtain the final Android agent.

[0013] Preferably, in step 1), the behavior cloning of the Android agent base model according to the expert demonstration data is specifically as follows: gradient descent training is performed on the Android agent base model according to the following formula to obtain the behavior cloned Android agent base model:

[0014]

[0015] In the formula, The loss function for gradient descent training; is the expected function; B is the model parameter of the Android agent base model; init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0016] Preferably, the step 2) includes: constructing a reward function according to a reinforcement learning method that explicitly controls the kl divergence and combining it with the RWR algorithm to infer the loss function L of the following formula, and using the loss function L to perform offline reinforcement learning training on the Android agent base model after behavior cloning,

[0017]

[0018] Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0019] Preferably, the step 2) further comprises: obtaining a loss function L of the following formula according to the advantage function and in combination with the AWR algorithm inference, and using the loss function L to perform offline reinforcement learning training on the Android agent base model after behavior cloning,

[0020]

[0021] Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0022] Preferably, in the step 1), after the Android agent base model is behaviorally cloned according to the expert demonstration data to obtain the Android agent base model after preliminary behavioral cloning, the loss function of the following formula is used to continue to perform loss training on the Android agent base model after preliminary behavioral cloning according to the expert demonstration data to obtain the Android agent base model after behavioral cloning,

[0023]

[0024] Where φ is the model parameter of the Android agent base model after the initial behavior cloning, and the initial parameters are is the expected function; B is the loss function for loss training; init Demonstration data for experts; x t is the input of the t-th step of the Android agent base model in the expert demonstration data; y t is the output of the t-th step of the Android agent base model in the expert demonstration data; t represents the t-th step of the task when the Android agent is doing the task; H represents the total number of steps in the task; λ represents the attention degree discount factor.

[0025] Preferably, the expert demonstration data is training data consisting of task-related XML text and action text extracted by annotators when performing Android app tasks.

[0026] Preferably, the large language model is a GLM-4 model or a Llama model.

[0027] In addition, the present invention also provides a strategy learning system for an Android agent that surpasses behavior cloning, which is characterized by comprising:

[0028] A behavior cloning module, which is used to use the large language model as an Android agent base model and perform behavior cloning on the Android agent base model according to expert demonstration data;

[0029] The offline reinforcement learning module is used to construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model according to expert demonstration data.

[0030] Moreover, the present invention also provides a strategy learning device for an Android agent that surpasses behavior cloning, which is characterized by comprising:

[0031] one or more processors;

[0032] A memory for storing one or more programs;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the strategy learning method for the Android agent to surpass behavioral cloning as described above.

[0034] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the strategy learning method for an Android agent to surpass behavioral cloning as described above are implemented.

[0035] Compared with the prior art, the strategy learning method and system of the Android agent of the present invention has one or more of the following beneficial technical effects:

[0036] 1. For Android agents, traditional behavior cloning methods often have difficulty learning the strategies for the next few steps and are affected by compound errors, which leads to the failure of the entire task. The present invention solves the problem that traditional behavior cloning methods are poor at fitting expert demonstrations.

[0037] 2. The present invention solves the problem that implicit behavior cloning cannot be applied to the action space when generating adversarial imitation learning, but depends on the strategy learning of the state space.

[0038] 3. The present invention solves the problem that the lower limit of the performance of offline reinforcement learning strategies is difficult to guarantee, and the problem that the discriminator network is difficult to train.

[0039] 4. The present invention solves the problem that most current behavior cloning and reinforcement learning methods cannot be applied to generative models.

[0040] 5. This invention solves the problem of sparse rewards in reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the strategy learning method of the Android agent of the present invention.

[0042] Figure 2 It is a schematic diagram of the composition of the strategy learning system of the Android agent of the present invention. DETAILED DESCRIPTION

[0043] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components described below or illustrated in the following figures. The present invention can have other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used here are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and its variations herein is intended to cover the items and their equivalents and additional items displayed below. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variations are widely used and cover direct installation and indirect installation, connection, support and connection. In addition, "connect" and "couple" are not limited to physical or mechanical connections or connections.

[0044] Furthermore, on the first aspect, in the disclosure of the present invention, the orientation or positional relationship indicated by terms such as "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second aspect, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0045] Before introducing the specific content of the present invention, some technical terms involved in the present invention are briefly introduced to help those skilled in the art to better understand the present invention.

[0046] 1. Intelligent agent: An intelligent agent is a system that can autonomously perceive the environment, make decisions and perform actions. It has basic characteristics such as autonomy, interactivity, responsiveness and adaptability.

[0047] 2. Behavioral cloning: It is to directly use the supervised learning method, regard the x in (x, y) in the expert data as the sample input, and y as the label, and learn the mapping relationship from x to y.

[0048] 3. Reinforcement learning: Reinforcement learning is a learning mechanism that learns how to map from state to behavior in order to maximize the cumulative rewards. The agent of online reinforcement learning needs to continuously experiment in the environment and continuously optimize the state-behavior correspondence through the feedback rewards given by the environment. Offline reinforcement learning is a reinforcement learning algorithm that does not require interaction with the environment.

[0049] 3. Compound error: The strategy learned through behavioral cloning cannot be completely optimized in the process of interacting with the environment. As long as there is a little deviation, it is possible that the next state encountered is not seen in the expert data. At this time, since it has not been trained in this state (or a relatively similar state), the strategy may randomly choose an action, which will cause the next state to further deviate from the data distribution encountered by the expert strategy. Ultimately, the strategy cannot achieve a good effect in the real environment, which is called the compound error problem of behavioral cloning.

[0050] 4. Underfitting: refers to the situation where the model performs poorly on the training set, validation set, and test set.

[0051] 5. Overfitting: This means that the model performs very well on the training set, but performs very poorly in the verification and testing stages, that is, the model's generalization ability is very poor.

[0052] 6. Target strategy: The optimal strategy that the agent hopes to learn.

[0053] 7. Behavioral strategy: The strategy actually adopted by the agent in the environment. In offline reinforcement learning, it is also called the in-data strategy.

[0054] 8. KL divergence: An indicator used to measure the similarity of two probability distributions. The smaller the KL divergence, the more similar the two distributions are.

[0055] 9. Discriminator network: determines what actions are good and what actions are not good in the current state, thereby helping the agent to update its strategy.

[0056] Figure 1 FIG. 4 is a flow chart showing the strategy learning method of the Android agent of the present invention. Figure 1 As shown, the strategy learning method of the Android agent beyond behavior cloning of the present invention comprises the following steps:

[0057] 1. Use the large language model as the Android agent base model and perform behavioral cloning on the Android agent base model according to expert demonstration data.

[0058] In the present invention, currently popular large language models, such as GLM4, Llama, etc., can be selected as the Android agent base model, and then the Android agent base model is behaviorally cloned using expert demonstration data, that is, supervised learning, to obtain the behaviorally cloned Android agent base model.

[0059] Among them, the expert demonstration data is training data composed of task-related XML text and action text extracted by the labeling personnel when doing Android app tasks. When acquiring the training data, conventional training data acquisition methods can be used. For example, first determine the task (for example, the determined task is "use Zhipu Qingyan app to make a video"), and then the labeling personnel operate Zhipu Qingyan app to make a video and record the labeling personnel's action data of touching the Android phone and the XML data of the Zhipu Qingyan app page during the process of using Zhipu Qingyan APP to make a video, thereby forming a training data in the training data set.

[0060] When performing behavior cloning, the Android agent base model is trained by gradient descent according to the following formula to obtain the Android agent base model after behavior cloning:

[0061]

[0062] In the formula, The loss function for gradient descent training, for example, cross entropy loss; is the expected function; B is the model parameter of the Android agent base model; itit is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0063] In addition, each task requires multiple operation steps. Experiments show that when the Android agent is doing each task, the closer it is to the next operation step, the lower the accuracy, so it is necessary to focus on learning the data of the next operation step. Therefore, in the present invention, after the Android agent base model is behaviorally cloned according to the expert demonstration data to obtain the Android agent base model after preliminary behavior cloning, the loss function of the following formula is used to continue to perform loss training on the Android agent base model after preliminary behavior cloning according to the expert demonstration data to obtain the Android agent base model after behavior cloning.

[0064]

[0065] Where φ is the model parameter of the Android agent base model after the initial behavior cloning, and the initial parameters are is the expected function; B is the loss function for loss training; init Demonstration data for experts; x t is the input of the t-th step of the Android agent base model in the expert demonstration data; y tis the output of the t-th step of the Android agent base model in the expert demonstration data; t represents the t-th step of the task when the Android agent is doing the task; H represents the total number of steps of the task; λ represents the attention degree discount factor, which can be set manually as needed.

[0066] Through the above loss training, it is possible to focus on learning the data of the subsequent operation steps when performing the task, thereby improving the capabilities of the Android agent base model after behavior cloning.

[0067] Second, construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model based on expert demonstration data to obtain the final Android agent. Specifically, it includes:

[0068] 1. The reward function constructed according to the reinforcement learning method with explicit control of kl divergence is combined with the RWR algorithm to infer the loss function L, and the loss function L is used to perform offline reinforcement learning training on the Android agent base model after behavior cloning.

[0069] It is known that the reinforcement learning objective of explicitly controlling KL divergence is:

[0070]

[0071] In the formula, To strengthen the learning data, the expert demonstration data B init ; is the reinforcement learning target strategy; r θ (x, y) is the reward model, θ is the reward model parameter; KL is the KL divergence; ∈ is π Initial The maximum allowed KL divergence value of and π.

[0072] According to the method introduced in the paper "Peng XB, Kumar A, Zhang G, et al. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning [J]. arXiv preprint arXiv:1910.00177, 2019", formula (1) can be transformed into the following formula:

[0073]

[0074] To simplify the representation, the reinforcement learning target strategy is expressed as follows:

[0075]

[0076] In the formula, Indicates that when the strategy is π Initial The unstandardized discounted state distribution in the case of .

[0077] The Lagrangian function is constructed as:

[0078]

[0079] where β and α correspond to Lagrange multipliers. Taking the derivative with respect to π, we get:

[0080]

[0081] because The optimal strategy for constrained reinforcement learning is:

[0082]

[0083] Let the partition function be:

[0084] Then the implicit reward function is:

[0085]

[0086] In an ideal situation, the distribution of the expert demonstration data is completely fitted, that is, π(y|x)=1, where Z(x) is a quantity related only to the state, and the invention is unified to 0. Then, in the expert demonstration data and the π trained by the expert data, Initial Under the model conditions, the implicit reward function can be written as follows:

[0087]

[0088] According to the RWR algorithm, the loss function L is as follows:

[0089]

[0090] Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0091] The above loss function L is used to perform offline reinforcement learning training on the behavior-cloned Android agent base model.

[0092] 2. According to the advantage function and combined with the AWR algorithm reasoning, the loss function L of the following formula is obtained, and the loss function L is used to perform offline reinforcement learning training on the Android agent base model after behavior cloning.

[0093] Assume that all expert demonstration data are from π Initial If the expert demonstration data obtained by model sampling is correct and unique, the value reward of each step can be set to 1, then according to the calculation formula of the advantage function, we can get:

[0094]

[0095] In the formula, The strategy is π Initial The advantage function of Indicates that when the strategy is π Initial The action value of taking action y in state x.

[0096] According to the AWR algorithm, the loss function L is as follows:

[0097]

[0098] Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

[0099] The above loss function L is used to perform offline reinforcement learning training on the behavior-cloned Android agent base model.

[0100] Through offline reinforcement learning RWR and AWR training, the final Android agent can be obtained.

[0101] The present invention combines the advantages of behavior cloning and offline reinforcement learning to design a strategy learning method that surpasses traditional behavior cloning. Compared with traditional behavior cloning methods, this method can learn better strategies, and is more suitable for the field of Android intelligent agents than implicit behavior cloning, generative adversarial imitation learning and other methods. It also occupies fewer resources and has a simple training mode. Compared with offline reinforcement learning methods, this method does not require training of the discriminator network and is also effective when the rewards are sparse.

[0102] Figure 2 The schematic diagram of the strategy learning system of the Android agent of the present invention is shown. Figure 2As shown, the strategy learning system of the Android agent beyond behavior cloning of the present invention includes:

[0103] 1. Behavior cloning module.

[0104] The behavior cloning module is used to use the large language model as the Android agent base model and perform behavior cloning on the Android agent base model according to the expert demonstration data;

[0105] 2. Offline reinforcement learning module.

[0106] The offline reinforcement learning module is used to construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model according to expert demonstration data.

[0107] In addition, the present invention also provides a strategy learning device for an Android agent that surpasses behavioral cloning, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the strategy learning method for an Android agent that surpasses behavioral cloning as described above.

[0108] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the strategy learning method for an Android agent to surpass behavioral cloning as described above.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A strategy learning method for an Android agent that goes beyond behavior cloning, characterized in that: The following steps are involved: 1) Using the large language model as the Android agent base model, and cloning the behavior of the Android agent base model according to the expert demonstration data; 2) Construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model according to expert demonstration data to obtain the final Android agent.

2. The strategy learning method for Android agent beyond behavior cloning according to claim 1 is characterized in that: In the step 1), the behavior cloning of the Android agent base model according to the expert demonstration data is specifically as follows: gradient descent training is performed on the Android agent base model according to the following formula to obtain the behavior cloned Android agent base model: In the formula, The loss function for gradient descent training; is the expected function; B is the model parameter of the Android agent base model; init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

3. The strategy learning method for Android agent beyond behavior cloning according to claim 2 is characterized in that: The step 2) includes: constructing a reward function according to a reinforcement learning method that explicitly controls the kl divergence and combining it with the RWR algorithm to infer the loss function L of the following formula, and using the loss function L to perform offline reinforcement learning training on the Android agent base model after behavior cloning, Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

4. The strategy learning method for Android agent beyond behavior cloning according to claim 3 is characterized in that: The step 2) also includes: according to the advantage function and in combination with the AWR algorithm, a loss function L of the following formula is obtained, and the loss function L is used to perform offline reinforcement learning training on the Android agent base model after behavior cloning, Where φ is the model parameter of the Android agent base model after behavior cloning, and the initial parameter is is the expected function; exp is the natural exponential function; is the loss function for offline reinforcement learning training; B init is the expert demonstration data; x is the input of the Android agent base model in the expert demonstration data; y is the output of the Android agent base model in the expert demonstration data.

5. The strategy learning method for an Android agent that transcends behavior cloning according to any one of claims 1 to 4, characterized in that: In the step 1), after the Android agent base model is behaviorally cloned according to the expert demonstration data to obtain the Android agent base model after preliminary behavioral cloning, the loss function of the following formula is used to continue to perform loss training on the Android agent base model after preliminary behavioral cloning according to the expert demonstration data to obtain the Android agent base model after behavioral cloning, Where φ is the model parameter of the Android agent base model after the initial behavior cloning, and the initial parameters are is the expected function; B is the loss function for loss training; init Demonstration data for experts; x t is the input of the t-th step of the Android agent base model in the expert demonstration data; y t is the output of the t-th step of the Android agent base model in the expert demonstration data; t represents the t-th step of the task when the Android agent is doing the task; H represents the total number of steps in the task; λ represents the attention degree discount factor.

6. The strategy learning method for Android agent beyond behavior cloning according to claim 5 is characterized in that: The expert demonstration data is training data consisting of task-related XML text and action text extracted by labelers when performing Android app tasks.

7. The strategy learning method for Android agent beyond behavior cloning according to claim 6 is characterized in that: The large language model is a GLM-4 model or a Llama model.

8. A strategy learning system for an Android agent that goes beyond behavior cloning, characterized in that: include: A behavior cloning module, which is used to use the large language model as an Android agent base model and perform behavior cloning on the Android agent base model according to expert demonstration data; The offline reinforcement learning module is used to construct a reward function based on the behavior-cloned Android agent base model, and perform offline reinforcement learning RWR and AWR training on the behavior-cloned Android agent base model according to expert demonstration data.

9. A strategy learning device for an Android agent that goes beyond behavior cloning, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the strategy learning method for an Android agent that surpasses behavioral cloning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the strategy learning method of an Android agent that transcends behavioral cloning are implemented as described in any one of claims 1-7.

Images