Instruction-driven end-side agent control method and system

Abstract

The invention discloses an instruction-driven end-side agent control method and system. A screen content understanding model is trained and tested; a screen content understanding data set required for making screen content understanding is collected; performing increment fine tuning training of a screen content understanding model by using the screen content understanding data set; training and testing an instruction translation model; collecting an instruction translation data set required for completing instruction translation; performing increment fine tuning training of an instruction translation model by using the instruction translation data set; and combining the intelligent agent and executing a user request: combining the screen content understanding model and the instruction translation model into the intelligent agent, and calling an Android ADB toolbox to execute the user request. According to the method, the phenomenon of low efficiency of an exploring type intelligent agent is avoided, and the requirement for high computing power of the multi-mode LLM is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agent research for mobile terminals, and in particular discloses a command-driven terminal-side intelligent agent control method and system. Background Art

[0002] An AI agent (or just an agent) is generally considered to be an agent that can autonomously achieve set goals through its ability to act. "Agent" is inseparable from "intelligence"; it has some human-like intelligence capabilities and behaviors, such as learning, reasoning, decision-making, and execution capabilities.

[0003] The development of AI end-side technology will present three stages, and it is currently in the stage of evolution from AI functionalization to AI nativeization. The first stage is the AI ​​APP stage. The feature is that various large models are made into APP products and installed on end-side devices, such as Wenxin Yiyan APP, Doubao APP, etc.; the second stage is the AI ​​functionalization stage. The feature is that large models are initially integrated into end-side applications to realize calls mainly based on pre-installed applications, which can realize functions such as photo editing, call translation, document summarization, etc.; the third stage is the AI ​​nativeization stage. Intelligent agents based on voice interaction will become the interaction hub of AI end-side devices in the next stage, and diversified and natural interactive experiences will become the mainstream, and AI intelligent agents will become the entrance to link the digital ecosystem.

[0004] Agents are considered to be an important carrier of future LLM (Large Language Model). Nowadays, various agent technologies and frameworks are booming, and a variety of agent basic frameworks have emerged, such as AutoGPT and MetaGPT. At present, it is rare to see mature end-side agent frameworks that can flexibly drive apps on end-side devices to accurately execute various instructions.

[0005] At the end of 2023, Tencent and Alibaba have respectively developed the prototypes of the end-side intelligent agents MobileAgent and AppAgent driven by the multimodal LLM model. However, due to many defects, they cannot be put into actual production use. These defects are manifested in the following two aspects: First, both solutions have very high performance requirements for the multimodal LLM model (the GPT-4V model needs to be frequently called by default), which leads to the computing power, bandwidth, and latency of the intelligent agent during operation exceeding expectations, and the user experience is very poor. Second, both solutions complete tasks through automatic exploration of large language models, resulting in a low success rate for the intelligent agent to complete tasks. In early 2024, Apple published an intelligent agent-related paper called REALM, which aims to enhance the intelligence of its own voice assistant Siri (Speech Interpretation & Recognition Interface) through a pre-trained 2B (To Business) model to understand the demonstrative pronouns in user instructions. However, this method is difficult to extend to other products for two reasons: first, REALM only understands the demonstrative pronouns that often appear when using Siri, and lacks understanding of the screen content; second, REALM must rely on a voice assistant to complete the task, and a command can only complete a simple action, and cannot complete a coherent complex command. Another more inspiring work is the Octopus-v2 model launched by Stanford researchers in early 2024. The model is trained on a large number of Android functions to enable it to have function call functions, so as to achieve the LLM model's accurate interpretation of user commands to agent actions. However, it has the following defects: first, it does not mention how to achieve behavior control after predicting the function; second, the paper pre-trains a 2B model, which cannot understand the user's screen content; third, the command comprehension ability is poor, and it is impossible to understand user commands or decompose user commands during the execution of tasks by the agent.

[0006] Therefore, the above-mentioned defects of existing terminal-side intelligent agents are technical problems that need to be solved urgently. Summary of the invention

[0007] The present invention provides a command-driven terminal-side intelligent agent control method and system, aiming to solve at least one defect existing in the above-mentioned prior art.

[0008] One aspect of the present invention relates to a command-driven terminal-side intelligent agent control method, comprising the following steps:

[0009] Train and test the screen content understanding model: collect the screen content understanding dataset required for screen content understanding; use the screen content understanding dataset to perform incremental fine-tuning training on the screen content understanding model;

[0010] Train and test the instruction translation model: collect the instruction translation dataset required for instruction translation; use the instruction translation dataset for incremental fine-tuning training of the instruction translation model;

[0011] Combine the agents and execute the user request: combine the screen content understanding model and the instruction translation model into an agent, and call the Android ADB toolbox to execute the user request.

[0012] Furthermore, in the step of training and testing the screen content understanding model, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module. The parameters of the screen content understanding model are calculated by the following formula:

[0013] W ScreenOCR =W LLM +ΔW ScreenOCR

[0014] where W ScreenOCR is the parameter of the screen content understanding model, W LLM is the pre-trained parameter matrix in the large language model, W LLM ∈R n×m , R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW ScreenOCR is the incremental parameter matrix of the screen content understanding model, and ΔW screenOCR can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW ScreenOCR =AB, (A∈R n×r , B∈R r×m , r<<min(m,n)), A and B are two low-rank adaptation weight matrices respectively, R r ×m and R n×r refer to real number parameters and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m,n).

[0015] Furthermore, in the step of training and testing the screen content understanding model, during the training process of the screen content understanding model, all parameters in the large language model are frozen, and only the parameters of the screen content understanding module are fine-tuned and updated until the loss function curve of the screen content understanding model tends to converge.

[0016] Furthermore, in the step of training and testing the instruction translation model, the instruction translation model includes a large language model and an incrementally trained instruction translation module. The parameters of the instruction translation model are calculated by the following formula:

[0017] W FunctionCalling =W LLM +ΔW FunctionCalling

[0018] Among them, W FunctionCalling is the parameter of the instruction translation model, and W LLM is the pre-trained parameter matrix in the large language model. W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW FunctionCalling is the instruction translation model incremental parameter matrix; ΔW FunctionCalling can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW FunctionCalling =AB, (A∈R n×r , B∈R r×m , r << min(m, n)), A and B are two low-rank adaptation weight matrices respectively, and R r×m and R n×r refer to real number parameters and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m, n).

[0019] Furthermore, in the steps of training and testing the instruction translation model, during the training process of the instruction translation model, all parameters in the large language model are frozen, and only the parameters of the instruction translation module are fine-tuned and updated until the loss function curve of the instruction translation model tends to converge.

[0020] Another aspect of the present invention relates to an instruction-driven end-side intelligent agent control system, including:

[0021] A training and testing screen content understanding model unit, which is used to collect the screen content understanding data set required for making screen content understanding; perform incremental fine-tuning training of the screen content understanding model using the screen content understanding data set;

[0022] A training and testing instruction translation model unit, which is used to collect the instruction translation data set required for completing instruction translation; perform incremental fine-tuning training of the instruction translation model using the instruction translation data set;

[0023] A combined intelligent agent and execute user request unit, which is used to combine the screen content understanding model and the instruction translation model into an intelligent agent, and call the Android ADB toolbox to execute user requests.

[0024] Furthermore, in the training and testing screen content understanding model unit, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module. The parameters of the screen content understanding model are calculated by the following formula:

[0025] W ScreenOCR =W LLM +ΔW ScreenOCR

[0026] Among them, W ScreenOCR is the parameter of the screen content understanding model, and W LLM is the pre-training parameter matrix in the large language model. W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW ScreenOCR is the screen content understanding model incremental parameter matrix. ΔW screenOCR can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW ScreenOCR = AB, (A∈R n×r , B∈R r×m , r << min(m,n)), A and B are two low-rank adaptation weight matrices respectively, and R r ×m and R n×r refer to real number parameters and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m,n).

[0027] Furthermore, during the training and testing of the screen content understanding model unit, during the training process of the screen content understanding model, all the parameters in the large language model are frozen, and only the parameters of the screen content understanding module are fine-tuned and updated until the loss function curve of the screen content understanding model tends to converge.

[0028] Furthermore, in the training and testing of the instruction translation model unit, the instruction translation model includes a large language model and an incrementally trained instruction translation module. The parameters of the instruction translation model are calculated by the following formula:

[0029] W FunctionCalling = W LLM + ΔW FunctionCalling

[0030] Among them, W FunctionCalling is the parameter of the instruction translation model, W LLM is the pre-training parameter matrix in the large language model, W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW FunctionCalling is the instruction translation model incremental parameter matrix; ΔW FunctionCalling can be converted into the product of two low-rank adaptation weight matrices W ScreenOCR and B, that is, ΔW FunctionCalling = AB, (A∈R n×r , B∈R r×m , r << min(m,n)), W ScreenOCR and B are two low-rank adaptation weight matrices respectively, and Rr×m and R n×r refers to a real number parameter and the dimensions of the current layer are r×m and n×r respectively; r represents the manually set dimension and r is much smaller than min(m,n).

[0031] Furthermore, in the instruction translation model unit, during the instruction translation model training process, all parameters in the large language model are frozen, and only the parameters of the instruction translation module are fine-tuned and updated until the instruction translation model loss function curve tends to converge.

[0032] The beneficial effects achieved by the present invention are:

[0033] The present invention provides a command-driven end-side intelligent agent control method and system, by training and testing the screen content understanding model: collecting the screen content understanding data set required for making screen content understanding; using the screen content understanding data set to perform incremental fine-tuning training of the screen content understanding model; training and testing the instruction translation model: collecting the instruction translation data set required for completing instruction translation; using the instruction translation data set to perform incremental fine-tuning training of the instruction translation model; combining intelligent agents and executing user requests: combining the screen content understanding model and the instruction translation model into an intelligent agent, calling the Android ADB toolbox to execute user requests. The command-driven end-side intelligent agent control method and system provided by the present invention, first, improves the control ability of the intelligent agent by accurately predicting functions and parameters, avoiding the phenomenon of low efficiency of exploratory intelligent agents; second, by adding a small number of parameters, the LLM is equipped with screen content understanding ability and instruction translation ability, reducing the high computing power requirements of the multimodal LLM; third, the screen content understanding model and the instruction translation model can be flexibly combined to support further expansion of other functions of the LLM. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of an embodiment of a command-driven terminal-side intelligent agent control method of the present invention. DETAILED DESCRIPTION

[0035] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] like Figure 1 As shown, the first embodiment of the present invention proposes a command-driven terminal-side intelligent agent control method, comprising the following steps:

[0037] Step S100, training and testing the screen content understanding model: collecting a screen content understanding dataset required for screen content understanding; using the screen content understanding dataset to perform incremental fine-tuning training of the screen content understanding model.

[0038] Collect the screen content understanding dataset required for screen content understanding. Collect mobile screenshots, use a general character recognition algorithm to extract and recognize the text in the screenshots, and manually construct the corresponding questions and correct answers. Extract 20% of the data as a test set and the rest as a training set.

[0039] Training the screen content understanding model: The purpose of training is to enable the intelligent agent to accurately understand the screen content. Specifically, the screen content understanding model (Screen OCR) is incrementally fine-tuned using the above-constructed dataset. Specifically, the screen content understanding model consists of an LLM (Large Language Model) and an incrementally trained screen content understanding module.

[0040] Test the agent’s ability to understand screen content: Use the screen content understanding test set to test whether the effect of the LLM+Screen OCR module can be accurately predicted.

[0041] Step S200, training and testing the instruction translation model: collecting the instruction translation data set required to complete the instruction translation; using the instruction translation data set to perform incremental fine-tuning training of the instruction translation model.

[0042] Collect the instruction translation dataset required to complete the instruction translation, and use it to train the model so that the test instruction translation model has instruction prediction capabilities. The instruction translation dataset contains user requests, system prompts, and corresponding indicative function descriptions. Among all the collected instruction translation datasets, 20% of the datasets are extracted as test sets, and the rest are used as training sets.

[0043] Training the instruction translation model: The purpose of training is to enable the agent to have the ability to predict indicative functions. Specifically, the instruction translation model (Function Calling) is incrementally fine-tuned using the above-constructed dataset. Specifically, the screen content understanding model consists of an LLM (Large Language Model) and an incrementally trained instruction translation module.

[0044] Test the agent's instruction translation capability: Use the instruction translation test set to test whether the effect of the LLM+Function Calling model can accurately predict the defined executable functions.

[0045] Step S300, combine the intelligent agent and execute the user request: combine the screen content understanding model and the instruction translation model into an intelligent agent, and call the Android ADB toolbox to execute the user request.

[0046] Composition of the intelligent agent: The intelligent agent consists of a screen content understanding model and a command translation model. Specifically, it consists of LLM and two additional incrementally trained modules. The LLM+Screen OCR module realizes the screen content understanding function, and the LLM+Function Calling module realizes the command translation function. The Screen OCR module and the Function Calling module are independent of each other and do not interfere with each other.

[0047] Understand user requests and execute tasks: The agent first calls LLM to understand user requests and converts them into "actions" and "objects". The prompt words can be used to perform specific operations. Then, the instruction translation model is called according to the "action" to generate the corresponding execution function. The screen content understanding model is called according to the "object" to determine whether the "object" is related to the screen content. If the content is related, the "object" related content is extracted. If it is not related, a null value is returned. Finally, the execution function and its associated parameters are combined to generate a shell command, and the Android ADB (Android Debug Bridge) toolbox is called to execute it. Taking the "call human resources" instruction as an example, LLM first understands the user instruction: the "action" is "call" and the "object" is "human resources". Then, the instruction translation model calls the Make_a_phone_call() function according to the "call" action. The screen content understanding model determines that the number is "0755-86975800" according to "human resources". Finally, the number is entered into the Make_a_phone_call() function and the shell command is used to execute to complete the instruction task.

[0048] Furthermore, in the command-driven terminal-side intelligent agent control method provided in this embodiment, in step S100, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module, and the parameters of the screen content understanding model are calculated by the following formula:

[0049] W ScreenOCR =W LLM +ΔW ScreenOCR (1)

[0050] In formula (1), W ScreenOCR is the parameter of the episode content understanding model, W LLM is the pre-trained parameter matrix in the large language model, W LLM ∈R n×m , R n×m is a real number parameter set with a dimension of n×m; R is a real number data representation, with n and m being the dimensions; ΔW ScreenOCR is the incremental parameter matrix of the screen content understanding model, ΔW screenOCR It can be converted into the product of two low-rank adaptive weight matrices A and B, namely ΔWScreenOCR = AB, (A ∈ R n×r , B ∈ R r×m , r << min(m, n)), A and B are two low-rank adaptive weight matrices respectively, R r×m and R n×r refer to real number parameters and the dimensions of the current layer are r × m and n × r respectively; r represents the dimension set artificially and r is much smaller than min(m, n).

[0051] The Screen OCR module fine-tunes and trains some fully connected layers of the LLM. All parameters of the LLM are frozen, and all parameters of the Screen OCR module are trained. The parameter scale of the Screen OCR module is much smaller than that of the LLM. This training idea is fundamentally to use a small amount to gain a large amount, adding a small number of parameters to enable the large model to have the ability to solve specific problems.

[0052] During the training process of the screen content understanding model, all parameters in the large language model are frozen, and only the parameters of the screen content understanding module are fine-tuned and updated until the loss function curve of the screen content understanding model converges.

[0053] The characteristic of the neural network convergence is that the loss function value gradually decreases as the number of training times increases and finally tends to be stable. When the maximum number of training times is reached, the training stops. If the network does not converge, the loss function value fluctuates violently up and down as the number of training times increases, and the network training fails.

[0054] Preferably, in the instruction-driven edge agent control method provided in this embodiment, in step S200, the instruction translation model includes a large language model and an incrementally trained instruction translation module, and the parameters of the instruction translation model are calculated by the following formula:

[0055] W FunctionCalling = W LLM + ΔW FunctionCalling (2)

[0056] In formula (2), W FunctionCalling is the parameter of the instruction translation model, W LLM is the pre-trained parameter matrix in the large language model, W LLM ∈ R n×m , R n×m is the set of real number parameters, the dimension is n × m; R is the real number data representation, n and m are dimensions; ΔW FunctionCalling is the instruction translation model incremental parameter matrix; ΔW FunctionCalling can be converted into the product of two low-rank adaptive weight matrices A and B, that is, ΔW FunctionCalling = AB, (A ∈ R n×r , B ∈ R r×m, r << min(m, n)), where A and B are two low-rank adaptation weight matrices, and R r×m and R n×r refer to real number parameters, and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m, n).

[0057] During the training process of the instruction translation model, all the parameters in the large language model are frozen, and only the parameters of the instruction translation module are fine-tuned and updated until the loss function curve of the instruction translation model tends to converge.

[0058] This embodiment also provides an instruction-driven end-side intelligent agent control system, including a unit for training and testing the screen content understanding model, a unit for training and testing the instruction translation model, and a unit for combining intelligent agents and executing user requests. Among them, the unit for training and testing the screen content understanding model is used to collect the screen content understanding data set required for making screen content understanding; perform incremental fine-tuning training of the screen content understanding model using the screen content understanding data set; the unit for training and testing the instruction translation model is used to collect the instruction translation data set required for completing instruction translation; perform incremental fine-tuning training of the instruction translation model using the instruction translation data set; the unit for combining intelligent agents and executing user requests is used to combine the screen content understanding model and the instruction translation model into an intelligent agent, and call the Android ADB toolbox to execute user requests.

[0059] The unit for training and testing the screen content understanding model collects the screen content understanding data set required for making screen content understanding. Collect mobile phone screen screenshots, extract and recognize the text in the screen screenshots using a general character recognition algorithm, and artificially construct corresponding questions and correct answers. Extract 20% of the data as the test set, and the rest as the training set.

[0060] The unit for training and testing the screen content understanding model trains the screen content understanding model: the purpose of training is to enable the intelligent agent to have the ability to accurately understand the screen content. Specifically, incremental fine-tuning training of the screen content understanding model (Screen OCR) is performed using the above-constructed data set. Specifically, the screen content understanding model consists of an LLM (Large Language Model) and an incrementally trained screen content understanding module.

[0061] The unit for training and testing the screen content understanding model tests the screen content understanding ability of the intelligent agent: uses the screen content understanding test set to test whether the effect of the LLM + Screen OCR module can accurately predict.

[0062] The training and testing instruction translation model unit collects instruction translation data sets required for completing instruction translation, which are used to train the model so that the test instruction translation model has instruction prediction capabilities. The instruction translation data set contains user requests, system prompt words, and corresponding indicative function descriptions. Among all the collected instruction translation data sets, 20% of the data sets are extracted as test sets, and the rest are used as training sets.

[0063] Training and testing the instruction translation model unit Training the instruction translation model: The purpose of training is to enable the agent to have the ability to predict indicative functions, specifically using the above-constructed data set to perform incremental fine-tuning training of the instruction translation model (FunctionCalling). Specifically, the screen content understanding model consists of an LLM (Large Language Model) and an incrementally trained instruction translation module.

[0064] Train and test the instruction translation model unit to test the agent's instruction translation ability: Use the instruction translation test set to test whether the effect of the LLM+Function Calling model can accurately predict the defined executable functions.

[0065] Composition of the intelligent agent: The intelligent agent consists of a screen content understanding model and a command translation model. Specifically, it consists of LLM and two additional incrementally trained modules. The LLM+Screen OCR module realizes the screen content understanding function, and the LLM+Function Calling module realizes the command translation function. The Screen OCR module and the Function Calling module are independent of each other and do not interfere with each other.

[0066] The combined agent and execute user request unit understands the user request and executes tasks: First, the agent calls the LLM to understand the user request, converts the user request into "actions" and "objects", and can specifically use prompt words for operations; then, according to the "actions", it calls the instruction translation model to generate the corresponding execution function, and according to the "objects", it calls the screen content understanding model to determine whether the "objects" are related to the screen content. If the content is related, it extracts the content associated with the "objects", and if not, it returns a null value; finally, it combines the execution function and its associated parameters to generate a shell command and calls the Android ADB (Android Debug Bridge) toolbox to execute. Taking the instruction "Call the human resources department" as an example, the LLM first understands the user instruction: the "action" is "call", and the "object" is "human resources". Then, the instruction translation model calls the Make_a_phone_call() function according to the "call" action, and the screen content understanding model determines that the number is "0755-86975800" according to "human resources". Finally, after entering the number into the Make_a_phone_call() function, it can be executed using a shell command to complete the instruction task.

[0067] Furthermore, in the instruction-driven edge agent control system provided in this embodiment, in the unit for training and testing the screen content understanding model, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module. The parameters of the screen content understanding model are calculated by the following formula:

[0068] W ScreenOCR =W LLM +ΔW ScreenOCR (3)

[0069] In formula (3), W ScreenOCR is the parameter of the screen content understanding model, W LLM is the pre-trained parameter matrix in the large language model, W LLM ∈R n×m , R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW ScreenOCR is the incremental parameter matrix of the screen content understanding model, and ΔW screenOCR can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW ScreenOCR =AB, (A∈R n×r , B∈R r×m , r<<min(m,n)), A and B are two low-rank adaptation weight matrices respectively, R r×m and R n×rIt refers to real - valued parameters, and the dimensions of the current layer are \(r\times m\) and \(n\times r\) respectively; \(r\) represents the dimension set manually, and \(r\) is much smaller than \(\min(m,n)\).

[0070] During the training and testing of the screen content understanding model unit, all parameters in the large - language model are frozen, and only the parameters of the screen content understanding module are fine - tuned and updated until the loss function curve of the screen content understanding model tends to converge.

[0071] Preferably, in the instruction - driven edge - side intelligent agent control system provided in this embodiment, in the training and testing of the instruction translation model unit, the instruction translation model includes a large - language model and an incrementally - trained instruction translation module. The parameters of the instruction translation model are calculated by the following formula:

[0072] W FunctionCalling =W LLM +ΔW FunctionCalling (4)

[0073] In formula (4), \(W\) FunctionCalling is the parameter of the instruction translation model, \(W\) LLM is the pre - trained parameter matrix in the large - language model, \(W\) LLM ∈R n×m ,R n×m is the set of real - valued parameters, with a dimension of \(n\times m\); \(R\) is the real - number data representation, \(n\) and \(m\) are dimensions; \(\Delta W\) FunctionCalling is the incremental parameter matrix of the instruction translation model; \(\Delta W\) FunctionCalling can be converted into the product of two low - rank adaptation weight matrices \(A\) and \(B\), that is, \(\Delta W\) FunctionCalling =AB, (\(A\in R\) n×r , \(B\in R\) r×m , \(r\ll\min(m,n)\)), \(A\) and \(B\) are two low - rank adaptation weight matrices respectively, \(R\) r×m and \(R\) n×r It refers to real - valued parameters, and the dimensions of the current layer are \(r\times m\) and \(n\times r\) respectively; \(r\) represents the dimension set manually, and \(r\) is much smaller than \(\min(m,n)\).

[0074] During the training and testing of the instruction translation model unit, all parameters in the large - language model are frozen, and only the parameters of the instruction translation module are fine - tuned and updated until the loss function curve of the instruction translation model tends to converge.

[0075] Compared with the prior art, the command-driven end-side intelligent agent control method and system provided in this embodiment trains and tests the screen content understanding model: collects the screen content understanding data set required for screen content understanding; uses the screen content understanding data set to perform incremental fine-tuning training of the screen content understanding model; trains and tests the instruction translation model: collects the instruction translation data set required for completing instruction translation; uses the instruction translation data set to perform incremental fine-tuning training of the instruction translation model; combines intelligent agents and executes user requests: combines the screen content understanding model and the instruction translation model into an intelligent agent, and calls the Android ADB toolbox to execute user requests. The command-driven end-side intelligent agent control method and system provided in this embodiment, first, improves the control ability of the intelligent agent by accurately predicting functions and parameters, avoiding the phenomenon of low efficiency of exploratory intelligent agents; second, by adding a small number of parameters, the LLM has the ability to understand screen content and the ability to translate instructions, reducing the high computing power requirements of the multimodal LLM; third, the screen content understanding model and the instruction translation model can be flexibly combined to support further expansion of other functions of the LLM.

[0076] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A command-driven terminal-side intelligent agent control method, characterized in that: The following steps are involved: Training and testing the screen content understanding model: collecting a screen content understanding dataset required for making screen content understanding; using the screen content understanding dataset to perform incremental fine-tuning training on the screen content understanding model; Training and testing the instruction translation model: collecting the instruction translation data set required to complete the instruction translation; using the instruction translation data set to perform incremental fine-tuning training of the instruction translation model; Combine the intelligent agent and execute the user request: combine the screen content understanding model and the instruction translation model into an intelligent agent, and call the Android ADB toolbox to execute the user request.

2. The command-driven terminal-side intelligent agent control method according to claim 1, characterized in that: In the step of training and testing the screen content understanding model, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module, and the parameters of the screen content understanding model are calculated by the following formula: IN ScreenOCR =In LLM +ΔW ScreenOCR Among them, W ScreenOCR is the parameter of the screen content understanding model, and W LLM is the pre-trained parameter matrix in the large language model. W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW ScreenOCR is the incremental parameter matrix of the screen content understanding model. ΔW screenOCR can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW ScreenOCR =AB, (A∈R n×r , B∈R r×m , r << min(m,n)). A and B are two low-rank adaptation weight matrices respectively, and R r×m and R n×r refer to real number parameters, and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m,n).

3. The command-driven terminal-side intelligent agent control method according to claim 2, characterized in that: In the step of training and testing the screen content understanding model, during the training process of the screen content understanding model, all parameters in the large language model are frozen, and only the parameters of the screen content understanding module are fine-tuned and updated until the loss function curve of the screen content understanding model tends to converge.

4. The command-driven terminal-side intelligent agent control method according to claim 1, characterized in that: In the step of training and testing the instruction translation model, the instruction translation model includes a large language model and an incrementally trained instruction translation module, and the parameters of the instruction translation model are calculated by the following formula: IN FunctionCalling =In LLM +ΔW FunctionCalling Among them, W FunctionCalling is the parameter of the instruction translation model, and W LLM is the pre-training parameter matrix in the large language model. W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW FunctionCalling is the incremental parameter matrix of the instruction translation model; ΔW FunctionCalling can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW FunctionCalling =AB, (A∈R n×r , B∈R r×m , r << min(m, n)), A and B are two low-rank adaptation weight matrices respectively, and R r×m and R n×r refer to real number parameters and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m, n).

5. The command-driven terminal-side intelligent agent control method according to claim 4, characterized in that: In the step of training and testing the instruction translation model, during the instruction translation model training process, all parameters in the large language model are frozen, and only the parameters of the instruction translation module are fine-tuned and updated until the instruction translation model loss function curve tends to converge.

6. A command-driven terminal-side intelligent agent control system, characterized in that: include: Training and testing the screen content understanding model unit to collect the screen content understanding data set required for making screen content understanding; Using the screen content understanding dataset to perform incremental fine-tuning training of a screen content understanding model; Training and testing the instruction translation model unit to collect the instruction translation data set required to complete the instruction translation; using the instruction translation data set to perform incremental fine-tuning training of the instruction translation model; The unit for combining intelligent agents and executing user requests is used to combine the screen content understanding model and the instruction translation model into an intelligent agent, and call the Android ADB toolbox to execute user requests.

7. The command-driven terminal-side intelligent agent control system according to claim 6, characterized in that: In the training and testing screen content understanding model unit, the screen content understanding model includes a large language model and an incrementally trained screen content understanding module, and the parameters of the screen content understanding model are calculated by the following formula: IN ScreenOCR =In LLM +ΔW ScreenOCR Among them, W ScreenOCR is the parameter of the episode content understanding model, W LLM is the pre-trained parameter matrix in the large language model, W LLM ∈R n×m , R n×m is a real number parameter set with a dimension of n×m; R is a real number data representation, with n and m being the dimensions; ΔW ScreenOCR is the incremental parameter matrix of the screen content understanding model, ΔW screenOCR It can be converted into the product of two low-rank adaptive weight matrices A and B, namely ΔW ScreenOCR =AB, (A∈R n×r ,B∈R r×m ,r< <min(m,n)),W ScreenOCR and B are two low-rank adaptation weight matrices, R r×m and R n×r refers to a real number parameter and the dimensions of the current layer are r×m and n×r respectively; r represents the manually set dimension and r is much smaller than min(m,n).

8. The command-driven terminal-side intelligent agent control system according to claim 7, characterized in that: The training and testing screen content understanding model unit is used to freeze all parameters in the large language model during the screen content understanding model training process, and only fine-tune and update the parameters of the screen content understanding module until the screen content understanding model loss function curve tends to converge.

9. The command-driven terminal-side intelligent agent control system according to claim 8, characterized in that: In the training and testing instruction translation model unit, the instruction translation model includes a large language model and an incrementally trained instruction translation module, and the parameters of the instruction translation model are calculated by the following formula: IN FunctionCalling =In LLM +ΔW FunctionCalling Among them, W FunctionCalling is the parameter of the instruction translation model, and W LLM is the pre-trained parameter matrix in the large language model. W LLM ∈R n×m , where R n×m is the set of real number parameters, with a dimension of n×m; R is the real number data representation, n and m are dimensions; ΔW FunctionCalling is the incremental parameter matrix of the instruction translation model; ΔW FunctionCalling can be converted into the product of two low-rank adaptation weight matrices A and B, that is, ΔW FunctionCalling =AB, (A∈R n×r , B∈R r×m , r << min(m, n)), A and B are two low-rank adaptation weight matrices respectively, and R r×m and R n×r refer to real number parameters and the dimensions of the current layer are r×m and n×r respectively; r represents the artificially set dimension, and r is much smaller than min(m, n).

10. The command-driven terminal-side intelligent agent control system according to claim 9, characterized in that: The training and testing instruction translation model unit is used to freeze all parameters in the large language model during the instruction translation model training process, and only fine-tune and update the parameters of the instruction translation module until the instruction translation model loss function curve tends to converge.

Images