A method and device for controlling intelligent body motion guided by a large language model
Through the large language model, the task is converted into the action sequence of the agent and input it into the agent control model, which solves the problem of poor control accuracy of the agent and realizes the precise control of the agent cursor.
Patent Information
- Application Number
- CN202410274114.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-03-11
AI Technical Summary
The existing agent control methods are insufficient in accuracy, which makes it difficult for the agent to match the expert's status distribution when performing tasks, and cannot achieve efficient computer control.
The large language model is used to convert the target task into the target action sequence of the agent, and these action sequences are input into the agent control model, so that the agent can be precisely controlled by predicting the movement and clicking actions of the cursor.
By disassembling complex tasks into simple action sequences, the task complexity is reduced, the control accuracy of the agent is improved, and the cursor can accurately execute user instructions.
Smart Images

Figure CN118092764B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the field of artificial intelligence technology, and more specifically, to an intelligent agent motion control method and device guided by a large language model. Background Art
[0002] Agent control is a method of achieving automatic control using artificial intelligence technology. It is often used in assistive technology, game AI, robot control and other fields to improve user experience and task efficiency. Improving the fluency of agent-to-agent operations so that they can efficiently perform various computer tasks is an inevitable challenge for agents to become human computer assistants.
[0003] At present, the intelligent agent control method mainly uses machine learning, reinforcement learning and other technologies to enable the intelligent agent to automatically adjust its behavior according to the task. However, the existing intelligent agent control has poor accuracy, which leads to a mismatch between the next state distribution of the intelligent agent when performing a task and the state distribution of the expert, making it difficult to solve the problem of accurate computer control. Summary of the invention
[0004] The purpose of the present disclosure is to provide a method and device for controlling the motion of an intelligent agent guided by a large language model, so as to solve the problem of poor accuracy of the current intelligent agent control.
[0005] A first aspect of the embodiments of the present disclosure provides an intelligent agent action control method guided by a large language model, comprising:
[0006] Convert the target task into the target action sequence of the agent based on the large language model;
[0007] The target action sequence is input into a first agent control model to achieve control of the agent.
[0008] A second aspect of the embodiments of the present disclosure provides an intelligent agent action control device guided by a large language model, comprising:
[0009] Action sequence generation module: used to convert the target task into the target action sequence of the intelligent agent based on the large language model;
[0010] Agent control module: used for inputting the target action sequence into the first agent control model to realize the control of the agent.
[0011] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned method for controlling intelligent body actions guided by a large language model when executing the computer program.
[0012] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for controlling the action of an intelligent agent guided by a large language model are implemented.
[0013] The beneficial effects of a large language model-guided intelligent agent action control method and device provided by the embodiments of the present disclosure are: the present invention introduces a large language model, which can decompose the target task and convert it into a target action sequence of the intelligent agent, so that complex tasks can be converted into simple action sequences, simplifying the complex and reducing the complexity of the task, thereby improving the control accuracy of the intelligent agent; then the target action sequence is input into the first intelligent agent control model, and the first intelligent agent control model is used to predict the movement and click action of the intelligent agent cursor. The intelligent agent cursor will perform corresponding movement and click actions according to the prediction results, and finally complete the entire target action sequence. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0015] Figure 1 A flowchart of an intelligent agent action control method guided by a large language model provided by an embodiment of the present disclosure;
[0016] Figure 2 A schematic diagram of a training process of a first agent control model provided in one embodiment of the present disclosure;
[0017] Figure 3 A structural block diagram of an intelligent agent motion control device guided by a large language model provided by an embodiment of the present disclosure;
[0018] Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0019] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0020] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.
[0021] Please refer to Figure 1 , Figure 1 A flow chart of a method for controlling an intelligent agent's motion guided by a large language model provided in one embodiment of the present disclosure, the method comprising:
[0022] S101: Convert the target task into the target action sequence of the agent based on the large language model.
[0023] A Large Language Model (LLM) is a machine learning model that is trained for Natural Language Processing (NLP) tasks such as language translation, sentiment analysis, and text summarization. These models are usually based on neural networks and are trained on large amounts of text data so that they can learn to understand.
[0024] Agent is a basic concept in the field of artificial intelligence, which refers to an entity that can perceive the environment and take actions to achieve specific goals. An agent can be a software program, a robot, a computer system, or other entities that can act autonomously. An agent can handle problems based on its knowledge and skills and make decisions in a given environment.
[0025] In this embodiment, the mouse is used as an intelligent agent. In the problem of computer control of moving the mouse cursor, once there is a slight deviation in the control, it will be difficult to complete the task, which places great demands on the accuracy of the control.
[0026] The target task refers to the language instruction input by the user.
[0027] The target action sequence refers to the final sequence of actions performed by the mouse.
[0028] In this embodiment, the user can input language instructions as needed, and by calling the Application Programming Interface (API) of the large language model, infer the target action sequence of the mouse corresponding to the language instructions. That is, complex target tasks can be converted into multiple action sequences, reducing the complexity of task processing.
[0029] In the case of large language models, an API is typically a set of functions that allow developers to access and use the capabilities of a large language model. An API can be used in a variety of ways, such as integrating a large language model into an application or website, or creating new tools and utilities that use the capabilities of a large language model.
[0030] In this embodiment, the API of the large language model performs reasoning specifically involving the following process:
[0031] Step a: Send the input language instruction (such as a sentence or paragraph) to a large language model as part of a request in a specific format.
[0032] Step b: Specify the desired task and output format in the request.
[0033] Step c: The large language model processes the request and generates a response. The response can be used in an application or website, or it can be used to create new tools and utilities that use the power of the large language model.
[0034] S102: Input the target action sequence into the first agent control model to achieve control of the agent.
[0035] In this embodiment, the target action sequence of the mouse is input into the first agent control model, which is a control model established in advance and is used to predict the movement and click action of the mouse cursor. The first agent control model will predict the movement and click action of the mouse cursor corresponding to each subsequence in the target action sequence in turn. The mouse cursor will perform the corresponding movement and click action according to the prediction results, and finally complete the entire target action sequence.
[0036] From the above, it can be concluded that the present embodiment introduces a large language model, which can decompose the target task and convert it into a target action sequence of the mouse, so that complex tasks can be converted into simple action sequences, simplifying the complex and reducing the complexity of the task, thereby improving the accuracy of mouse control; the target action sequence is then input into the first intelligent agent control model, and the first intelligent agent control model is used to predict the movement and click actions of the mouse cursor. The mouse cursor will perform corresponding movement and click actions according to the prediction results, and finally complete the entire target action sequence.
[0037] In one embodiment of the present disclosure, converting a target task into a target action sequence of an agent based on a large language model includes:
[0038] Based on the large language model, the target task is decomposed into a sub-task sequence of the agent;
[0039] The subtask sequence is modified according to the environmental state information of the subtask sequence to obtain the target action sequence of the intelligent agent.
[0040] In this embodiment, the target action sequence of the mouse corresponding to the language instruction is inferred by calling the application programming interface (API) of the large language model. In the large language model, the reasoning process usually involves decomposing the input target task into a sequence of subtasks and then performing reasoning on each subtask. This enables the model to process tasks more efficiently and perform more effective training on each subtask.
[0041] The large language model breaks down the target task into a series of subtask sequences. First, input the language instruction, and then call the API in the large language model to infer the subtask sequence of the mouse. That is:
[0042] LLM(task)={task_1,…,task_n}
[0043] Among them, task is a language instruction (a natural language description), and {task_1,…,task_n} represents a subtask sequence consisting of n subtasks.
[0044] Traditional agent-based mouse control methods have limited precision control capabilities. One of the main reasons is the covariate shift problem, where the agent may fail to clone the expert's behavior and the predicted behavior has interference, resulting in a mismatch between the next state distribution and the expert's state distribution. During training, interference is usually caused by changes in input observations due to the dynamics of the environment.
[0045] In this embodiment, the environmental status information of the subtask sequence is introduced to obtain a more accurate target work sequence.
[0046] In agent mouse control, environment state information usually refers to all external information related to mouse operation. Specifically, this may include but is not limited to the following categories:
[0047] Mouse position: The agent needs to know the current position of the mouse on the screen in order to determine the next action.
[0048] Screen state: The agent needs to know what is currently displayed on the screen in order to react appropriately.
[0049] Application state: If the agent is operating a specific application, then the application state information is very important. For example, the agent may need to know whether a certain application is running in the foreground, whether there are any unread messages, etc.
[0050] Operating system status: For example, the agent may need to know the current operating system version, whether an update is available, etc.
[0051] Hardware status: For example, the agent may need to know the computer's memory usage, CPU load, etc.
[0052] All of the above information may become the environment state information of the agent when it is controlling the mouse. This information can help the agent execute the user's commands more accurately.
[0053] Specifically, the subtask sequence is modified according to the environmental state information of the subtask sequence to obtain the target action sequence of the mouse, including the following steps:
[0054] S201: predicting the environment state information corresponding to each subtask in the subtask sequence to obtain a predicted environment state information sequence.
[0055] In practical applications, we can crawl the language descriptions and environmental status data of a series of target tasks on the Internet, retrieve each subtask in the subtask sequence from a series of language descriptions, and obtain the environmental status information of tasks similar to the subtask sequence, thereby obtaining the predicted environmental status information sequence for each subtask, and use the predicted environmental status information sequence as external knowledge to assist the large language model.
[0056] S202: Inferring current environment state information of each subtask in the subtask sequence based on the large language model.
[0057] In practical applications, a large language model can easily show forgetfulness on a longer subtask sequence, so the obtained subtask sequence is likely to be wrong. In this embodiment, the subtask sequence generated by the large language model is checked and corrected through multiple reasonings of the world model inside the large language model.
[0058] Before correction, it is necessary to obtain the current environment status information of each subtask in the subtask sequence through the large language model.
[0059] S203: When there is a difference between the current environment state information of each subtask and the predicted environment state information sequence, correction opinion information is obtained according to the difference.
[0060] Specifically, when there is a difference between the current environment state information of each subtask and the predicted environment state information corresponding to the current environment state information of each subtask in the predicted environment state information sequence, correction opinion information of each subtask is obtained according to the difference.
[0061] First, the world model infers the new environment state information corresponding to the current subtask based on the current subtask and the current environment state information corresponding to the current subtask; then it infers the new environment state information corresponding to the next subtask, until the new environment state information corresponding to each subtask is inferred; finally, the new environment state information corresponding to each subtask in the subtask sequence is compared with the predicted environment state information corresponding to each subtask. If there is a difference between the new environment state information corresponding to each subtask in the subtask sequence and the predicted environment state information corresponding to each subtask, it means that there is a problem with the corresponding subtask, and an error message needs to be returned to the large language model.
[0062] S204: Based on the large language model, the subtask sequence is corrected according to the correction opinion information to obtain the target action sequence of the agent.
[0063] In this embodiment, the error information is returned to the large language model, which gives correction suggestions based on the returned error information, re-reasons the subtask sequence, and forms a new subtask sequence. Finally, the large language model forms a target action sequence in a certain order based on the new subtask sequence.
[0064] In one embodiment of the present disclosure, inputting a target action sequence into a first agent control model to implement control of the agent includes:
[0065] The target action sequence is input into the first agent control model, and the movement or click action of the agent cursor is predicted to realize the control of the agent.
[0066] In this embodiment, the target action sequence is the action sequence of moving or clicking the mouse cursor. After the target action sequence is input into the first agent control model, the action of the mouse cursor is predicted, and the mouse cursor completes the corresponding operation according to the predicted action.
[0067] Compared with the traditional mouse control method, the present embodiment takes into account the environmental state information of the mouse. By acquiring and analyzing the environmental state information of the mouse, the agent can execute the user's instructions more accurately. For example, through the mouse position and speed information, the agent can accurately locate the cursor, and through the screen state information, the agent can determine when to perform a click or drag operation. Environmental information is an important basis for agent learning. By collecting and analyzing environmental state information, the agent can continuously optimize its own strategy and improve learning efficiency. It can also dynamically adjust its own behavior according to the environmental state information to meet the needs of the user.
[0068] In one embodiment of the present disclosure, the agent action control method guided by the large language model further includes:
[0069] Inputting the training task into the second agent control model to obtain the first agent control model;
[0070] The second agent control model includes the first agent control model and a cursor prediction module.
[0071] Specifically, the training task is input into the second agent control model to obtain the first agent control model, including:
[0072] Input the training task into the target network module to predict the movement and click actions of the mouse cursor;
[0073] The supervision module supervises the agent to learn the clicks and movements of the mouse;
[0074] The position of the mouse cursor is sensed through the cursor prediction module.
[0075] In this embodiment, the first agent control model is a pre-trained model, and the first agent control model is obtained by training the second agent control model. Because in the actual operation process, it is not necessary to predict the mouse cursor position, but only to know the mouse cursor movement and click actions, so the cursor position prediction module is removed to obtain the first agent control model.
[0076] The first agent control model includes a target network module and a supervision module. The target network module is used to predict the movement and click action of the mouse cursor, and the supervision module is used to supervise the agent to learn the click and movement of the mouse.
[0077] The second agent control model includes the first agent control model and a cursor prediction module, and the cursor prediction module is used to sense the position of the mouse cursor during the training process.
[0078] See also Figure 2 In this embodiment, the target network module includes a convolutional neural network (CNN), a self-attention model (Transformer) and an action prediction module (Action Head), the supervision module includes loss_button (loss function) and loss_camera (loss function), and the cursor prediction module is loss_cursor.
[0079] In this embodiment, the training process of the first agent control model is:
[0080] After the image passes through the CNN convolutional neural network and Transformer, it passes through an Action Head module to predict the movement and click actions of the mouse cursor, and supervises the agent to learn the click and movement of the mouse through loss_button and loss_camera. At the same time, because computer control needs to accurately perceive the position of the mouse cursor, we introduced a self-perceived loss loss_cursor to accurately perceive the position of the mouse cursor; specifically, during training, a cursor predictor module is added to predict the position of the mouse cursor and calculate the loss with the actual position of the mouse cursor, so that the convolutional neural network focuses on the position information of the mouse cursor.
[0081] During the testing phase, the cursor predictor module was removed, leaving only the Action Head module to predict mouse clicks and movements. After the image passes through the CNN convolutional neural network and Transformer, it passes through the Action Head module to obtain the mouse movements.
[0082] In a specific implementation, the present disclosure also provides a specific implementation example of an intelligent agent action control method guided by a large language model, and the implementation steps are as follows:
[0083] S1: Input the task "Please make an item B" into the large language model. The large language model corrects the task execution plan "{drag Z to the second grid, drag W to the fourth grid, and drag U to the eighth grid}" through multi-step reasoning of the world model;
[0084] S2: Form the action sequence “{(Z, 2), (W, 4), (U, 8)}”;
[0085] S3: Execute move_1(Z, 2), move_2(W, 4), move_3(U, 8) in sequence.
[0086] In this embodiment, the starting point and end point of the target action sequence are integrated into the CNN convolutional neural network as task conditions. The CNN convolutional neural network output integrates the visual features of the task conditions (starting point, end point), and then makes a decision in the Transformer to output the action of moving or clicking the mouse cursor. In training, in addition to using conventional loss_button and loss_camera to supervise the actions of the learning agent and imitate the agent's ability to move the mouse, we also use loss_cursor to supervise the decision network to enhance the mouse's self-perception ability and solve the problem of poor accuracy of the current agent's mouse control.
[0087] Corresponding to the above embodiment, a method for controlling intelligent agent actions guided by a large language model is described. Figure 3 This is a block diagram of a large language model-guided intelligent action control device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 3 The intelligent agent action control device 20 guided by a large language model includes: an action sequence generation module 21 and a mouse control module 22.
[0088] The action sequence generation module 21 is used to convert the target task into a target action sequence of the mouse based on the large language model;
[0089] Mouse control module 22: used for inputting the target action sequence into the first agent control model to realize the control of the mouse.
[0090] In one embodiment of the present disclosure, the action sequence generation module 21 is specifically used to:
[0091] Decomposing the target task based on the large language model to obtain a subtask sequence of the mouse;
[0092] The subtask sequence is modified according to the environmental state information of the subtask sequence to obtain the target action sequence of the mouse.
[0093] In one embodiment of the present disclosure, the action sequence generation module 21 is specifically used to:
[0094] Predicting the environmental state information corresponding to each subtask in the subtask sequence to obtain a predicted environmental state information sequence;
[0095] Inferring current environment state information of each subtask in the subtask sequence based on the large language model;
[0096] When there is a difference between the current environment state information of each subtask and the predicted environment state information sequence, the revised opinion information is obtained according to the difference;
[0097] The subtask sequence is modified according to the modification opinion information based on the large language model to obtain the target action sequence of the mouse.
[0098] In one embodiment of the present disclosure, the action sequence generation module 21 is specifically used to:
[0099] When there is a difference between the current environment state information of each subtask and the predicted environment state information corresponding to the current environment state information of each subtask in the predicted environment state information sequence, correction opinion information of each subtask is obtained according to the difference.
[0100] In one embodiment of the present disclosure, the mouse control module 22 is specifically used for:
[0101] The target action sequence is input into the first agent control model to predict the movement or click action of the mouse cursor to achieve the control of the mouse.
[0102] In one embodiment of the present disclosure, a large language model guided intelligent agent action control device further includes:
[0103] Training module: inputting the training task into the second agent control model to obtain the first agent control model;
[0104] The second agent control model includes the first agent control model and a cursor prediction module.
[0105] In one embodiment of the present disclosure, the training module is specifically used to:
[0106] Input the training task into the target network module to predict the movement and click actions of the mouse cursor;
[0107] Supervise the agent to learn mouse clicks and movements through a supervision module;
[0108] The position of the mouse cursor is sensed by the cursor prediction module.
[0109] See also Figure 4 , Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 3 The functions of modules 21 to 22 are shown.
[0110] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0111] The input device 302 may include a touch panel, a fingerprint sensor (for collecting fingerprint information of the user and fingerprint direction information), a microphone, etc. The output device 303 may include a display (LCD, etc.), a speaker, etc.
[0112] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.
[0113] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the intelligent agent action control method guided by a large language model provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.
[0114] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, wherein the program instructions are executed by a processor to implement all or part of the processes in the above-mentioned embodiment method, and may also be completed by instructing the relevant hardware through the computer program, wherein the computer program may be stored in a computer-readable storage medium, and wherein the computer program may be executed by the processor to implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signal and telecommunication signal.
[0115] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0116] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0118] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.
[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.
[0120] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0121] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A method for controlling intelligent agent actions guided by a large language model. It is characterized in that include: Convert the target task into the target action sequence of the agent based on the large language model; Taking the mouse as the agent, the target action sequence refers to the action sequence finally generated by the mouse; The method of converting the target task into a target action sequence of the agent based on the large language model includes: Decomposing the target task based on the large language model to obtain a subtask sequence of the agent; Modifying the subtask sequence according to the environmental state information of the subtask sequence to obtain the target action sequence of the agent; The environmental status information refers to all external information related to the mouse operation; the environmental status information includes the mouse position, screen status, application status, operating system status and hardware status; The step of modifying the subtask sequence according to the environmental state information of the subtask sequence to obtain the target action sequence of the agent includes: Predicting the environmental state information corresponding to each subtask in the subtask sequence to obtain a predicted environmental state information sequence; Inferring current environment state information of each subtask in the subtask sequence based on the large language model; When there is a difference between the current environment state information of each subtask and the predicted environment state information sequence, the revised opinion information is obtained according to the difference; Based on the large language model, the subtask sequence is modified according to the modified opinion information to obtain the target action sequence of the intelligent agent; The target action sequence is input into a first agent control model to achieve control of the agent.
2. The method for controlling an intelligent agent's motion guided by a large language model as claimed in claim 1, It is characterized in that When there is a difference between the current environment state information of each subtask and the predicted environment state information sequence, the correction opinion information is obtained according to the difference, including: When there is a difference between the current environment state information of each subtask and the predicted environment state information corresponding to the current environment state information of each subtask in the predicted environment state information sequence, correction opinion information of each subtask is obtained according to the difference.
3. The method for controlling an intelligent agent's motion guided by a large language model as claimed in claim 1, It is characterized in that The step of inputting the target action sequence into a first agent control model to realize control of the agent includes: The target action sequence is input into the first agent control model to predict the movement and click actions of the agent to achieve control of the agent.
4. The method for controlling an intelligent agent's motion guided by a large language model as claimed in claim 1, It is characterized in that Also includes: Inputting the training task into the second agent control model to obtain the first agent control model; The second agent control model includes the first agent control model and a cursor prediction module.
5. The method for controlling an intelligent agent's motion guided by a large language model as claimed in claim 4, It is characterized in that The step of inputting the training task into the second agent control model to obtain the first agent control model includes: Input the training task into the target network module to predict the movement and click actions of the agent's cursor; Supervising the agent to learn the movement and click actions of the agent through a supervision module; The position of the agent cursor is sensed by the cursor prediction module.
6. An intelligent action control device guided by a large language model, It is characterized in that include: Action sequence generation module: used to convert the target task into the target action sequence of the intelligent agent based on the large language model; Taking the mouse as the agent, the target action sequence refers to the action sequence finally generated by the mouse; The action sequence generation module is specifically used to decompose the target task based on the large language model to obtain a sub-task sequence of the intelligent agent; Modifying the subtask sequence according to the environmental state information of the subtask sequence to obtain the target action sequence of the agent; The environmental status information refers to all external information related to the mouse operation; the environmental status information includes the mouse position, screen status, application status, operating system status and hardware status; The action sequence generation module is further specifically used to predict the environmental state information corresponding to each subtask in the subtask sequence to obtain a predicted environmental state information sequence; Inferring current environment state information of each subtask in the subtask sequence based on the large language model; When there is a difference between the current environment state information of each subtask and the predicted environment state information sequence, the revised opinion information is obtained according to the difference; Based on the large language model, the subtask sequence is modified according to the modified opinion information to obtain the target action sequence of the intelligent agent; Agent control module: used for inputting the target action sequence into the first agent control model to realize the control of the agent.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
