Task instruction execution method and device, storage medium and electronic device

Through the multimodal large language model and search enhancement model built in the device agent center, combined with the interface information and historical decision information of the terminal device, efficient and accurate task instruction execution without manual operation is achieved, and the execution in the existing technology is solved The problem of slow speed and low success rate.

CN119088899AInactive Publication Date: 2024-12-06BEIJING JUEYING INTELLIGENT TECH CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410869829.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to execute task instructions efficiently and accurately without manually operating the terminal device, and requires a large number of user-assisted operations, resulting in slow execution speed and low success rate.

Method used

A task instruction execution method is designed, using the multimodal large language model, intelligent large language model and search enhancement model built in the device agent center, and generate and execute the interface information, system prompt information and historical decision information of the terminal device. Execute instructions to realize the automated execution of task instructions.

Benefits of technology

It improves the execution efficiency and accuracy of task instructions, reduces the inference thinking time of model repredicting, and improves the execution success rate of task instructions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088899A_ABST
    Figure CN119088899A_ABST
Patent Text Reader

Abstract

The invention discloses a task instruction execution method and device, a storage medium and an electronic device, is applied to an equipment agent center, and comprises the following steps: obtaining a task instruction of a target object to application software on terminal equipment; obtaining a first operation instruction set corresponding to the task type to which the task instruction belongs from a task operation manual by using a retrieval enhancement model; obtaining interface description information corresponding to the interface information displayed by the terminal equipment in real time by using the multi-modal large language model; inputting the interface information, interface operation information corresponding to the interface description information, system prompt information generated based on the task instruction and historical decision information corresponding to the first operation instruction set into an intelligent large language model to obtain a first execution instruction output by the intelligent large language model; and sending the first execution instruction to the terminal equipment, so that the terminal equipment executes the first execution instruction and obtains a first execution result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent large language models, and in particular to a method and device for executing a task instruction, a storage medium and an electronic device. Background Art

[0002] At present, as the functions of terminal devices are constantly improving, more and more users use terminal devices to obtain services. Of course, users also have higher requirements for the service quality and service efficiency of terminal devices. For example, when the vehicle is driving, users cannot manually operate the terminal device of the vehicle computer normally.

[0003] The current solution is to use models to replace user operations, but these solutions require a lot of user-assisted operations and cannot perform user tasks in one step. Not only is the speed of task execution slow due to time-consuming, but the success rate of task execution is also very low.

[0004] Therefore, in the related art, there is a technical problem of how to efficiently and accurately execute task instructions in the case of not manually operating the terminal device. However, no effective solution has been proposed yet for this problem. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for executing a task instruction, a storage medium and an electronic device.

[0006] According to one aspect of an embodiment of the present invention, there is provided a method for executing a task instruction, which is applied to a device proxy center, wherein a data transmission channel is established between the device proxy center and a terminal device of a target object, and the device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in, comprising: obtaining the task instruction of the target object for the application software on the terminal device; using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from a task operation manual; using the multimodal large language model to obtain interface description information corresponding to interface information displayed in real time by the terminal device; inputting the interface information, interface operation information corresponding to the interface description information, system prompt information generated based on the task instruction, and historical decision information corresponding to the first set of operation instructions into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model; and sending the first execution instruction to the terminal device so that the terminal device executes the first execution instruction and obtains a first execution result.

[0007] Further, in an exemplary embodiment, the task instruction is matched with an operation instruction set in a task operation manual; and the retrieval enhancement model is used to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual, including: in response to a failure to match the task instruction with the operation instruction set in the task operation manual, using the retrieval enhancement model to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual.

[0008] Furthermore, in an exemplary embodiment, the method also includes: in response to a successful match between the task instruction and the operation instruction set in the task operation manual, using the retrieval enhancement model to obtain a second operation instruction set corresponding to the task instruction from the task operation manual; when it is determined that the instruction type of the task instruction is a single-step execution instruction, determining the operation instruction in the second operation instruction set that matches the task instruction as a second execution instruction, sending the second execution instruction to the terminal device, and obtaining a second execution result obtained after the terminal device executes the second execution instruction; when it is determined that the instruction type of the task instruction is a batch execution instruction, sending all operation instructions in the second operation instruction set that match the task instruction to the terminal device, and obtaining a third execution result after the terminal device executes all the operation instructions in a linked execution order, wherein the linked execution order represents the order in which all the operation instructions are linked to be executed.

[0009] Furthermore, in an exemplary embodiment, the method also includes: generating basic operation data based on historical execution results obtained after the terminal device executes historical execution instructions; establishing the retrieval enhancement model based on the basic operation data; and obtaining successful execution instructions that are successfully executed from the historical execution instructions, and generating the task operation manual based on the successful execution instructions and the execution results corresponding to the successful execution instructions.

[0010] Furthermore, in an exemplary embodiment, the method also includes: obtaining interface operation information corresponding to the interface description information, including: displaying the interface image displayed in real time by the terminal device to the operation object through a web page display tool, so as to generate a target interface image with element marking information based on the annotation operation of the operation object on the interface image; and generating the interface operation information according to the target interface image and the operation information indicated by the target interface image.

[0011] Further, in an exemplary embodiment, the multimodal large language model includes at least: a visual transformation module, a multi-layer perception module, and a visual language model. The multimodal large language model is used to obtain interface description information corresponding to the interface information displayed in real time by the terminal device, including: performing text recognition on the interface image displayed in real time by the terminal device to obtain text description information of the interface image; using the visual transformation module to encode the interface image to obtain a first image feature vector; using the multi-layer perception module to perform a vector adjustment operation on the first image feature vector to obtain a second image feature vector, wherein the vector adjustment operation includes at least one of the following: a linear transformation operation, a bias operation, an activation operation, a propagation operation, and an update operation; aligning the second image feature vector with the text feature vector corresponding to the text description information, and inputting the aligned second image feature vector and the text feature vector into the visual language model to obtain the interface description information output by the visual language model.

[0012] Furthermore, in an exemplary embodiment, after inputting the aligned second image feature vector and the text feature vector into the visual language model to obtain the interface description information output by the visual language model, the method further includes: acquiring information of elements of the interface image and association relationships between a plurality of different elements from the interface description information; and correcting the association relationships according to correction instructions to obtain corrected interface description information.

[0013] Furthermore, in an exemplary embodiment, before using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device, the method also includes: using a historical interface image corresponding to the historical interface information of the terminal device and historical text description information of the historical interface image as input samples, and using the historical interface description information of the historical interface image as output samples to train the original multimodal large language model to obtain the multimodal large language model, wherein multiple different elements in the corresponding interface have been marked on the historical interface image.

[0014] According to another aspect of an embodiment of the present invention, there is also provided a device for executing a task instruction, comprising: a device proxy center, a data transmission channel is established between the device proxy center and the terminal device of the target object, the device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in; the device proxy center is used to execute the above-mentioned task instruction execution method. The device proxy center is at least used to execute: obtaining the task instruction of the target object for the application software on the terminal device; using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual; using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device; inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction and the historical decision information corresponding to the first set of operation instructions into the intelligent large language model, and obtaining the first execution instruction output by the intelligent large language model; sending the first execution instruction to the terminal device so that the terminal device executes the first execution instruction and obtains the first execution result.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute a method for executing the above-mentioned task instructions when running.

[0016] According to another aspect of an embodiment of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the execution method of the task instruction through the computer program.

[0017] According to the present invention, a device proxy center based on a data transmission channel established between the device proxy center and the terminal device of the target object obtains the task instructions of the target object for the application software on the terminal device; a retrieval enhancement model of the device proxy center is used to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from a task operation manual; a multimodal large language model of the device proxy center is used to output interface description information corresponding to the interface information displayed in real time by the terminal device; the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction and the historical decision information corresponding to the first operation instruction set are input into the intelligent large language model of the device proxy center to obtain a first execution instruction output by the intelligent large language model; the first execution instruction is sent to the terminal device so that the terminal device executes the first execution instruction and obtains a first execution result. By adopting the above technical scheme, a retrieval enhancement model that can retrieve the task operation manual is designed, a multimodal large language model that can generate interface description information in an integrated manner, and an intelligent large language model that generates execution instructions according to the input interface information, interface operation information, system prompt information and historical decision information. After accurately understanding the user instruction, the retrieval enhancement model is used to obtain the operation instruction set corresponding to the task instruction, the multimodal large language model is used to identify the interface information to generate the interface description information, and then the intelligent large language model is used to generate the execution instruction for the interface description information and other relevant information (i.e., interface information, system prompt information and historical decision information). This scheme, even if there is no execution instruction that can be directly executed from the task operation manual, can also search for similar operations from the task operation manual and submit them to the intelligent large language model, so that the intelligent large language model can make subsequent operations by referring to historical successful cases, reduce the reasoning and thinking time required for each step of the model re-prediction, and improve the execution efficiency and execution success rate of the task instruction. Therefore, the technical scheme of the present invention solves the technical problem of how to efficiently and accurately execute task instructions in the case of non-manual operation of terminal devices in the related technology, thereby improving the execution efficiency and execution accuracy of task instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a server of the method for executing task instructions of an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for executing a task instruction according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a process of outputting execution instructions using an intelligent large language model according to an embodiment of the present invention;

[0022] Figure 4 is a flowchart of processing a task based on an operation manual according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of a generation principle of a task path data set according to an embodiment of the present invention;

[0024] Figure 6 is a flow chart of a semi-supervised workflow according to an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of a principle for generating a UI element description data set according to an embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of a process for generating image description information according to an embodiment of the present invention;

[0027] Fig. 9 4 is a structural block diagram of a device for executing task instructions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms and "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The method embodiments provided in the embodiments of the present invention can be executed in a server, a mobile terminal, a device proxy center, or a similar computing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server of a method for executing a task instruction according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the execution method of the task instructions in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, which is equivalent to implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a method for executing a task instruction is provided, which is applied to a device proxy center. A data transmission channel is established between the device proxy center and a terminal device of a target object. The device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in.

[0034] It is understandable that in the field of vehicle-mounted technology, the above-mentioned device proxy center is specifically a vehicle-mounted communication gateway (Telematics Control Unit, TCU) or vehicle-mounted information service platform (In-Vehicle Infotainment, IVI) system that connects the server and the user's vehicle-mounted terminal, responsible for processing data transmission, communication and information exchange inside and outside the vehicle. In general, the device proxy center realizes data transmission, communication and information exchange by connecting the server and the user's vehicle-mounted terminal equipment, providing users with a convenient, safe and intelligent driving experience.

[0035] For example, in terms of device form, the device proxy center may be an integrated electronic device, including one or more physical components such as a communication module, a computing unit, a storage device, and a user interface, which can be installed inside a vehicle.

[0036] From a functional analysis, the device proxy center can send information such as the vehicle's status, location, speed, etc. to a remote server, and also receive data and instructions from the server. It can also serve as a bridge for the vehicle to communicate with other devices (such as smartphones, wearable devices, etc.) to achieve data synchronization and remote control. It can also provide a user interface such as a touch screen, voice recognition, or physical buttons so that the driver or passenger can interact with the system. At the same time, it can also integrate multiple services such as navigation, entertainment, emergency rescue, and remote diagnosis.

[0037] According to the functions and uses, the device proxy center can be divided into different types, such as basic, advanced and professional. The basic type only provides basic communication and navigation functions, while the advanced type can further provide advanced services, and the professional type can include more advanced services and customized functions.

[0038] The device agent center of the present application can be applied to different application scenarios. For example, for navigation and map update scenarios, the device agent center can provide real-time traffic information and route planning. For emergency rescue scenarios, the device agent center can automatically send a request for help when an accident occurs. For remote monitoring and diagnosis scenarios, the device agent center can allow car owners or service providers to remotely check the status of the vehicle. For entertainment and information scenarios, the device agent center can provide multimedia services such as music, news, and weather. For vehicle control scenarios, the device agent center can realize functions such as remote start, locking / unlocking doors, etc. For data analysis scenarios, the device agent center can collect and analyze vehicle usage data to optimize vehicle performance and driving habits.

[0039] In practical applications, the device proxy center often needs to follow a series of communication standards and protocols, such as cellular networks (4G / 5G), WiFi, Bluetooth, etc., to ensure compatibility with different devices and services. And because the device proxy center involves a large amount of personal and vehicle data, strict security measures need to be taken to protect data security and user privacy.

[0040] Optionally, the execution method of the task instruction of this embodiment can also be applied to operating systems such as vehicle-mounted systems, mobile phone systems, and computer systems.

[0041] Figure 2 is a flow chart of a method for executing a task instruction according to an embodiment of the present invention, the flow chart comprising the following steps:

[0042] Step S202, obtaining a task instruction of the target object for the application software on the terminal device;

[0043] It is understandable that the above-mentioned method of acquiring task execution may include actively monitoring the current state of the user (i.e., the target object), actively generating corresponding task instructions for the user based on the user state, or passively receiving task instructions such as voice instructions or gesture instructions from the user, etc., and this application does not limit this. Among them, for the case of actively generating task instructions, it can be a default instruction generated according to user settings and triggered to be executed in a specific time period or under a specific state.

[0044] Step S204, using the retrieval enhancement model to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual;

[0045] Here, the retrieval enhancement model is a neural network model used to retrieve the corresponding set of operation instructions according to the task instructions. It can parse the content of relatively abstract and complex task instructions, obtain attribute information such as the type of task instructions, and retrieve a set of specific operation instructions that match the content or type of the task instructions in the task operation manual. The set of retrieved operation instructions can be a set constructed by specific operation instructions used to complete the task instructions. When the operation instructions in the retrieved operation instruction set are executed, the execution result obtained is the task execution result expected to be obtained by the corresponding task instruction.

[0046] Step S206, using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device;

[0047] Among them, the interface information displayed in real time by the terminal device may be the interface information corresponding to the screenshot of the terminal device, and the interface information may include the image information in the screenshot. The interface description information represents information used to describe the interface features. Specifically, the interface description information may include but is not limited to at least one of the element layout of the interface, the attribute information of the elements in the interface, and the association relationship between the elements in the interface. Among them, the element layout may include: the coordinates of the elements and / or the relative positions between the elements; the attribute information of the elements may include: at least one of the name, type, function, executable operation type, etc. of the element. The multimodal large prediction model may be a pre-trained neural network model for parsing user interface information to obtain interface description information.

[0048] Step S208, inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction, and the historical decision information corresponding to the first operation instruction set into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model;

[0049] The intelligent large language model is a large model used to predict the specific operation instructions required to complete the task instruction based on the task instruction issued to the terminal device and the information related to the anthropomorphic execution of the task instruction, such as the interface information and interface description information of the terminal device. The intelligent large language model can be obtained in advance through training of a data set containing labeled information.

[0050] The historical decision information may include the decision process information from the historical task instructions to the execution instructions during the execution of the historical tasks. Optionally, the historical decision information may also include evaluation information, which indicates whether the execution result after the corresponding execution instruction in the execution of the historical task meets the expected execution result of the historical task. Optionally, in the training process of the intelligent large language model, the historical decision information can be used to assist the intelligent large language model in learning the decomposition logic from the historical task instructions to the specific execution instructions; in the process of using the intelligent large language model to process the task instructions issued by the target object, the historical decision information can also be used to assist the intelligent large language model in predicting the execution instructions corresponding to the task instructions. Among them, each execution instruction corresponds to an operation step on the terminal device, for example, a sequence of execution instructions a, b, c corresponds to starting an application on the terminal device, selecting a main function in the application, and performing further sub-function operations according to the execution result of the main function.

[0051] Furthermore, this embodiment can also use the historical interface information, the historical interface operation information corresponding to the historical interface description information, the historical system prompt information generated based on the historical task instructions and all historical decision information as input samples, and use the historical execution instructions as output samples to train the initial intelligent language large model to obtain the intelligent large language model.

[0052] Step S210: Send the first execution instruction to the terminal device, so that the terminal device executes the first execution instruction and obtains a first execution result.

[0053] Through the above steps, based on the device proxy center that has established a data transmission channel with the terminal device of the target object, the task instructions of the target object for the application software on the terminal device are obtained; the retrieval enhancement model of the device proxy center is used to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual; the multimodal large language model of the device proxy center is used to output interface description information corresponding to the interface information displayed in real time by the terminal device; the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction and the historical decision information corresponding to the first operation instruction set are input into the intelligent large language model of the device proxy center to obtain a first execution instruction output by the intelligent large language model; the first execution instruction is sent to the terminal device so that the terminal device executes the first execution instruction and obtains a first execution result. By adopting the above technical scheme, a retrieval enhancement model that can retrieve the task operation manual is designed, a multimodal large language model that can generate interface description information in an integrated manner, and an intelligent large language model that generates execution instructions according to the input interface information, interface operation information, system prompt information and historical decision information. After accurately understanding the user instruction, the retrieval enhancement model is used to obtain the operation instruction set corresponding to the task instruction, the multimodal large language model is used to identify the interface information to generate the interface description information, and then the intelligent large language model is used to generate the execution instruction for the interface description information and other relevant information (i.e., interface information, system prompt information and historical decision information). This scheme, even if there is no execution instruction that can be directly executed from the task operation manual, can also search for similar operations from the task operation manual and submit them to the intelligent large language model, so that the intelligent large language model can make subsequent operations by referring to historical successful cases, reduce the reasoning and thinking time required for each step of the model re-prediction, and improve the execution efficiency and execution success rate of the task instruction. Therefore, the technical scheme of the present invention solves the technical problem of how to efficiently and accurately execute task instructions in the case of non-manual operation of terminal devices in the related technology, thereby improving the execution efficiency and execution accuracy of task instructions.

[0054] In one embodiment, in combination Figure 3 The generation process of the intelligent large language model is described below. Taking the user instruction (i.e., task instruction) as "I want Andy Lau's Love Potion" as an example, first, obtain the operation of this instruction on each UI page (i.e., the above interface image), and then, based on the system prompt information ( Figure 3 System Prompt), historical decision-making process ( Figure 3 History Decision), the current UI page ( Figure 3 Image Input) and Operations ( Figure 3Current Action) to build an intelligent large language model ( Figure 3 Finally, the current COT output (i.e., execution instruction) is obtained through the intelligent large language model and used in the next step of execution. The intuitive COT execution process can be referred to as follows in conjunction with Figure 5 to Figure 6 Description content.

[0055] Furthermore, in this process, a reflection mechanism of the intelligent large language model can be designed to achieve self-improvement of the intelligent large language model. The reflection mechanism of the intelligent large language model is a dynamic adaptive improvement process based on the feedback incentive mechanism. In this process, the intelligent large language model can judge the consistency between the output execution instructions and the user's expected results based on positive or negative feedback incentives, so that by continuously accepting new data and user feedback, it can continuously update and optimize its knowledge base and parameters. Under the feedback incentive mechanism, the intelligent large language model can adjust parameters and optimize the structure according to the data and data feedback, and then adjust the internal learning strategy, improve the adaptability to the task, and learn behaviors with better results. In the absence of external annotations, the model can also use its own predictions and outputs to generate training signals to achieve unsupervised self-learning and improvement, and further combine data enhancement and synthesis techniques to improve its generalization ability.

[0056] Optionally, different data sources and verification methods may be combined for data fusion and verification to ensure the accuracy and reliability of the generated task path data.

[0057] In an optional embodiment, the task instruction can also be matched with the set of operation instructions in the task operation manual; and the use of the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual includes: in response to the task instruction failing to match the set of operation instructions in the task operation manual, using the retrieval enhancement model to obtain the first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual. This embodiment proposes that the matching result of matching the task instruction with the set of operation instructions in the task operation manual is used as a selection condition for selecting a set of operation instructions from the task operation manual, that is, for the case where the matching fails, an instruction set (that is, the first set of operation instructions) that is close to the task instruction is selected from the task operation manual. For the case where the matching is successful, the instruction set corresponding to the task instruction (that is, the second set of operation instructions) can be directly selected to use the operation instructions in the secondary instruction set.

[0058] In one embodiment, the situation of successful matching is further explained, that is, in response to the task instruction successfully matching the operation instruction set in the task operation manual, the retrieval enhancement model is used to obtain a second operation instruction set corresponding to the task instruction from the task operation manual; when it is determined that the instruction type of the task instruction is a single-step execution instruction, the operation instruction in the second operation instruction set that matches the task instruction is determined as a second execution instruction, the second execution instruction is sent to the terminal device, and a second execution result obtained after the terminal device executes the second execution instruction is obtained; when it is determined that the instruction type of the task instruction is a batch execution instruction, all operation instructions in the second operation instruction set that match the task instruction are sent to the terminal device, and a third execution result is obtained after the terminal device executes all the operation instructions in a linked execution order, and the linked execution order represents the order in which all the operation instructions are linked to be executed.

[0059] Among them, for task instructions, the corresponding instruction type can be set according to the complexity of its execution. In this application, the instruction type of single-step execution instruction is set for task instructions that can be executed in only one step, and the instruction type of batch execution instruction is set for task instructions that require at least two steps to be executed. For example, if the user instruction is "light up the screen", which can be completed in only one step, it is set to the instruction type of single-step execution instruction. If the user instruction is "play Andy Lau's Love Water", at least "open the music player software", "find Andy Lau's song list", "search for Love Water from Andy Lau's song list", "play Love Water" and other steps are required, it is set to the instruction type of batch execution instruction. Among them, the sequence between "open the music player software", "find Andy Lau's song list", "search for Love Water from Andy Lau's song list", and "play Love Water" is the linkage execution order of the terminal device.

[0060] This embodiment can save the previous user instructions and the corresponding successfully executed cases (i.e., the second operation instruction set) as a task operation manual in advance, so as to directly obtain the operation instruction set corresponding to the same user instruction from the task operation manual. In particular, for the operation manuals with strong repetitiveness (for example, listening to Andy Lau's songs, listening to Jacky Cheung's songs), it can also be saved as a batch cache. In this way, in the subsequent online reasoning, if the user instructions are the same, the batch cache can be hit, and the corresponding operation instructions can be directly returned, which greatly speeds up the execution speed and success rate of repeated tasks, and the aforementioned intelligent large language model can also synchronously support users to modify and change tasks or ask questions at any time. Moreover, compared with existing open source projects, the solution of the present invention greatly saves the time of executing steps and has a faster execution speed. For example, when a user is driving, he can express his demands by voice. At this time, the device proxy center completes the corresponding needs based on the built-in model according to the application of the user's device, and realizes the operation of the application on behalf of the user, such as completing the review of song reviews, ordering takeout, and controlling the vehicle control device. Tasks, and it is fast and accurate, saving a lot of user energy.

[0061] Further, combined with Figure 4 The process shown in the figure, when identifying the user instruction ( Figure 4 's query), and then request the RAG operation manual, RAG batch matching, and auxiliary algorithms at the same time. For the batch matching of the RAG model, when the operation instruction is matched, the information completion and variable replacement are performed to obtain the operation execution. When batch reuse processing is supported, the operation instructions are issued in batches. When batch processing is not supported, all information is assembled and input into the LLM (Large Language Model), and then the decision result of the LLM is issued in a single step. The decision result is encapsulated and sent to the device side, requesting the device to execute the corresponding steps. After executing all the steps of this batch, the device side continues to upload the execution results to the cloud. The cloud then judges the task based on the execution result for subsequent processing.

[0062] Among them, if the device side confirms that the task has been completed, the task information is saved and no instructions are issued to the client. If the device side confirms that the task has not been completed, the subsequent actions are continued.

[0063] It should be noted that RAG (Retrieval-Augmented Generation) is a machine learning method that combines retrieval and generation to improve the performance of natural language processing tasks. The RAG model is the above retrieval-augmented model. When generating text, this model not only relies on the knowledge within the model, but also provides relevant information by retrieving external knowledge bases. In this way, complex questions can be answered more accurately or richer text can be generated.

[0064] The RAG Operation Manual can be understood as a guide document for using a system, device or software, which is used to provide solutions when users need to understand how to use the system, how to perform tasks, how to find specific operation steps or parameter settings, or how to solve problems encountered during use. The RAG Operation Manual includes, for example, an operation guide, which provides detailed step-by-step instructions to teach users how to perform various operations. The above-mentioned first operation instruction set, second operation instruction set, etc. can be saved here.

[0065] Furthermore, the RAG operation manual may also include other contents, as follows:

[0066] System Overview: Introduces the functions and main features of the system.

[0067] Installation and Configuration: Instruct users on how to install and configure the system.

[0068] User Interface: Describes the layout and functionality of the user interface.

[0069] Troubleshooting: Provides methods and tips for solving common problems.

[0070] Maintenance and upgrade: guide users on how to maintain the system and perform software upgrades.

[0071] To help users understand how to use the operation manual to query instructions, the following operation steps are further proposed:

[0072] 1. Clarify needs: Determine the specific questions or needs that need to be inquired.

[0073] 2. Find related content: Find content related to the query based on the table of contents or index of the operation manual.

[0074] 3. Read the instructions: Read the instructions and steps in the relevant sections carefully to understand how to perform the required operations.

[0075] 4. Follow the steps: Follow the instructions in the operation manual and perform the operations step by step.

[0076] 5. Record and feedback: During use, record the problems encountered and operation results, and provide feedback when necessary.

[0077] Furthermore, in an exemplary embodiment, the following steps may be further implemented:

[0078] Step S11, generating basic operation data according to the historical execution results obtained after the terminal device executes the historical execution instructions;

[0079] Step S12, establishing the retrieval enhancement model based on the basic operation data;

[0080] Step S13, and, obtaining a successfully executed successful execution instruction from the historical execution instructions, and generating the task operation manual according to the successfully executed instruction and the execution result corresponding to the successfully executed instruction.

[0081] In this embodiment, in addition to the above-mentioned solution of saving the successful cases of previous user requests as an operation manual, it is also possible to generate general operation data of similar software by combining the data of historical successful operations and the software usage documents publicly available on the Internet, and then combine relevant retrieval and generation algorithms to construct a task operation manual.

[0082] Through the above steps, this embodiment realizes modeling and filtering of the operation manual, and generates a combination of operation instructions for completing specific tasks for operation manuals with strong repetitiveness (for example: listening to Andy Lau's songs, listening to Jacky Cheung's songs), which can then be saved as a batch processing cache for subsequent direct query and use.

[0083] Furthermore, in an exemplary embodiment, a specific implementation process for obtaining interface operation information corresponding to the interface description information is provided, including: step S21, displaying the interface image displayed in real time by the terminal device to the operation object through a web page display tool, so as to generate a target interface image with element marking information based on the annotation operation of the operation object on the interface image; step S22, generating the interface operation information according to the target interface image and the operation information indicated by the target interface image.

[0084] In this embodiment, a web page display tool is developed to display UI screenshots (i.e., interface images) captured from the terminal device on the end side, and allows the annotator to select and annotate operations, thereby establishing a semi-supervised tool. An effective semi-supervised tool can be developed by combining the advantages of automation and manual annotation to complete the collection and annotation of task paths. The specific process of generating a task path dataset is as follows: Figure 5As shown in the figure, the generated task dataset can accurately reflect the problem of the task path of the device agent center after understanding the human instructions to operate the application, and accurately capture the page screenshots and thought chain data of each step of the operation. Therefore, the collection and annotation of the task path based on the semi-supervised tool is helpful to generate a thought chain process that matches each step of the operation, so that the model can simulate the thinking process of humans during operation. The specific processing process of the semi-supervised workflow is as follows: Figure 6 shown.

[0085] Further, in an exemplary embodiment, the multimodal large language model includes at least: a visual transformation module, a multi-layer perception module, and a visual language model. The implementation process of using the multimodal large language model in the above step S206 to obtain the interface description information corresponding to the interface information displayed in real time by the terminal device may include: performing text recognition on the interface image displayed in real time by the terminal device to obtain the text description information of the interface image; encoding the interface image using the visual transformation module to obtain a first image feature vector; performing a vector adjustment operation on the first image feature vector using the multi-layer perception module to obtain a second image feature vector, wherein the vector adjustment operation includes at least one of the following: a linear transformation operation, a bias operation, an activation operation, a propagation operation, and an update operation; aligning the second image feature vector with the text feature vector corresponding to the text description information, and inputting the aligned second image feature vector and the text feature vector into the visual language model to obtain the interface description information output by the visual language model.

[0086] In this embodiment, OCR (Optical Character Recognition) technology can be used to identify and transcribe the text in the UI elements in the interface image, so as to obtain the text description information of the interface image, thereby improving the accuracy of the UI element name. By further aligning the MLP with the text input, the accuracy of the description information can be further improved, and the quality of the interface description information can be optimized and improved by fusion of multiple strategies.

[0087] In one embodiment, a process is provided for implementing the interface description information corresponding to the interface information displayed in real time by the terminal device using the multimodal large language model. Specifically, after the UI screenshot is encoded by VisionTransformer to obtain image features, it is further aligned with the text input through MLP, and sent to VLM for prefill, and the final result is the description information. This process can be understood as follows: first, the image is encoded using VisionTransformer to obtain image features; then the image features are adjusted through Multi-Layer Perceptron; then feature alignment is performed to make the image features more consistent with the feature representation of the text input; finally, the aligned image features and text input are sent to the Visual Language Model for prefill, so as to achieve better feature fusion and knowledge transfer in subsequent tasks.

[0088] In the above process, Vision Transformer (ViT) is a Transformer-based image processing model. It divides the image into a series of small blocks (called patches), and then uses these small blocks as input sequences and processes them through the Transformer architecture to obtain the feature representation of the image.

[0089] Multi-Layer Perceptron (MLP) is a basic neural network structure consisting of multiple linear layers and non-linear activation functions. In this process, MLP may be used to adjust or transform the image features encoded by ViT to better align with the text input.

[0090] Feature Alignment is a process of mapping feature representations of different modalities (such as images and text) into a common feature space. In this process, the image features are adjusted through MLP to make them more consistent with the feature representation of the text input.

[0091] Visual Language Model (VLM) is a model that combines visual and language information and can handle the joint representation of images and text. In this process, the image features that have been feature aligned are sent to the VLM together with the text input for prefill. The word "prefill" here means that during the training process, the feature representation of the image and text is pre-input into the VLM to facilitate better feature fusion and knowledge transfer in subsequent tasks.

[0092] Optionally, when generating a UI element description dataset (i.e., interface description information), in view of the difficulty of automatically identifying and describing UI elements and accurately describing the associations between elements, a method can be used based on Figure 7 The generation process shown uses the marked original images and the XML information of the images (i.e., interface operation information) output by the data acquisition module to train a multimodal large model capable of processing multiple data types. The multimodal large model is used to automatically understand and analyze visual elements and their functions, and then automatically generates descriptive information such as the name, coordinates, and function description of the elements in the UI screenshot.

[0093] like Figure 7 As shown, when labeling the task pairs between the original image (ie, UI screenshot) and the description information according to the above reasoning stage, the multimodal large model can be fine-tuned based on the open source model.

[0094] in, Figure 7 The XML information of the picture shown generally refers to an Extensible Markup Language (XML) file used to describe image content, attributes, structure, and metadata.

[0095] XML is a markup language used to store and transmit data. It is self-descriptive, flexible, and extensible. In image processing and metadata management, XML format is often used to store detailed information about images, including but not limited to the following aspects:

[0096] 1. Basic information of the image: including file name, file size, image format (such as JPEG, PNG, GIF, etc.), image dimensions (width and height, resolution), etc.

[0097] 2. Shooting information: If the image is a photograph, its XML information may include data such as the camera model, shutter speed, aperture, ISO value, focal length, etc.

[0098] 3. Copyright information: the author of the image, copyright owner, copyright statement, usage permissions and restrictions, etc.

[0099] 4. Keywords and description: Keywords or tags used to describe the content of the image, as well as a detailed description of the image.

[0100] 5. Geographic location information: If available, the location where the image was taken, which may include latitude and longitude coordinates, city, landmarks, etc.

[0101] 6. Time information: the creation time, modification time, shooting time, etc. of the picture.

[0102] 7. Color information: the color mode of the image (such as RGB, CMYK, etc.) and color profile.

[0103] 8. Structural information: If it is a complex image consisting of multiple layers or parts, the XML information may describe the structure and relationship of these parts.

[0104] 9. Metadata: Other metadata related to the image, such as device information, software information, editing history, etc.

[0105] 10. Access and usage information: image storage path, URL, access permissions, etc.

[0106] The XML information training of the image can further improve the recognition accuracy of the multimodal large model in different scenarios:

[0107] The different scenarios may include: organizing and managing image library scenarios, automated processing scenarios, copyright and authorization management scenarios, and content identification and indexing scenarios.

[0108] For organizing and managing picture libraries: Through XML information, you can quickly search, classify and retrieve pictures.

[0109] For automated processing scenarios: In large-scale image processing or editing tasks, XML information can guide automated scripts or programs to perform specific operations.

[0110] For copyright and authorization management scenarios: ensure that the use of images complies with the regulations of the copyright owner.

[0111] For content identification and indexing scenarios: Keywords and descriptions can make it easier to identify and index image content.

[0112] In actual operation, XML information can also be stored with the image file, or added to the image file through metadata editing tools. If you need to view or edit the XML information of the image, you can use a text editor that supports XML editing or a dedicated metadata editing tool.

[0113] in, Figure 7 The open source large model can represent a Meta Large Language Model (MLLM). The Meta Large Language Model is specifically an open source high-level language model such as the OPT model. The OPT (Open Pretrained Transformer) model is a large, open source pre-trained transformer model.

[0114] Optionally, the open source large model may also include a multimodal large language model (Multimodal LargeLanguage Model) for processing multiple types of input data, such as text, images, and sound, in order to better understand and generate natural language, a modular large language model (Modular Large Language Model) with a modular design that allows flexible addition or replacement of components to adapt to different tasks, and a multitask large language model (Multitask Large Language Model) for performing multiple tasks simultaneously, such as translation, summarization, question and answer, etc.

[0115] Generally speaking, large language models are based on deep learning techniques such as the Transformer architecture to process natural language data. These models learn the patterns and structures of language by pre-training on large amounts of text data, which can then be used for a variety of downstream tasks. Pre-training usually involves unsupervised learning or self-supervised learning, such as the Masked Language Model (MLM) task, in which the model learns to predict randomly masked words in the text.

[0116] based on Figure 8 The model prediction process shown in the figure, this embodiment uses an open source large language model to fine-tune the model, which greatly reduces the development costs of researchers and developers, without having to train their own models from scratch, thereby improving development efficiency. This promotes research and application development in the field of natural language processing and lowers the threshold for developing high-quality natural language processing applications.

[0117] Furthermore, in an exemplary embodiment, after the aligned second image feature vector and the text feature vector are input into the visual language model to obtain the interface description information output by the visual language model, information of the elements of the interface image and the association relationship between multiple different elements can be obtained from the interface description information; the association relationship is corrected according to the correction instruction to obtain the corrected interface description information.

[0118] The above correction instructions may be provided manually, or may be automatically triggered according to default settings.

[0119] Exemplary, for example, CLIP (Contrastive Language-Image Pre-training) model can also be used to further correct the UI element description of the description information according to each UI element coordinate reflected by the description information. The CLIP model can improve the accuracy of the description by contrastive learning and image-text matching. This embodiment combines the results of OCR and CLIP models to optimize and improve the quality of annotations by a multi-strategy fusion approach. Among them, the CLIP model is a pre-training model for computer vision and natural language processing tasks based on contrastive learning. By simultaneously learning the feature representations of images and texts, a cross-modal association between vision and language is achieved, and image and text data can be understood and processed simultaneously, completing multiple cross-modal tasks such as image-text matching, visual question answering, and image annotation. At the same time, the CLIP model adopts a contrastive learning method to learn their feature representations by comparing the similarities between different image and text pairs. This learning method enables the model to capture the subtle differences between images and texts.

[0120] Furthermore, in this embodiment, a hierarchical structure may be used to organize information of UI elements, such as organizing information including names, coordinates, function descriptions, and descriptions of associations between elements, to generate description information of a hierarchical structure that is better understood and used.

[0121] Furthermore, in an exemplary embodiment, before executing the above-mentioned step S206 of using the multimodal large language model to obtain the interface description information corresponding to the interface information displayed in real time by the terminal device, the multimodal large language model can be obtained by the following steps: using the historical interface image corresponding to the historical interface information of the terminal device and the historical text description information of the historical interface image as input samples, and using the historical interface description information of the historical interface image as output samples to train the original multimodal large language model to obtain the multimodal large language model, wherein multiple different elements in the corresponding interface have been marked on the historical interface image.

[0122] It should be noted that when marking different elements, annotators can do the marking, but this requires the annotators to accurately understand human instructions and select the correct UI elements for operation, which may vary due to individual differences. In order to improve the annotation accuracy of the annotators, the annotators can be trained to ensure that they can accurately understand human instructions and select the correct UI elements.

[0123] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. In order to better understand the execution method of the above-mentioned task instructions, the above-mentioned process is described below in conjunction with the embodiments, but it is not intended to limit the technical solution of the embodiments of the present invention, specifically:

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0125] In the present embodiment, a kind of execution device of task instruction is also provided, and this device is used to realize the above-mentioned embodiment and preferred implementation mode, and what has been described is not repeated here. As used below, the term "module" can realize the combination of software and / or hardware of predetermined function. Although the equipment described in the following embodiment is preferably realized by software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0126] Fig. 9 is a structural block diagram of a device for executing a task instruction according to an embodiment of the present invention, the device comprising:

[0127] A device proxy center 90, a data transmission channel is established between the device proxy center and the terminal device of the target object, and the device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in; the device proxy center is used to execute the execution method of the above-mentioned task instructions.

[0128] Further, in an exemplary embodiment, the device proxy center is at least used to execute: obtaining the task instructions of the target object for the application software on the terminal device; using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instructions belong from the task operation manual; using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device; inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instructions and the historical decision information corresponding to the first set of operation instructions into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model; sending the first execution instruction to the terminal device so that the terminal device executes the first execution instruction and obtains a first execution result.

[0129] According to the present invention, a device proxy center based on a data transmission channel established between the device proxy center and the terminal device of the target object obtains the task instructions of the target object for the application software on the terminal device; a retrieval enhancement model of the device proxy center is used to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from a task operation manual; a multimodal large language model of the device proxy center is used to output interface description information corresponding to the interface information displayed in real time by the terminal device; the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction and the historical decision information corresponding to the first operation instruction set are input into the intelligent large language model of the device proxy center to obtain a first execution instruction output by the intelligent large language model; the first execution instruction is sent to the terminal device so that the terminal device executes the first execution instruction and obtains a first execution result. By adopting the above technical scheme, a retrieval enhancement model that can retrieve the task operation manual is designed, a multimodal large language model that can generate interface description information in an integrated manner, and an intelligent large language model that generates execution instructions according to the input interface information, interface operation information, system prompt information and historical decision information. After accurately understanding the user instruction, the retrieval enhancement model is used to obtain the operation instruction set corresponding to the task instruction, the multimodal large language model is used to identify the interface information to generate the interface description information, and then the intelligent large language model is used to generate the execution instruction for the interface description information and other relevant information (i.e., interface information, system prompt information and historical decision information). This scheme, even if there is no execution instruction that can be directly executed from the task operation manual, can also search for similar operations from the task operation manual and submit them to the intelligent large language model, so that the intelligent large language model can make subsequent operations by referring to historical successful cases, reduce the reasoning and thinking time required for each step of the model re-prediction, and improve the execution efficiency and execution success rate of the task instruction. Therefore, the technical scheme of the present invention solves the technical problem of how to efficiently and accurately execute task instructions in the case of non-manual operation of terminal devices in the related technology, thereby improving the execution efficiency and execution accuracy of task instructions.

[0130] Further, in an exemplary embodiment, the device agent center is also used to use the task instruction to match the operation instruction set in the task operation manual; and the use of the retrieval enhancement model to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual, including: in response to a failure to match the task instruction with the operation instruction set in the task operation manual, using the retrieval enhancement model to obtain a first operation instruction set corresponding to the task type to which the task instruction belongs from the task operation manual.

[0131] Furthermore, in an exemplary embodiment, the device agent center is also used to, in response to a successful match between the task instruction and the operation instruction set in the task operation manual, use the retrieval enhancement model to obtain a second operation instruction set corresponding to the task instruction from the task operation manual; when it is determined that the instruction type of the task instruction is a single-step execution instruction, determine the operation instruction in the second operation instruction set that matches the task instruction as a second execution instruction, send the second execution instruction to the terminal device, and obtain a second execution result obtained after the terminal device executes the second execution instruction; when it is determined that the instruction type of the task instruction is a batch execution instruction, send all operation instructions in the second operation instruction set that match the task instruction to the terminal device, and obtain a third execution result after the terminal device executes all the operation instructions in a linked execution order, wherein the linked execution order represents the order in which all the operation instructions are linked to be executed.

[0132] Furthermore, in an exemplary embodiment, the device agent center is also used to generate basic operation data based on historical execution results obtained after the terminal device executes historical execution instructions; establish the retrieval enhancement model based on the basic operation data; and obtain successful execution instructions that are successfully executed from the historical execution instructions, and generate the task operation manual based on the successful execution instructions and the execution results corresponding to the successful execution instructions.

[0133] Furthermore, in an exemplary embodiment, the device agent center is also used to obtain interface operation information corresponding to the interface description information, including: displaying the interface image displayed in real time by the terminal device to the operation object through a web page display tool, so as to generate a target interface image with element marking information based on the annotation operation of the operation object on the interface image; and generating the interface operation information according to the target interface image and the operation information indicated by the target interface image.

[0134] Furthermore, in an exemplary embodiment, the multimodal large language model includes at least: a visual transformation module, a multi-layer perception module, and a visual language model. The device agent center is also used to perform text recognition on the interface image displayed in real time by the terminal device to obtain text description information of the interface image; use the visual transformation module to encode the interface image to obtain a first image feature vector; use the multi-layer perception module to perform a vector adjustment operation on the first image feature vector to obtain a second image feature vector, wherein the vector adjustment operation includes at least one of the following: a linear transformation operation, a bias operation, an activation operation, a propagation operation, and an update operation; align the second image feature vector with the text feature vector corresponding to the text description information, and input the aligned second image feature vector and the text feature vector into the visual language model to obtain the interface description information output by the visual language model.

[0135] Furthermore, in an exemplary embodiment, the device agent center is also used to obtain information on elements of the interface image and association relationships between multiple different elements from the interface description information after inputting the aligned second image feature vector and the text feature vector into the visual language model to obtain interface description information output by the visual language model; and to correct the association relationship according to a correction instruction to obtain corrected interface description information.

[0136] Furthermore, in an exemplary embodiment, the device agent center is also used to train the original multimodal large language model with a historical interface image corresponding to the historical interface information of the terminal device and historical text description information of the historical interface image as input samples, and with the historical interface description information of the historical interface image as output samples, before using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device, so as to obtain the multimodal large language model, wherein multiple different elements in the corresponding interface have been marked on the historical interface image.

[0137] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0138] S1, obtaining a task instruction of the target object for the application software on the terminal device;

[0139] S2, using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual;

[0140] S3, using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device;

[0141] S4, inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction, and the historical decision information corresponding to the first operation instruction set into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model;

[0142] S5: Send the first execution instruction to the terminal device, so that the terminal device executes the first execution instruction and obtains a first execution result.

[0143] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0144] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0145] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0146] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0147] S1, obtaining a task instruction of the target object for the application software on the terminal device;

[0148] S2, using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual;

[0149] S3, using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device;

[0150] S4, inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction, and the historical decision information corresponding to the first operation instruction set into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model;

[0151] S5: Send the first execution instruction to the terminal device, so that the terminal device executes the first execution instruction and obtains a first execution result.

[0152] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0153] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0154] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for executing a task instruction, It is characterized in that Applied to a device proxy center, a data transmission channel is established between the device proxy center and the terminal device of the target object, and the device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in. The method includes: Obtaining a task instruction from the target object for the application software on the terminal device; Using the retrieval enhancement model, obtaining a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual; Using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device; Inputting the interface information, the interface operation information corresponding to the interface description information, the system prompt information generated based on the task instruction, and the historical decision information corresponding to the first operation instruction set into the intelligent large language model to obtain a first execution instruction output by the intelligent large language model; The first execution instruction is sent to the terminal device, so that the terminal device executes the first execution instruction and obtains a first execution result.

2. The method for executing a task instruction according to claim 1, It is characterized in that The method further comprises: Using the task instruction to match the set of operation instructions in the task operation manual; and The step of using the retrieval enhancement model to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual includes: In response to a failure in matching the task instruction with the set of operation instructions in the task operation manual, the retrieval enhancement model is used to obtain a first set of operation instructions corresponding to the task type to which the task instruction belongs from the task operation manual.

3. The method for executing a task instruction according to claim 2, It is characterized in that The method further comprises: In response to the task instruction successfully matching the set of operation instructions in the task operation manual, using the retrieval enhancement model to obtain a second set of operation instructions corresponding to the task instruction from the task operation manual; In the case where it is determined that the instruction type of the task instruction is a single-step execution instruction, determining an operation instruction in the second operation instruction set that matches the task instruction as a second execution instruction, sending the second execution instruction to the terminal device, and obtaining a second execution result obtained after the terminal device executes the second execution instruction; When it is determined that the instruction type of the task instruction is a batch execution instruction, all operation instructions in the second operation instruction set that match the task instruction are sent to the terminal device, and a third execution result is obtained after the terminal device executes all the operation instructions in a linked execution order, wherein the linked execution order represents the order in which all the operation instructions are executed in a linked manner.

4. The method for executing a task instruction according to claim 1, It is characterized in that The method further comprises: Generate basic operation data according to the historical execution results obtained after the terminal device executes the historical execution instructions; Establishing the retrieval enhancement model based on the basic operation data; And, a successfully executed successful execution instruction is obtained from the historical execution instructions, and the task operation manual is generated according to the successfully executed instruction and the execution result corresponding to the successfully executed instruction.

5. The method for executing a task instruction according to claim 1, It is characterized in that The method further comprises: Obtaining interface operation information corresponding to the interface description information includes: The interface image displayed in real time by the terminal device is displayed to the operation object through a web page display tool, so as to generate a target interface image with element marking information based on the annotation operation of the operation object on the interface image; The interface operation information is generated according to the target interface image and the operation information indicated by the target interface image.

6. The method for executing a task instruction according to claim 1, It is characterized in that The multimodal large language model at least includes: a visual transformation module, a multi-layer perception module, and a visual language model. The multimodal large language model is used to output interface description information corresponding to the interface information displayed in real time by the terminal device, including: Performing text recognition on the interface image displayed in real time by the terminal device to obtain text description information of the interface image; Encode the interface image using the visual transformation module to obtain a first image feature vector; Using the multi-layer perception module to perform a vector adjustment operation on the first image feature vector to obtain a second image feature vector, wherein the vector adjustment operation includes at least one of the following: a linear transformation operation, a bias operation, an activation operation, a propagation operation, and an update operation; The second image feature vector is aligned with the text feature vector corresponding to the text description information, and the aligned second image feature vector and the text feature vector are input into the visual language model to obtain the interface description information output by the visual language model.

7. The method for executing a task instruction according to claim 6, It is characterized in that After inputting the aligned second image feature vector and the text feature vector into the visual language model to obtain the interface description information output by the visual language model, the method further includes: Acquire information of elements of the interface image and association relationships between a plurality of different elements from the interface description information; The association relationship is modified according to the modification instruction to obtain modified interface description information.

8. The method for executing a task instruction according to claim 6, It is characterized in that Before using the multimodal large language model to obtain interface description information corresponding to the interface information displayed in real time by the terminal device, the method further includes: The original multimodal large language model is trained using the historical interface image corresponding to the historical interface information of the terminal device and the historical text description information of the historical interface image as input samples, and the historical interface description information of the historical interface image is used as output samples to obtain the multimodal large language model, wherein multiple different elements in the corresponding interface have been marked on the historical interface image.

9. A device for executing a task instruction, It is characterized in that include: A device proxy center, wherein a data transmission channel is established between the device proxy center and the terminal device of the target object, and the device proxy center has at least a multimodal large language model, an intelligent large language model, and a retrieval enhancement model built in; the device proxy center is used to execute the method described in any one of claims 1 to 7 above.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, It is characterized in that The memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

12. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Citation Information

Cited By

  • Construction method, device and equipment of intelligent criminal investigation platform and storage medium

    CN119357339A

  • Variable environment visual language navigation method and system based on historical reflection

    CN119826835A

  • Smart phone automatic operation method and system based on sandbox and large language model

    CN120850278A

  • Closed-loop reasoning method, device and equipment based on multi-modal large model

    CN120996209A

  • Robot control method and device, equipment, medium and program product

    CN121200008A