Task processing method and device

Abstract

The invention discloses a task processing method and device, and relates to the technical field of man-machine interaction and artificial intelligence. The task processing method comprises the following steps: in response to an obtained to-be-executed target task, performing first operation through a virtual machine to determine a first operation to be executed currently by the target task; pre-executing the first operation through the virtual machine so as to perform a second operation after the pre-execution of the first operation is completed, the second operation being used for determining the next operation of the first operation; executing the first operation through a physical machine according to a mode parallel to the second operation, and synchronously displaying the operation process, so as to execute and display the next operation of the first operation after the execution and display of the first operation are completed; wherein the time consumed by the virtual machine for pre-executing one step of operation is less than the time consumed by the physical machine for executing the same step of operation and synchronously displaying the operation process.

Description

Technical Field

[0001] This application relates to the fields of human-computer interaction and artificial intelligence technology, and in particular to a task processing method and apparatus. Background Technology

[0002] Current mainstream GUI Agent (Graphical User Interface Agent) models have fixed interaction logic limitations during task execution and must follow a certain serial process. This results in users spending most of their time waiting for the GUI Agent to complete the calculation and not being able to intuitively perceive the specific working process of the GUI Agent during the calculation phase, which seriously affects the user's experience and trust in the GUI Agent. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A task processing method, comprising:

[0005] In response to obtaining a target task to be executed, a first operation is performed through a virtual machine to determine the first operation to be executed for the target task.

[0006] The first operation is pre-executed by the virtual machine so that a second operation is performed after the pre-execution of the first operation is completed, and the second operation is used to determine the next operation of the first operation;

[0007] The first operation is executed by a physical machine in parallel with the second operation, and the operation process is displayed synchronously, so that the next operation of the first operation is executed and displayed after the execution and display of the first operation is completed;

[0008] The time taken for the virtual machine to pre-execute one step of the operation is less than the time taken for the physical machine to execute the same step of the operation and synchronously display the operation process.

[0009] Optionally, the virtual machine is generated by image processing of the physical machine, and the system environment corresponding to the virtual machine and the system environment corresponding to the physical machine meet the consistency condition.

[0010] Optionally, a first computation is performed using a virtual machine to determine the first operation to be performed on the target task, including:

[0011] Based on the task description information of the target task and the current user interface screenshot, a first calculation is performed to determine the first operation information required for the first operation to be performed by the target task.

[0012] Optionally, a first calculation is performed based on the task description information of the target task and the current user interface screenshot to determine the first operation information required for the first operation to be performed by the target task, including:

[0013] The target model is invoked through the virtual machine;

[0014] Input the task description information of the target task and the current user interface screenshot into the target model to obtain at least some information from the operation type, the target element, and the operation result of the first operation to be performed by the target task, calculated by the target model based on the input information.

[0015] Optionally, before performing the first operation via a physical machine, the method further includes:

[0016] The first operation information is sent to the physical machine so that the physical machine can perform the first operation based on the first operation information.

[0017] Optionally, the first operation can be performed on a physical machine in parallel with the second operation, and the operation process can be displayed synchronously, including one of the following:

[0018] After the first operation is performed through the virtual machine, the first operation is executed through the physical machine in parallel with the virtual machine's pre-execution of the first operation and the performance of the second operation, and the operation process is displayed synchronously.

[0019] After the first operation is pre-executed through the virtual machine, the first operation is executed through the physical machine in parallel with the second operation performed by the virtual machine, and the operation process is displayed synchronously.

[0020] Optionally, if the physical machine executes the first operation after the virtual machine has pre-executed the first operation, the method further includes:

[0021] If the virtual machine's pre-execution result of the first operation indicates that the operation process of the first operation is abnormal, the first operation information required for the first operation is corrected so that the physical machine executes the first operation based on the corrected first operation information.

[0022] Optionally, the task processing method further includes:

[0023] After the pre-execution of the first operation is completed, a second operation is performed through the virtual machine to determine the second operation to be performed next to the first operation.

[0024] The second operation is pre-executed by the virtual machine so that a third operation is performed after the pre-execution of the second operation is completed, the third operation being used to determine the next operation of the second operation;

[0025] The second operation is executed by the physical machine in parallel with the third operation, and the operation process is displayed synchronously. After the execution and display of the second operation are completed, the next operation of the second operation is executed and displayed, until the virtual machine completes the pre-execution of each operation included in the target task and the physical machine completes the execution and display of each operation included in the target task.

[0026] Optionally, the task processing method further includes:

[0027] In response to receiving an interrupt instruction, the current processing of the virtual machine and the physical machine is stopped to exit the target task, or the processing of the target task is corrected in response to receiving a correction instruction after the current processing is stopped.

[0028] A task processing device, comprising:

[0029] The computation module is used to perform a first computation through a virtual machine in response to obtaining a target task to be executed, so as to determine the first operation to be executed by the target task.

[0030] A first processing module is configured to pre-execute the first operation through the virtual machine, so as to perform a second operation after the pre-execution of the first operation is completed, wherein the second operation is used to determine the next operation of the first operation;

[0031] The second processing module is used to execute the first operation through a physical machine in a manner parallel to the second operation, and to simultaneously display the operation process, so as to execute and display the next operation of the first operation after the execution and display of the first operation are completed;

[0032] The time taken for the virtual machine to pre-execute one step of the operation is less than the time taken for the physical machine to execute the same step of the operation and synchronously display the operation process.

[0033] An electronic device, comprising:

[0034] Memory is used to store computer instruction sets;

[0035] A processor is configured to implement any of the task processing methods provided above by executing a set of computer instructions in the memory.

[0036] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to perform any of the task processing methods provided above. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating one of the task processing methods provided in this application;

[0039] Figure 2 This is a schematic diagram illustrating one implementation method of the dual-machine asynchronous collaborative mechanism provided in this application;

[0040] Figure 3 This is a schematic diagram illustrating another implementation method of the dual-machine asynchronous collaborative mechanism provided in this application;

[0041] Figure 4 This is another flowchart illustrating the task processing method provided in this application;

[0042] Figure 5 This is another flowchart illustrating the task processing method provided in this application;

[0043] Figure 6 This is another flowchart illustrating the task processing method provided in this application;

[0044] Figure 7 This is a structural diagram of the task processing device provided in this application;

[0045] Figure 8 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] This application provides a task processing method and apparatus for eliminating or mitigating user waiting perception in a graphical user interface interaction scenario based on a GUI Agent, and enabling users to perceive the operation process of the device, thereby improving user experience and trust in the GUI Agent.

[0048] The task processing method can be applied to electronic devices in a variety of general or special computing environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0049] See Figure 1 The flowchart shown illustrates the task processing method provided in this application embodiment, which may include the following steps 101 to 103, which are described in detail below.

[0050] Step 101: In response to obtaining the target task to be executed, perform a first calculation through the virtual machine to determine the first operation to be executed for the target task.

[0051] The solution provided in this application is mainly applied to graphical user interface interaction scenarios based on GUI Agents. It uses a GUI Agent to understand the user's natural language commands and automatically execute graphical interface operations, thereby improving device interaction efficiency and simplifying complex task processes. By integrating technologies such as computer vision and natural language processing, the GUI Agent can accurately understand the dynamic UI (User Interface) and generate an operation sequence that matches the interface interaction task to execute the required operations.

[0052] Correspondingly, the target task can be a graphical user interface interaction task that the user inputs based on their needs and is to be executed by the GUI Agent. The GUI Agent can replace the user's operation to complete the automated processing of the required task, such as various intelligent office tasks based on mouse movement, clicking, input, dragging and dropping.

[0053] Optionally, the target task can be in natural language. In practical applications, users can input the target task into electronic devices (such as GUI agents in electronic devices) by means of text input or voice, but not limited to, as needed. Examples include editing a paragraph of text or an image in a Word document, automatically importing Excel data into an ERP (Enterprise Resource Planning) system and generating charts, and automatically completing tax declarations after reading data from financial software.

[0054] Current mainstream GUI agents have fixed interaction logic limitations during task execution, requiring a sequential "calculation-operation" process. This means that for a task, the current operation to be performed must first be calculated, such as calculating the necessary information for that operation, before executing the operation based on the calculation results. Furthermore, in practice, the "calculation" process typically takes much longer than the "operation" process. This results in users spending most of their time waiting for the GUI agent to complete calculations (while the subsequent operation is completed quickly), without being able to intuitively perceive the specific work being done by the GUI agent during the calculation phase. This severely impacts user experience and trust in the GUI agent.

[0055] To address the aforementioned issues, this application proposes a GUI intelligent agent interaction framework based on dual-machine asynchronous collaboration to break the traditional serial dependency relationship of "computation-operation".

[0056] The dual-machine setup consists of a physical machine and a virtual machine. The physical machine is the actual physical machine running on the electronic device, while the virtual machine is generated by mirroring the physical machine running on the electronic device.

[0057] The system environment corresponding to the virtual machine and the system environment corresponding to the physical machine meet the consistency conditions.

[0058] The consistency condition may refer to the system environment corresponding to the virtual machine being the same as the system environment corresponding to the physical machine, or the similarity between the system environment corresponding to the virtual machine and the system environment corresponding to the physical machine reaching a threshold, so as to be able to highly simulate the key characteristics of the original physical machine environment in the virtual machine.

[0059] The virtual machine can be started when the electronic device is turned on and a graphical user interface interaction task initiated by the user to be executed through the GUI Agent is detected for the first time, or it can be started temporarily every time a graphical user interface interaction task initiated by the user to be executed based on the GUI Agent is detected, or it can be started based on the user's mode switching operation. There are no restrictions on this, and it can be determined according to the actual application.

[0060] In some descriptions, embodiments of this application refer to graphical user interface interaction tasks that need to be performed through a GUI Agent as "tasks".

[0061] A task requires at least one operation to complete. An "operation" refers to an atomic operation that can be recognized and executed by a computer, including but not limited to mouse / keyboard or touch operations that can be performed by a GUI Agent on behalf of the user, such as mouse movement, hovering, clicking, and information input. For example, suppose the target task is "to set the font of the third paragraph in Word document X to xxx," and suppose Word document X is currently open and displayed. This target task can be broken down into multiple operations such as "selecting the text paragraph," "moving the mouse," "hovering," and "clicking."

[0062] In response to obtaining the target task to be executed, in this step, the GUI Agent can obtain the task description information and the current user interface screenshot of the target task through the virtual machine, and perform a first calculation based on the task description information and the current user interface screenshot, so as to determine the first operation information required for the first operation to be executed by the target task.

[0063] In practice, optionally, the virtual machine can call the target model and input the task description information of the target task and the current user interface screenshot into the target model, so that the target model can perform a first calculation based on the input information, thereby obtaining the first operation information required for the first operation to be performed by the target task.

[0064] The task description information of the target task may include, but is not limited to, one or more of the following: the task object required by the target task (such as an element in an application), the task requirements for the target object, and the task objective. For example, "Set the font of the third paragraph in Word document X to xxx, the background color to xxx, and the font size to xxx".

[0065] The first operation information may include, but is not limited to, at least some of the following: the operation type of the first operation, the target element, and the operation result (i.e., the expected operation result). For example, if the first operation is "selecting a text paragraph", then the operation type is "selection" and the target element is "a text paragraph"; if the first operation is "mouse movement operation", then the operation type is "movement", the target element is "mouse cursor", and the operation result is "the mouse cursor moves from the starting position to the target position".

[0066] The target model can be a large or small AI model. The large model can be a general-purpose large model, such as a general-purpose large language model (LLM), a visual large model, a multimodal large model, etc., or it can be an expert large model obtained by fine-tuning a general-purpose large model based on specific needs. This application does not limit this. The large model learns the features and patterns of the input information by training on a large amount of diverse data. It typically has hundreds of millions to hundreds of billions of model parameters (such as the weights corresponding to the model network layers, used to control model behavior), and can capture complex relationships and patterns in the input information for accurate and efficient inference / computation.

[0067] Step 102: Pre-execute the first operation through the virtual machine to perform a second operation after the pre-execution of the first operation is completed. The second operation is used to determine the next operation of the first operation.

[0068] After determining the first operation to be executed for the target task, the first operation can be pre-executed by the virtual machine based on the first operation information required by the first operation, such as the operation type, the target element, and the operation result. After the pre-execution of the first operation is completed, the virtual machine can continue to perform the second operation to determine the next operation of the first operation.

[0069] In this application embodiment, "pre-execution" essentially refers to rehearsing the current step operation of the target task to be executed in the system environment of the virtual machine, so as to know the operation result of the current step operation in advance. For example, by pre-executing the mouse movement operation, a screenshot of the interface after the mouse moves from one position to another on the document interface can be obtained, so as to serve as one of the bases for determining the next operation (such as performing a second calculation).

[0070] The execution result after the first pre-execution is not actually fed back to the user or presented in the graphical user interface, so the user will not be aware of the pre-execution result.

[0071] Step 103: Execute the first operation through a physical machine in parallel with the second operation, and display the operation process synchronously, so that after the execution and display of the first operation are completed, the next operation of the first operation is executed and displayed.

[0072] Before executing the first operation on the physical machine, the virtual machine can first synchronize the determined first operation information to the physical machine. Based on this, the physical machine can execute the first operation in parallel with the second operation, according to the received first operation information such as the operation type, the target element, and the corresponding operation result, and simultaneously display the operation process of the first operation on the graphical user interface.

[0073] The time taken for the virtual machine to pre-execute one operation is less than the time taken for the physical machine to execute the same operation and synchronously display the operation process.

[0074] In other words, compared to the physical machine's execution and synchronous display of a single operation (such as clicking or inputting), the virtual machine's pre-execution of that operation is a rapid rehearsal process, taking less time. This allows for quick and timely knowledge of the operation's result in advance, facilitating the use of the result as one of the criteria for determining the next operation (such as performing a second calculation). In contrast, the physical machine executes the operation in a "slow-motion" manner, taking longer, and then visualizes the operation on the graphical user interface. For example, the millisecond-level pre-execution process of mouse movement, hovering, and click feedback on the virtual machine is extended to a second-level execution and visualization process on the physical machine.

[0075] In summary, through the aforementioned GUI intelligent agent interaction framework based on dual-machine asynchronous collaboration, the solution of this application embodiment can achieve parallel processing of computation and operation (such as the parallel execution of the second operation by the virtual machine and the execution and display of the first operation by the physical machine), breaking the traditional serial dependency relationship of "computation-operation". Furthermore, by rapidly pre-executing one step of the operation in the virtual machine and executing and displaying that step of the operation in the physical machine according to the "slow-motion" display logic, sufficient time can be reserved for the virtual machine to execute the second operation in a timely manner after completing the pre-execution of the current first operation and before the physical machine completes the execution of the first operation and needs to execute the next operation. This allows the physical machine to determine the next operation of the first operation without having to wait for the computation process of the second operation when it needs to execute the next operation after completing the first operation, or at least shortens the waiting time, thus eliminating or alleviating the user's perception of waiting.

[0076] In addition, this application performs the current first operation on the physical machine according to the "slow motion" display logic and visualizes the operation process on the graphical user interface. This allows users to observe the operation process of the device in real time during the model calculation, transforming the blank time of "simply waiting for calculation" into effective interactive time of "observing the progress of operation", thereby significantly reducing the user's perception of waiting for calculation. Moreover, the operation process of the device is not too fast, so that users do not feel that it is fleeting and cannot clearly understand the operation process, thus enhancing the user's immersive interaction with the GUI Agent.

[0077] In an optional embodiment, step 103 of the task processing method provided in this application, which involves executing the first operation on a physical machine in parallel with the second operation and simultaneously displaying the operation process, can be implemented as any one of the following embodiments:

[0078] 11) After the first operation is performed through the virtual machine, the first operation is performed through the physical machine in parallel with the virtual machine's pre-execution of the first operation and the performance of the second operation, and the operation process is displayed synchronously.

[0079] See Figure 2 The illustrated embodiment 11) is a schematic diagram of parallel processing of computation and operation achieved through a dual-machine asynchronous collaborative mechanism of physical machine and virtual machine. In this embodiment 11), after the virtual machine completes the first calculation and obtains the first operation information required for the first operation to be executed by the target task, it synchronizes the first operation information to the physical machine. Then, dual-machine parallel processing is started, in which the virtual machine pre-executes the first operation based on the first operation information, such as pre-executing mouse movement, hovering, or clicking operations according to the operation type and target element in the first operation information. After completing the pre-execution of the first operation, it automatically captures the corresponding interface (background virtualization interface) and inputs the obtained interface screenshot as the input information for the next calculation, i.e., the second operation, into the target model. The target model performs the second operation based on the interface screenshot after the pre-execution of the first operation and the task description information of the target task. The second operation determines the operation information required for the next operation of the first operation, so that the next operation can be further pre-executed in the virtual machine based on the operation information required for the next operation, thereby continuously advancing the "computation-operation (pre-execution)-screenshot" cycle of the virtual machine task, realizing high-speed and imperceptible task pre-execution.

[0080] At the same time, the physical machine performs the first operation based on the first operation information and displays the operation process synchronously, in a manner that is parallel to the virtual machine's pre-execution of the first operation and the performance of the second operation.

[0081] like Figure 2 As shown, the time taken for the physical machine to execute each operation and simultaneously display the process is longer than that taken for the virtual machine to execute the same operation. Based on this time consumption characteristic and combined with the asynchronous collaborative mechanism between the two machines, the physical machine does not need to wait for the computation process, or at least the waiting time is shortened. Furthermore, by executing operations on the physical machine according to the "slow-motion" display logic and visually displaying the operation process on the graphical user interface, users can observe the device's operation process in real time during the model computation, transforming the originally blank time of "simply waiting for computation" into effective interactive time of "observing the progress of the operation".

[0082] 12) After the first operation is pre-executed through the virtual machine, the first operation is executed through the physical machine in parallel with the second operation performed by the virtual machine, and the operation process is displayed synchronously.

[0083] See Figure 3 The schematic diagram of embodiment 12) shown is a dual-machine asynchronous collaborative mechanism of physical machine and virtual machine to realize parallel processing of computing and operation. Compared with the previous embodiment, the starting time node of parallel processing of physical machine and virtual machine is different. Specifically, after the virtual machine pre-executes the first operation, the virtual machine and physical machine start parallel processing.

[0084] like Figure 3 As shown in embodiment 12), after the virtual machine completes the first calculation and obtains the first operation information required for the first operation to be executed by the target task, it can also synchronize the first operation information to the physical machine. At the same time, the virtual machine pre-executes the first operation based on the first operation information, such as pre-executing mouse movement, hovering, or clicking operations.

[0085] After the pre-execution of the first operation is completed, parallel asynchronous processing between the two machines is initiated. In this parallel asynchronous processing, the virtual machine automatically captures the interface (background virtualization interface) formed after the pre-execution of the first operation. This screenshot is used as input information for the next computation, the second operation, and is fed into the target model. The target model performs the second operation based on the screenshot and the task description information of the target task. This second operation determines the operation information required for the next operation after the first operation, allowing the virtual machine to pre-execute the next operation based on this information. Simultaneously, the physical machine executes the first operation according to a "slow-motion" display logic based on the first operation information synchronized from the virtual machine, and synchronously displays the operation process.

[0086] In this implementation, the time taken for the physical machine to execute each operation and simultaneously display the operation process is also longer than the time taken for the virtual machine to execute the same operation. Based on this time consumption characteristic and the asynchronous collaborative mechanism between the two machines, this implementation 12) also allows the physical machine to avoid waiting for the calculation process or at least shorten the waiting time. Furthermore, it allows users to observe the device's operation process in real time during model calculation, transforming the original blank time of "simply waiting for calculation" into effective interactive time of "observing the progress of the operation," thereby significantly reducing the user's perception of waiting for calculation and enhancing the immersive experience of the user's interaction with the GUI Agent.

[0087] Optionally, we can also have an implementation method 13), which differs from the previous implementation method 12) in that the starting time point for parallel processing of the physical machine and the virtual machine is different. Specifically, in this implementation method 13), the virtual machine and the physical machine begin parallel processing after the virtual machine takes a screenshot. Other contents are similar to those in the above implementation method 12), and will not be repeated here.

[0088] In an alternative embodiment, see Figure 4 The flowchart shown illustrates that if the physical machine executes the first operation after the virtual machine pre-executes the first operation, the task processing method provided in this application may further include the following processing between step 102 and step 103:

[0089] Step 401: If the virtual machine's pre-execution result of the first operation indicates that the operation process of the first operation is abnormal, correct the first operation information required for the first operation so that the physical machine executes the first operation based on the corrected first operation information.

[0090] Optionally, the abnormality of the operation process of the first operation can be determined by comparing the pre-execution result of the first operation with the expected operation result of the first operation. More specifically, after the virtual machine completes the pre-execution of the first operation, the pre-execution result of the first operation can be obtained. Then, the pre-execution result is compared with the expected operation result in the first operation information. If the two do not satisfy the consistency relationship (such as the two are different), it indicates that the operation process of the first operation is abnormal. Otherwise, the operation process of the first operation is not abnormal.

[0091] For example, suppose the first operation is "open file X". When the target model calculates the first operation information required for the first operation, it determines the operation type of the first operation as "click", the target element as "the icon of file X", and the expected operation result as "file X is opened". When the virtual machine pre-executes the first operation according to the first operation information, it will click the icon of file X. Obviously, this operation will not get the expected operation result of "file X is opened", so it can be determined that the operation process of the first operation is abnormal.

[0092] In the event of an anomaly during the first operation, optionally, the first operation information required for the first operation can be corrected by combining the abnormal operation result and the expected operation result of the first operation. For example, in the example above, the operation type in the first operation information can be corrected to "double-click" so that file X can be opened by double-clicking the file X icon. In practical applications, the correction of the first operation information required for the first operation can be achieved through AI models such as large models, but is not limited to.

[0093] After correcting the first operation information required for the first operation, the virtual machine synchronizes the corrected first operation information to the physical machine, so that the physical machine can execute the first operation based on the corrected first operation information, thereby ensuring that the operation and synchronization display process of the first operation on the physical machine is accurate.

[0094] This embodiment corrects the first operation information required for the first operation in the event of an anomaly during the pre-execution process of the first operation in the virtual machine, and enables the physical machine to execute the first operation based on the corrected first operation information. This ensures that the execution and synchronous display process of the first operation by the physical machine is accurate and error-free, thereby improving the accuracy of graphical user interface interaction based on the GUI Agent, and consequently enhancing the user's experience and trust in the GUI Agent.

[0095] In an alternative embodiment, see Figure 5 The flowchart shown in this application illustrates that the task processing method provided may further include the following processes:

[0096] Step 501: After completing the pre-execution of the first operation, a second operation is performed through the virtual machine to determine the second operation to be performed next in the first operation.

[0097] After the virtual machine pre-executes the first operation based on the first operation information, such as pre-executing the first operation like mouse movement, hovering, or clicking, it can automatically obtain a screenshot of the interface after the first operation is pre-executed. Then, based on the task description information of the target task and the screenshot of the interface after the first operation is pre-executed, it can perform a second operation to determine the second operation information, such as the operation type required for the second operation, the target element, and the operation result (expected operation result).

[0098] In practical applications, the virtual machine can also use the target model to perform the second operation. Specifically, the screenshot of the interface after the first operation is executed is input into the target model, and the target model performs the second operation based on the screenshot of the interface after the first operation is executed and the task description information of the target task, so as to determine the second operation information required for the second operation through the second operation.

[0099] Step 502: Pre-execute the second operation through the virtual machine to perform a third operation after the pre-execution of the second operation is completed. The third operation is used to determine the next operation of the second operation.

[0100] The virtual machine can pre-execute the second operation based on the second operation information. Specifically, it can perform the corresponding operation on the indicated target element according to the operation type indicated in the second operation information to obtain the expected operation result. For example, it can change the position of the mouse cursor according to the indicated operation type "move / slide" to move the mouse cursor from the starting position to the expected position.

[0101] After the virtual machine completes the pre-execution of the second operation, it automatically captures the interface (background virtualization interface) formed after the pre-execution of the second operation, and performs the third calculation based on the task description information of the target task and the screenshot of the interface after the pre-execution of the second operation to determine the third operation information required for the third operation. This continuously promotes the "calculation-operation-screenshot" cycle of the virtual machine task, realizing high-speed and imperceptible task pre-execution.

[0102] Step 503: The second operation is executed by the physical machine in parallel with the third operation, and the operation process is displayed synchronously. After the execution and display of the second operation are completed, the next operation of the second operation is executed and displayed, until the virtual machine completes the pre-execution of each operation included in the target task and the physical machine completes the execution and display of each operation included in the target task.

[0103] After obtaining the second operation information required for the second operation through the second computation, the virtual machine synchronizes the second operation information to the physical machine. Based on this, the physical machine executes the second operation in parallel with the third computation, using the second operation information.

[0104] Similarly to the implementation method of executing the first operation and synchronously displaying the operation process on a physical machine, in this step, after the second operation is performed through a virtual machine, the second operation can be executed through a physical machine in parallel with the virtual machine pre-executing the second operation and performing the third operation, and the operation process can be synchronously displayed.

[0105] However, this is not the only possibility. In other embodiments, after the second operation is pre-executed by the virtual machine, the second operation can be executed by the physical machine in parallel with the third operation performed by the virtual machine, and the operation process can be displayed synchronously. Furthermore, in this embodiment, if the pre-execution result of the second operation by the virtual machine indicates an anomaly in the operation process of the second operation, the second operation information required for the second operation can be corrected, so that the physical machine executes the second operation based on the corrected second operation information.

[0106] Easy to understand, see Figure 2 and Figure 3 In practical applications, for any step in the operation sequence required by the target task, both the virtual machine and the physical machine can achieve parallel and asynchronous execution of virtual machine computation and physical machine operation through one of the two implementation methods described above, until the virtual machine completes the pre-execution of each operation included in the target task and the physical machine completes the execution and display of each operation included in the target task.

[0107] This application's embodiments are based on a dual-machine asynchronous collaborative GUI intelligent agent interaction framework. By using the physical machine as the "perception end" for user interaction and the virtual machine as the "computation and rapid pre-execution end," the two environments synchronously enter the task flow, and computation and operation are executed in parallel on the two machines in the two environments. This breaks the serial dependency relationship of "computation-operation," avoids or alleviates the waiting time for the computation process in GUI agent-based graphical user interface interaction scenarios, and shortens the waiting time. At the same time, it allows users to observe the device's operation process in real time during model computation, transforming the blank time of "simply waiting for computation" into effective interaction time of "observing operation progress," thereby significantly reducing the user's perception of computation waiting and enhancing the user's immersive interaction with the GUI agent.

[0108] In an alternative embodiment, see Figure 6 The flowchart of the method described in this application shows that the task processing method may further include the following processing:

[0109] Step 601: In response to receiving an interrupt instruction, stop the current processing of the virtual machine and the physical machine to exit the target task, or, after stopping the current processing, correct the processing of the target task in response to receiving a correction instruction.

[0110] In the process of electronic devices processing target tasks based on virtual machines and physical machines, this embodiment supports users to issue interrupt commands as needed to interrupt the processing of target tasks.

[0111] The interruption command may include, but is not limited to, interruption commands issued by the user when they need to stop / exit the target task, or when the user finds that the displayed visual operation process does not match the expected operation process (which may be due to errors in the target model calculation or errors in the execution of the virtual machine / physical machine), or when they find that the displayed operation has an erroneous tendency (such as about to click on an incorrect file or enter an incorrect command), or when the user temporarily changes the task requirements.

[0112] Optionally, the interrupt instruction can be in natural language form, specifically, but not limited to, a natural language instruction that the user inputs through text input or voice to interrupt the task.

[0113] In response to receiving an interrupt instruction, the electronic device suspends the current processing of the virtual machine and the physical machine.

[0114] If no other instructions related to the target task are received within a preset time after receiving the interrupt instruction, the processing of the target task can be terminated directly.

[0115] If a correction instruction from the user is received within a preset time after the interruption instruction is received, the processing of the target task will be corrected based on the correction instruction.

[0116] The correction instructions may include, but are not limited to, instructions to adjust task requirements and / or instructions to adjust operations involved in task execution, depending on the actual application requirements. For example, the original task requirement "set the font of the third paragraph of Word document X to font A" may be adjusted to "set the font of the third paragraph of Word document X to font B and the background color to color C", and / or the operation "click an application icon" may be adjusted to "double-click an application icon to open the application", etc.

[0117] This embodiment allows users to exit the task or modify the task execution process as needed during the target task execution, thereby meeting the diverse needs of users in the process of graphical user interface interaction based on GUI Agent, and avoiding damage to user computer files, system settings, etc. caused by erroneous operations, improving the security and controllability of GUI Agent use, and further enhancing the user's experience and trust in GUI Agent.

[0118] Corresponding to the above-described task processing method, this application embodiment also provides a task processing device, the composition of which is as follows: Figure 7 As shown, it includes:

[0119] The computation module 701 is used to perform a first computation through a virtual machine in response to obtaining a target task to be executed, so as to determine the first operation to be executed by the target task.

[0120] The first processing module 702 is configured to pre-execute the first operation through the virtual machine, so as to perform a second operation after the pre-execution of the first operation is completed, wherein the second operation is used to determine the next operation of the first operation;

[0121] The second processing module 703 is used to execute the first operation through a physical machine in a manner parallel to the second operation, and to simultaneously display the operation process, so as to execute and display the next operation of the first operation after the execution and display of the first operation are completed;

[0122] The time taken for the virtual machine to pre-execute one step of the operation is less than the time taken for the physical machine to execute the same step of the operation and synchronously display the operation process.

[0123] In one optional implementation, the virtual machine is generated by image processing of the physical machine, and the system environment corresponding to the virtual machine and the system environment corresponding to the physical machine meet the consistency condition.

[0124] In an optional implementation, when the computing module 701 performs a first calculation via a virtual machine to determine the first operation to be performed on the target task, it is specifically used for:

[0125] Based on the task description information of the target task and the current user interface screenshot, a first calculation is performed to determine the first operation information required for the first operation to be performed by the target task.

[0126] In an optional embodiment, when the calculation module 701 performs a first calculation based on the task description information of the target task and the current user interface screenshot to determine the first operation information required for the first operation to be performed by the target task, it is specifically used for:

[0127] The target model is invoked through the virtual machine;

[0128] Input the task description information of the target task and the current user interface screenshot into the target model to obtain at least some information from the operation type, the target element, and the operation result of the first operation to be performed by the target task, calculated by the target model based on the input information.

[0129] In an optional embodiment, the above apparatus further includes a sending module for sending the first operation information to the physical machine before performing the first operation through the physical machine, so that the physical machine performs the first operation based on the first operation information.

[0130] In one alternative implementation, the second processing module 703 is specifically configured to perform one of the following:

[0131] After the first operation is performed through the virtual machine, the first operation is executed through the physical machine in parallel with the virtual machine's pre-execution of the first operation and the performance of the second operation, and the operation process is displayed synchronously.

[0132] After the first operation is pre-executed through the virtual machine, the first operation is executed through the physical machine in parallel with the second operation performed by the virtual machine, and the operation process is displayed synchronously.

[0133] In an optional embodiment, the above apparatus further includes a correction module, configured to: when the physical machine executes the first operation after the virtual machine has pre-executed the first operation, if the virtual machine's pre-execution result of the first operation indicates that the operation process of the first operation is abnormal, correct the first operation information required for the first operation, so that the physical machine executes the first operation based on the corrected first operation information.

[0134] In an optional embodiment, the calculation module 701 is further configured to perform a second calculation through the virtual machine after the pre-execution of the first operation is completed, so as to determine the second operation to be performed next to the first operation;

[0135] The first processing module 702 is further configured to pre-execute the second operation through the virtual machine, so as to perform a third operation after the pre-execution of the second operation is completed, the third operation being used to determine the next operation of the second operation;

[0136] The second processing module 703 is further configured to execute the second operation through the physical machine in a manner parallel to the third operation, and to simultaneously display the operation process, so as to execute and display the next operation of the second operation after the execution and display of the second operation is completed, until the virtual machine completes the pre-execution of each operation included in the target task and the physical machine completes the execution and display of each operation included in the target task.

[0137] In an optional embodiment, the above apparatus further includes an interrupt control module, configured to: stop the current processing of the virtual machine and the physical machine in response to receiving an interrupt instruction to exit the target task, or, after stopping the current processing, correct the processing of the target task in response to receiving a correction instruction.

[0138] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 8 As shown, it includes at least:

[0139] Memory 10 is used to store the computer instruction set;

[0140] Computer instruction sets can be implemented in the form of computer programs.

[0141] The processor 20 is configured to implement the task processing method provided in any of the above method embodiments by executing a set of computer instructions in the memory.

[0142] The processor 20 can be a central processing unit (CPU), a graphics processor, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.

[0143] Optionally, electronic devices may also include storage resources such as memory and cache.

[0144] Optionally, the electronic device may also include an image acquisition device.

[0145] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0146] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0147] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the task processing method provided in any of the above method embodiments.

[0148] In summary, the task processing method, apparatus, electronic device, and storage medium provided in this application have at least the following technical advantages compared to known technologies:

[0149] 1. Eliminate the perception of waiting and improve the user experience: By processing computation and operation in parallel, users can observe the computer's operation process in real time during the model computation, turning the blank time of "simply waiting for computation" into effective interactive time of "observing the progress of operation", thereby significantly reducing the user's perception of computation waiting and enhancing the user's immersive interaction with the GUI Agent.

[0150] 2. Enhance operational controllability and mitigate security risks: Since the operation process is simultaneously visualized, users can monitor in real time whether the operation meets expectations. When an error tendency is detected (such as clicking on an incorrect file or entering an incorrect command), an interruption command can be triggered manually or via voice to immediately stop the GUI Agent's operation or correct the operation process. This fundamentally avoids damage to user computer files, system settings, etc., caused by erroneous operations, thereby improving the security and controllability of the GUI Agent.

[0151] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0152] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0153] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0154] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A task processing method, comprising: In response to obtaining a target task to be executed, a first operation is performed through a virtual machine to determine the first operation to be executed for the target task. The first operation is pre-executed by the virtual machine so that a second operation is performed after the pre-execution of the first operation is completed, and the second operation is used to determine the next operation of the first operation; The first operation is executed by a physical machine in parallel with the second operation, and the operation process is displayed synchronously, so that the next operation of the first operation is executed and displayed after the execution and display of the first operation is completed; The time taken for the virtual machine to pre-execute one step of the operation is less than the time taken for the physical machine to execute the same step of the operation and synchronously display the operation process.

2. The task processing method according to claim 1, wherein the virtual machine is generated by image processing of the physical machine, and the system environment corresponding to the virtual machine and the system environment corresponding to the physical machine meet the consistency condition.

3. The task processing method according to claim 1, wherein a first operation is performed by a virtual machine to determine the first operation to be performed on the target task, comprising: Based on the task description information of the target task and the current user interface screenshot, a first calculation is performed to determine the first operation information required for the first operation to be performed by the target task.

4. The task processing method according to claim 3, wherein a first calculation is performed based on the task description information of the target task and the current user interface screenshot to determine the first operation information required for the first operation to be performed by the target task, including: The target model is invoked through the virtual machine; Input the task description information of the target task and the current user interface screenshot into the target model to obtain at least some information from the operation type, the target element, and the operation result of the first operation to be performed by the target task, calculated by the target model based on the input information.

5. The task processing method according to claim 3 or 4, further comprising, before performing the first operation via a physical machine: The first operation information is sent to the physical machine so that the physical machine can perform the first operation based on the first operation information.

6. The task processing method according to claim 1, wherein the first operation is executed on a physical machine in parallel with the second operation, and the operation process is displayed synchronously, includes one of the following: After the first operation is performed through the virtual machine, the first operation is executed through the physical machine in parallel with the virtual machine's pre-execution of the first operation and the performance of the second operation, and the operation process is displayed synchronously. After the first operation is pre-executed through the virtual machine, the first operation is executed through the physical machine in parallel with the second operation performed by the virtual machine, and the operation process is displayed synchronously.

7. The task processing method according to claim 6, wherein if the physical machine executes the first operation after the virtual machine pre-executes the first operation, the method further comprises: If the virtual machine's pre-execution result of the first operation indicates that the operation process of the first operation is abnormal, the first operation information required for the first operation is corrected so that the physical machine executes the first operation based on the corrected first operation information.

8. The task processing method according to claim 1, further comprising: After the pre-execution of the first operation is completed, a second operation is performed through the virtual machine to determine the second operation to be performed next to the first operation. The second operation is pre-executed by the virtual machine so that a third operation is performed after the pre-execution of the second operation is completed, the third operation being used to determine the next operation of the second operation; The second operation is executed by the physical machine in parallel with the third operation, and the operation process is displayed synchronously. After the execution and display of the second operation are completed, the next operation of the second operation is executed and displayed, until the virtual machine completes the pre-execution of each operation included in the target task and the physical machine completes the execution and display of each operation included in the target task.

9. The task processing method according to any one of claims 1-8, further comprising: In response to receiving an interrupt instruction, the current processing of the virtual machine and the physical machine is stopped to exit the target task, or the processing of the target task is corrected in response to receiving a correction instruction after the current processing is stopped.

10. A task processing apparatus, comprising: The computation module is used to perform a first computation through a virtual machine in response to obtaining a target task to be executed, so as to determine the first operation to be executed by the target task. A first processing module is configured to pre-execute the first operation through the virtual machine, so as to perform a second operation after the pre-execution of the first operation is completed, wherein the second operation is used to determine the next operation of the first operation; The second processing module is used to execute the first operation through a physical machine in a manner parallel to the second operation, and to simultaneously display the operation process, so as to execute and display the next operation of the first operation after the execution and display of the first operation are completed; The time taken for the virtual machine to pre-execute one step of the operation is less than the time taken for the physical machine to execute the same step of the operation and synchronously display the operation process.

Images