Control method and device

Abstract

The invention provides a control method and device. The method comprises the following steps: receiving a target task sent by a first system; obtaining desktop image data of a second system based on the target task; generating prompt information corresponding to the target task, and inputting the prompt information and the desktop image data into a target model to obtain a target instruction corresponding to the target task; executing the target instruction in the second system to obtain an execution result of the target task; and determining corresponding result feedback information based on the execution result, and transmitting the result feedback information to the first system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a control method and device. Background Art

[0002] With the rapid development and popularization of artificial intelligence technology, especially breakthroughs in deep learning, computer vision and natural language processing, more and more application scenarios are beginning to integrate multiple technologies to achieve a higher level of intelligence. The current automation system relies on preset scripts or rules to control the execution of tasks. The success rate of task execution is low, and other operations during the execution of tasks will interrupt the execution of tasks, and the flexibility of task execution is low. Summary of the invention

[0003] The present application provides a control method and device to solve one or more problems existing in the related art.

[0004] According to the first aspect of the present application, a control method is provided, which includes: receiving a target task sent by a first system; obtaining desktop image data of a second system based on the target task; generating prompt information corresponding to the target task, and inputting the prompt information and the desktop image data into a target model to obtain a target instruction corresponding to the target task; executing the target instruction in the second system to obtain an execution result of the target task; determining corresponding result feedback information based on the execution result, and transmitting the result feedback information to the first system.

[0005] According to one embodiment of the present application, executing the target instruction in the second system to obtain the execution result of the target task includes: determining a target control related to the target instruction in the second system; determining a target execution position of the target instruction based on the target control; executing the target instruction at the target execution position to obtain the execution result of the target instruction; and determining the execution result of the target task based on the execution result of the target instruction.

[0006] According to one embodiment of the present application, determining the target control related to the target instruction in the second system includes: obtaining attribute information corresponding to multiple controls in the second system, the attribute information including control position, control type and text label; determining the target control based on the attribute information and the target instruction.

[0007] According to one embodiment of the present application, the method also includes: obtaining historical input information of the target model; the historical input information includes one or more of historical prompt information and historical desktop image data; obtaining current desktop image data after executing the target instruction; based on the historical input information, the current desktop image data and the prompt information corresponding to the target task, predicting subsequent instructions corresponding to the target instruction through the target model.

[0008] According to one embodiment of the present application, the method also includes: determining process feedback information corresponding to the target instruction during the execution of the target instruction; transmitting the process feedback information and the target task to the first system to display a control interface of the target task in a display area of ​​the first system; and receiving a control instruction for the target task sent by the first system, wherein the control instruction is an instruction generated based on input data received by the control interface.

[0009] According to one embodiment of the present application, the method further includes: obtaining desktop image data of the second system during execution of the target instruction; transmitting the desktop image data to the first system to display the desktop image data in a display area of ​​the first system.

[0010] According to one embodiment of the present application, the method also includes: obtaining process feedback information of the target instruction, the process feedback information including one or more of execution progress, execution results, and decision information; transmitting the process feedback information in the process of executing the target instruction to the first system to display the process feedback information in the display area of ​​the first system.

[0011] According to a second aspect of the present application, a control method is provided, the method comprising: obtaining a target task; sending the target task to a second system, wherein the second system obtains a target instruction for executing the target task through prompt information corresponding to the target task and desktop image data of the second system input into a target model; receiving result feedback information corresponding to the execution result of the target task sent by the second system; and displaying the result feedback information in a display area of ​​the first system.

[0012] According to one embodiment of the present application, the method also includes: receiving process feedback information and target task sent by the second system; determining display control instructions corresponding to the feedback information based on the process feedback information and target task; displaying a control interface of the target task in a display area of ​​the first system based on the display control instructions; receiving input data for the target task from the control interface; generating control instructions for the target task based on the input data, and sending the control instructions to the second system.

[0013] According to an embodiment of the present application, the method further includes: receiving desktop image data sent by the second system; and displaying the desktop image data in a display area of ​​the first system.

[0014] According to one embodiment of the present application, the method also includes: receiving process feedback information sent by the second system; the process feedback information includes one or more of execution progress, execution results, and decision information; and displaying the process feedback information in a display area of ​​the first system.

[0015] According to a third aspect of the present application, a control device is provided, which is applied to a second system, and the device includes: a receiving module, which is used to receive a target task sent by a first system; an obtaining module, which is used to obtain desktop image data of the second system based on the target task; a generating module, which is used to generate prompt information corresponding to the target task, and input the prompt information and the desktop image data into a target model to obtain a target instruction corresponding to the target task; an executing module, which is used to execute the target instruction in the second system to obtain an execution result of the target task; and a feedback module, which is used to determine corresponding result feedback information based on the execution result, and transmit the result feedback information to the first system.

[0016] According to a fourth aspect of the present application, a control device is provided, which is applied to a first system, and the device includes: a task acquisition module, which is used to obtain a target task; a sending module, which is used to send the target task to a second system, wherein the second system obtains a target instruction for executing the target task by inputting prompt information corresponding to the target task and desktop image data of the second system into a target model; a feedback receiving module, which is used to receive result feedback information corresponding to the execution result of the target task sent by the second system; and a display module, which is used to display the result feedback information in a display area of ​​the first system.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0019] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0020] Figure 1The process flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 1 ;

[0021] Figure 2 The process flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 2 ;

[0022] Figure 3 The process flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 3 ;

[0023] Figure 4 The detailed processing flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 1 ;

[0024] Figure 5 The detailed processing flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 2 ;

[0025] Figure 6 The detailed processing flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 3 ;

[0026] Figure 7 The detailed processing flow diagram of the control method provided in the embodiment of the present application is shown in FIG. Figure 4 ;

[0027] Figure 8 A schematic diagram of an application scenario of the control method provided in an embodiment of the present application is shown;

[0028] Fig. 9 A schematic diagram of a control structure of an embodiment of the present application is shown;

[0029] Fig.10 A schematic diagram of another control structure of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] The processing flow in the control method provided in the embodiment of the present application is described. Figure 1 , Figure 1 This is a schematic diagram of the processing flow of the control method provided in the embodiment of the present application. Figure 1 , will combine Figure 1 Steps S101-S105 are shown for explanation.

[0035] Step S101, receiving a target task sent by a first system.

[0036] In some embodiments, an application in the second system can receive a target task sent by the first system. The application in the first system can be used for: allocating target tasks, monitoring the status of target task execution, and receiving feedback on the execution results of target tasks. The application in the first system can display an interactive interface in the display area of ​​the first system, allowing the user to enter commands in the interactive interface or view the execution status of the target task. The application in the second system can be used for: according to the target task sent by the first system, after executing the target task, the execution result corresponding to the target task is transmitted to the first system in the form of result feedback information. The target task may include: the target task input by the user in the first system through text or voice and the target task automatically generated by the first system. Specifically, the target task may include: tasks such as sending emails, online purchases, processing images, and searching for information. The target task may also include other tasks, which are not limited in the embodiments of the present application.

[0037] Step S102: obtaining desktop image data of the second system based on the target task.

[0038] Step S103, generating prompt information corresponding to the target task, and inputting the prompt information and desktop image data into the target model to obtain the target instruction corresponding to the target task. Specifically, obtaining the target instruction corresponding to the target task generated by the target model.

[0039] In some embodiments, the desktop image data may include: a desktop image of the second system captured by an application in the second system in response to the target task. The desktop image data may specifically include: data such as desktop layout, icons, windows, text information, and background images. The desktop image data may be the entire desktop image of the second system, or the desktop image data may be a partial desktop image of the second system. The prompt information may include: a prompt word corresponding to the target task. The prompt information may be used to assist the target model in understanding the context information of the target task and the specific requirements of the target task. The target model may include: a pre-trained algorithm model. The target model may receive input data for image processing and text processing, and output a target instruction. The target instruction may be used to complete the target task. The target instruction may include: an application in the second system performs a certain step in the target task. The second system may be a system running in a virtual machine, and the virtual machine may run in the first system or in the cloud. The target model may generate a target instruction based on the desktop image data and the prompt information, and the target instruction may be a text, specifically a natural language text, or a structured text, such as a Jason text. The target instruction may include control information, operation information, etc., such as the control information corresponding to the purchase button in the ticket purchasing software, and the operation information corresponding to the click operation.

[0040] Specifically, the target model can be a large language model or a multimodal large model. The number of adjustable parameters of the target model is greater than 500 million. The target model generates multiple output text units in sequence according to the prompt information, and the multiple output text units are used to form the output text. Further, the target model includes a target module, and the target module can convert the first feature sequence into a second feature sequence according to the association information between the feature elements in the first feature sequence. Optionally, the target module is an attention layer module. Further, the first feature sequence has a first feature element, and the second feature sequence has a second feature element corresponding to the first feature element. The second feature element is determined according to the association information between the first feature element and the target feature element. The target feature element is arranged before the first feature element in the first feature sequence. Optionally, the target module is a mask attention layer module.

[0041] Step S104, executing the target instruction in the second system to obtain the execution result of the target task.

[0042] In some embodiments, the application in the second system may execute the target instruction in the second system to obtain the execution result of the target task. Executing the target instruction may include: executing the target instruction at the target execution location in the second system to obtain the execution result of the target instruction. The execution result of the target task may be determined by the execution results of multiple target instructions.

[0043] Step S105: determine corresponding result feedback information based on the execution result, and transmit the result feedback information to the first system.

[0044] In some embodiments, an application in the second system can execute a target instruction in the second system to obtain an execution result of the target task. Executing the target instruction may include: executing the target instruction at a target execution position in the second system to obtain an execution result of the target instruction. The execution result of the target task can be determined by the execution results of multiple target instructions. The result feedback information is used to characterize the final state, effect, performance index and other information of the execution of the target task. Specifically, by analyzing the execution result, the result feedback information can include the final state of the target task, the performance index of the target task and the error code corresponding to the target task.

[0045] In some embodiments, the execution result includes a desktop image of the second system after executing the target instruction. Determining the corresponding result feedback information based on the execution result includes: obtaining a subsequent instruction corresponding to the target instruction generated by the target model by using the execution result and the prompt information target model, wherein the target instruction and the subsequent instruction each correspond to a step required to execute the target task; after the instruction corresponding to the last step of the corresponding target task is generated and executed, the result feedback information is obtained. When the virtual machine corresponding to the second system is running in the cloud, the result feedback information is sent to the first system via the remote desktop protocol.

[0046] In an embodiment of the present application, by receiving a target task from a first system, an application in a second system automatically acquires and analyzes desktop image data according to the target task, generates accurate target instructions through a data model, thereby improving the success rate of task execution, and being able to execute the target task in the second system. During the process of executing the task in the second system, other operations can be performed in the first system at the same time without interrupting the execution of the task, thereby improving the flexibility of task execution.

[0047] In some embodiments, the processing flow of the control method is shown as follows: Figure 2 ,like Figure 2 As shown, executing the target instruction in the second system in step S104 to obtain the execution result of the target task includes:

[0048] Step S201, determining a target control related to a target instruction in a second system.

[0049] In some embodiments, step S201 may include: obtaining attribute information corresponding to multiple controls in the second system respectively; determining the target control based on the attribute information and the target instruction. Among them, the controls may include: buttons, text boxes, drop-down menus and other controls. The controls may also include other controls, which are not limited in the embodiments of the present application. The attribute information may include information such as the control position, control type and text label. The attribute information may also include other attribute information, which is not limited in the embodiments of the present application. Pre-identifying the desktop image data of the second system through the target model can obtain the recognition results corresponding to the multiple controls in the second system respectively. The attribute information corresponding to the multiple controls in the second system is obtained from the recognition results.

[0050] As an example, the attribute information corresponding to multiple controls in the second system is obtained. The attribute information may include the control position, the control type and the text label. The control position may include the position coordinates in the system desktop, as well as the width and height of the control. The control type may include: buttons, text boxes and sliders. The text label may include: the text displayed on the control, such as "OK", "Save" or "Cancel" on the button. The application in the second system receives the target instruction from the first system, parses the target instruction to obtain the description of the target instruction as "click the 'Save' button". The application in the second system determines the control that matches the target instruction based on the description in the target instruction and the attribute information corresponding to the multiple controls, and determines the control as the target control. For example, a control with a text label of "Save" is searched, and the control with a text label of "Save" is determined as the target control.

[0051] Step S202, determining a target execution location of a target instruction based on the target control.

[0052] Step S203, executing the target instruction at the target execution location to obtain the execution result of the target instruction.

[0053] Step S204: determining the execution result of the target task based on the execution result of the target instruction.

[0054] In some embodiments, the application in the second system determines a target control on the system desktop, such as a specific "Submit" button, and then determines the position coordinates of the button in the system desktop as the target execution position. The automated script simulates a user click operation to execute the target instruction at the target execution position. After execution is completed, the application in the second system captures the feedback information generated after the button is clicked, and the feedback information may include a message box or page jump indicating successful form submission. The execution result of the target instruction is determined based on the feedback information. Based on the execution results of multiple target instructions, the application in the second system can determine whether the entire target task is successfully completed. The execution result of the target task may include the successful completion of the entire target task and the failure of the entire target task to execute.

[0055] In the embodiment of the present application, the target control and execution position corresponding to the target instruction are determined through the attribute information of the control, thereby ensuring the accurate execution of the target instruction. By positioning the target control, not only the success rate of task execution is improved, but also the control method can adapt to different user interfaces and environments, thereby improving the scope of application of the control method. Based on the execution result of the target instruction, the execution result of the target task can be automatically evaluated, thereby improving the execution efficiency of the task. Moreover, the target task can be executed in the second system, and in the process of executing the task in the second system, other operations can be performed in the first system at the same time, without interrupting the execution of the task, thereby improving the flexibility of task execution.

[0056] In some embodiments, the processing flow of the control method is shown as follows: Figure 3 ,like Figure 3 As shown, including:

[0057] Step S301, obtaining historical input information of the target model.

[0058] Step S302, obtaining the current desktop image data after the target instruction is executed.

[0059] Step S303, based on the historical input information, the current desktop image data and the prompt information corresponding to the target task, predict the subsequent instructions corresponding to the target instruction through the target model.

[0060] In some embodiments, the target model can process visual data and language data simultaneously to achieve multimodal learning of input information. The historical input information can include historical prompt information, historical desktop image data, and historical prompt information and historical desktop image data. Subsequent instructions can include: instructions that need to be executed after the target instruction to complete the target task.

[0061] As an example, the prompt information corresponding to the target task may include: renaming "File 1" in "Folder 1" on the desktop to "File 2". Target instruction 1 is to open "Folder 1" on the desktop. Target instruction 1 has been executed. Obtaining historical input information of the target model may include prompt information and desktop image data received by the target model when generating target instruction 1, and prompt information and desktop image data received by the target model when generating target instruction 2. Target instruction 2 is to search for "File 1" in "Folder 1". Obtain the current desktop image data after executing target instruction 2. Input the historical input information, current desktop image data and prompt information corresponding to the target task into the target model, and predict the input data through the target model. The subsequent instruction corresponding to the target instruction is to perform a renaming operation of renaming "File 1" to "File 2".

[0062] In the embodiment of the present application, historical input information and current desktop image data are used to identify patterns and trends in task execution, and then the subsequent instructions of the target instruction can be accurately predicted. By continuously optimizing the computer vision model and large language model included in the target model, the user's operating habits and preferences can be adapted. Accurate target instructions are generated through data models, which reduces manual operations, improves the efficiency of task execution and the success rate of task execution, and can perform other operations in the first system while executing tasks in the second system, without interrupting the execution of tasks, thereby improving the flexibility of task execution.

[0063] In some embodiments, the detailed optional processing flow diagram of the control method provided in the embodiments of the present application is as follows: Figure 1 ,like Figure 4 As shown, including:

[0064] Step S401: an application in the first system obtains a target task.

[0065] Step S402, the application in the first system sends a target task to the application in the second system, wherein the second system inputs a target model through prompt information corresponding to the target task and desktop image data of the second system to obtain a target instruction for executing the target task.

[0066] In some embodiments, the target task may include: a target task input by a user and a target task automatically generated by the first system. The first system may be a main system of a computer. The second system may be a virtual system running on a virtual machine. The main system may send specific requirements of the target task to the virtual system on the virtual machine. After receiving the target task, the virtual system uses virtualization technology to load the corresponding operating system image and application program to execute the target task.

[0067] Step S403: the application program in the second system obtains the desktop image data of the second system based on the target task.

[0068] Step S404: the application in the second system generates prompt information corresponding to the target task, and inputs the prompt information and desktop image data into the target model to obtain a target instruction corresponding to the target task.

[0069] Step S405 , the application program in the second system executes the target instruction in the second system to obtain the execution result of the target task.

[0070] Step S406: The application in the second system determines corresponding result feedback information based on the execution result.

[0071] Step S407: the application in the second system sends result feedback information to the application in the first system.

[0072] The specific description of each step of steps S403-S407 is the same as that of the above steps S102-S105, and will not be repeated here.

[0073] Step S408: The application in the first system displays result feedback information in the display area of ​​the first system.

[0074] In some embodiments, the display area of ​​the first system may include: the desktop of the first system, the display interface of the application in the first system, and the display area of ​​the setting display area of ​​the first system. The display area of ​​the first system may also include other display areas, which are not limited in the embodiments of the present application. The display result feedback information may include: summary information showing the execution result of the target task, chart information showing the execution result of the target task, and display methods such as key indicators of the execution result of the target task. The display result feedback information may also include other display methods, which are not limited in the embodiments of the present application.

[0075] In the embodiment of the present application, by receiving the target task from the first system, the application in the second system automatically obtains and analyzes the desktop image data according to the target task, generates accurate target instructions through the data model, improves the success rate of task execution, can execute the target task in the second system, and transmits the result feedback information of the execution result to the first system, and the application in the first system provides an intuitive monitoring interface for the user by displaying the result feedback information in the display area of ​​the first system. The monitoring interface may include detailed execution logs, possible error messages, and statistical data of task execution, which improves the user experience. And in the process of executing tasks in the second system, other operations can be performed in the first system at the same time, without interrupting the execution of tasks, and improving the flexibility of task execution.

[0076] Figure 5 This is a detailed optional processing flow diagram of the control method provided in the embodiment of the present application. Figure 2 ,like Figure 5 As shown, including:

[0077] Step S501 : During the process of executing a target instruction, an application program in the second system determines process feedback information corresponding to the target instruction.

[0078] Step S502: the application in the second system sends process feedback information and target tasks to the application in the first system.

[0079] Step S503: the application in the first system determines a display control instruction corresponding to the feedback information based on the process feedback information and the target task.

[0080] Step S504: the application in the first system displays a control interface of the target task in the display area of ​​the first system based on the display control instruction.

[0081] In some embodiments, the process feedback information may include feedback information such as execution progress, execution results, and decision information. The execution progress may include: the current stage of completion of the target instruction. The execution progress may be represented by a percentage or a stage mark. The execution result may include: the final state after the target instruction is executed. The execution result may indicate that the target instruction is executed successfully, the target instruction fails to execute, or other specific results of the target instruction. The decision information may include: information about the decision point that the system or user needs to make during the execution of the target instruction. The process feedback information may also include other feedback information, which is not limited in the embodiments of the present application. The application in the first system compares the process feedback information with the original target task to determine whether the target task is carried out as planned, or whether the target task needs to be adjusted. The display control instruction is used to define how to display the control interface of the target task in the display area of ​​the first system.

[0082] Step S505 : the application program in the first system receives input data for the target task from the control interface.

[0083] Step S506: the application program in the first system generates a control instruction for the target task based on the input data.

[0084] Step S507: the application in the first system sends a control instruction to the application in the second system.

[0085] Step S508: the application in the second system receives the control instruction for the target task sent by the first system.

[0086] As an example, an application in the first system sends a target task of playing audio 1 and audio 2 in sequence to an application in the second system. The application in the second system may include playing audio 1 according to the target instruction corresponding to the target task, and in the process of executing the target instruction to play audio 1 in the second system, the feedback information of the determination process may include the progress of audio playback. The application in the second system sends the audio playback progress and the target task to the application in the first system. The application in the first system determines the display control instruction corresponding to the feedback information based on the audio playback progress and the target task. The application in the first system displays the control interface of audio playback in the display area of ​​the first system based on the display control instruction. The control interface may include a pause control, a playback progress adjustment control, and a playback volume adjustment control. The application in the first system receives input data for audio playback from the control interface including clicking a pause control. The application in the first system generates a control instruction for the target task based on the input data, including a control instruction for pausing the playback of audio 1. The application in the first system sends a control instruction for pausing the playback of audio 1 to the application in the second system, and the application in the second system receives the control instruction for pausing the playback of audio 1 sent by the first system, and pauses the playback of audio 1 according to the control instruction.

[0087] In the embodiment of the present application, the application in the first system dynamically generates display control instructions based on process feedback information and target tasks, thereby realizing intelligent adaptation of the control interface of the target task. Through the control interface, task control and data input can be performed more intuitively and conveniently, thereby improving the user experience. Through the control interface, the execution of the target instruction can be authorized. If the target instruction is executed incorrectly, the target instruction can be controlled to be re-executed, thereby improving the success rate of task execution.

[0088] Figure 6 This is a detailed optional processing flow diagram of the control method provided in the embodiment of the present application. Figure 3 ,like Figure 6 As shown, including:

[0089] Step S601: The application program in the second system obtains the desktop image data of the second system during the process of executing the target instruction.

[0090] Step S602: The application in the second system sends desktop image data to the application in the first system.

[0091] Step S603: the application in the first system receives the desktop image data sent by the second system.

[0092] Step S604: the application program in the first system displays the desktop image data in the display area of ​​the first system.

[0093] As an example, the executed target instruction may include starting a software installation program. During the software installation process, the application in the second system may obtain real-time desktop image data of the second system. The real-time desktop image data of the second system is transmitted to the application in the first system. The application in the first system may display the real-time desktop image data of the second system on the desktop of the first system. If the virtual machine of the second system runs in the first system, the desktop image of the second system is obtained through the window of the virtual machine software. If the virtual machine of the second system runs in the cloud, the desktop image of the second system is obtained through the remote desktop protocol.

[0094] In the embodiment of the present application, the desktop image data of the second system is displayed in real time in the display area of ​​the first system, so as to realize the monitoring of the target task. Through the desktop image data of the second system, task control and data input can be performed more intuitively and conveniently, which improves the user experience. Through the desktop image data, the task progress can be checked in real time. If the target instruction is executed incorrectly, the target instruction can be controlled to be re-executed, which improves the success rate of task execution.

[0095] Figure 7 This is a detailed optional processing flow diagram of the control method provided in the embodiment of the present application. Figure 4 ,like Figure 7 As shown, including:

[0096] Step S701: an application in the second system obtains process feedback information of a target instruction.

[0097] Step S702: the application in the second system sends process feedback information to the application in the first system.

[0098] Step S703: the application in the first system receives process feedback information sent by the second system.

[0099] Step S704: the application in the first system displays the process feedback information in the display area of ​​the first system.

[0100] In some embodiments, the process feedback information may include feedback information such as execution progress, execution results, and decision information. The execution progress may include: the current stage of completion of the target instruction. The execution progress may be represented by a percentage or a stage mark. The execution result may include: the final state after the target instruction is executed. The execution result may indicate that the target instruction is executed successfully, the target instruction fails to execute, or other specific results of the target instruction. The decision information may include: information about the decision points that the system or user needs to make during the execution of the target instruction. The process feedback information may also include other feedback information, which is not limited in the embodiments of the present application.

[0101] As an example, the executed target instruction may include starting a software installation program. During the software installation process, the application in the second system may obtain real-time software installation progress. The real-time software installation progress is transmitted to the application in the first system. The application in the first system may display the real-time software installation progress in a status notification bar of the first system.

[0102] In the embodiment of the present application, the process feedback information sent by the second system is displayed in real time in the display area of ​​the first system, thereby realizing the monitoring of the target task. Through the process feedback information, the task progress can be viewed more intuitively, thereby improving the user experience. The execution of the target instruction can be authorized through the process feedback information. If the target instruction is executed incorrectly, the problem of the target instruction can be responded to in a timely manner, and the target instruction can be controlled to be re-executed or the target task can be adjusted, thereby improving the success rate of task execution.

[0103] refer to Figure 8 , a schematic diagram of an application scenario of the control method provided in an embodiment of the present application, which is applied to automatically execute a target task. Among them, the first system is a main system, and the second system is a virtual system running on a virtual machine. The processing of the second system is performed in the background. In some embodiments, the system desktop of the second system is hidden and does not need to be displayed. The second system and the first system are two independent systems, and input operations for the second system do not need to occupy input devices.

[0104] The application in the second system can receive the target task sent by the first system. The target task can be to control window B1 to perform three steps. Based on the target task, the desktop image data of the second system is obtained. The desktop image data may include window B1, window B2 and window B3. Prompt information corresponding to the target task is generated, and the prompt information and the desktop image data are input into the target model to obtain the target instruction corresponding to the target task. The target model may include an inference module, and the application in the second system may include a control module, which may be used to control the second system to perform the target task through the keyboard and mouse events of the second system. The inference module may include a composite model of a computer vision model and a large language model. The target instruction corresponding to the target task may include target instruction 1, target instruction 2 and target instruction 3. Target instruction 1 may include controlling window B1 to perform the first step; target instruction 2 may include controlling window B1 to perform the second step; target instruction 3 may include controlling window B1 to perform the third step.

[0105] After the target instruction 1 is successfully executed, the process feedback information of the target instruction 1 is obtained, and the process feedback information may include execution progress, execution results and decision information. The process feedback information during the execution of the target instruction 1 is transmitted to the first system, and the process feedback information of the target instruction 1 is displayed in the control interface A1 in the display area of ​​the first system as successful execution.

[0106] After the target instruction 2 is successfully executed, the process feedback information of the target instruction 2 is obtained, and the process feedback information may include execution progress, execution results and decision information. The process feedback information during the execution of the target instruction 2 is transmitted to the first system, so that the process feedback information of the target instruction 2 is displayed in the control interface A1 in the display area of ​​the first system as successful execution.

[0107] During the execution of the target instruction 3, process feedback information of the target instruction 3 is obtained, and the process feedback information may include execution progress, execution results and decision information. The process feedback information during the execution of the target instruction 1 is transmitted to the first system, so that the decision information of the target instruction 3 is displayed in the control interface A1 in the display area of ​​the first system as whether to agree to continue the execution. If the input data received from the control interface A1 for the decision information of the target instruction 3 is to agree to continue the execution, a control instruction for the target instruction 3 is generated based on the input data, and the control instruction is sent to the second system.

[0108] The following is a description of an exemplary structure of a software module included in a control device for a second system provided in an embodiment of the present application. In some embodiments, Fig. 9 As shown, the control device applied to the second system may include: a receiving module 801, which can be used to receive the target task sent by the first system; an obtaining module 802, which can be used to obtain desktop image data of the second system based on the target task; a generating module 803, which can be used to generate prompt information corresponding to the target task, and input the prompt information and the desktop image data into the target model to obtain the target instruction corresponding to the target task; an executing module 804, which can be used to execute the target instruction in the second system to obtain the execution result of the target task; a feedback module 805, which can be used to determine the corresponding result feedback information based on the execution result, and transmit the result feedback information to the first system.

[0109] In some embodiments, the execution module 804 can be used to: determine a target control related to a target instruction in the second system; determine a target execution location of the target instruction based on the target control; execute the target instruction at the target execution location to obtain an execution result of the target instruction; and determine an execution result of the target task based on the execution result of the target instruction.

[0110] In some embodiments, the execution module 804 can also be used to: obtain attribute information corresponding to multiple controls in the second system, the attribute information including control position, control type and text label; determine the target control based on the attribute information and the target instruction.

[0111] In some embodiments, the control device applied to the second system also includes a prediction module, which can be used to: obtain historical input information of the target model; the historical input information includes one or more of historical prompt information and historical desktop image data; obtain current desktop image data after executing the target instruction; based on the historical input information, the current desktop image data and the prompt information corresponding to the target task, predict the subsequent instructions corresponding to the target instruction through the target model.

[0112] In some embodiments, the feedback module 805 can be used to: determine the process feedback information corresponding to the target instruction during the execution of the target instruction; transmit the process feedback information and the target task to the first system to display the control interface of the target task in the display area of ​​the first system; the receiving module 801 can be used to receive the control instruction for the target task sent by the first system, and the control instruction is an instruction generated based on the input data received by the control interface.

[0113] In some embodiments, the feedback module 805 can be used to: obtain desktop image data of the second system during the execution of the target instruction; transmit the desktop image data to the first system to display the desktop image data in the display area of ​​the first system.

[0114] In some embodiments, the feedback module 805 can be used to: obtain process feedback information of the target instruction, the process feedback information including one or more of execution progress, execution results, and decision information; transmit the process feedback information during the execution of the target instruction to the first system to display the process feedback information in the display area of ​​the first system.

[0115] The following is a description of an exemplary structure of a software module included in a control device for a first system provided in an embodiment of the present application. In some embodiments, Fig.10 As shown, the control device applied to the first system may include: a task acquisition module 901, which can be used to obtain a target task; a sending module 902, which can be used to send the target task to a second system, wherein the second system inputs a target model through prompt information corresponding to the target task and desktop image data of the second system to obtain a target instruction for executing the target task; a feedback receiving module 903, which can be used to receive result feedback information corresponding to the execution result of the target task sent by the second system; and a display module 904, which can be used to display the result feedback information in the display area of ​​the first system.

[0116] In some embodiments, the feedback receiving module 903 can be used to: receive process feedback information and target tasks sent by the second system; the display module 904 can be used to determine the display control instructions corresponding to the feedback information based on the process feedback information and the target task; based on the display control instructions, display the control interface of the target task in the display area of ​​the first system; the sending module 902 can be used to receive input data for the target task from the control interface; generate control instructions for the target task based on the input data, and send the control instructions to the second system.

[0117] In some embodiments, the feedback receiving module 903 may be used to receive desktop image data sent by the second system; the display module 904 may be used to display the desktop image data in the display area of ​​the first system.

[0118] In some embodiments, the feedback receiving module 903 can be used to receive process feedback information sent by the second system; the process feedback information includes one or more of execution progress, execution results, and decision information; the display module 904 can be used to display the process feedback information in the display area of ​​the first system.

[0119] It should be noted that the description of the control device in the embodiment of the present application has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 8 The present invention can be understood by referring to the description of any one of the accompanying drawings.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A control method, the method comprising: Receiving the target task sent by the first system; Based on the target task, obtaining desktop image data of the second system; Generate prompt information corresponding to the target task, and input the prompt information and the desktop image data into the target model to obtain the target instruction corresponding to the target task; executing the target instruction in the second system to obtain an execution result of the target task; Corresponding result feedback information is determined based on the execution result, and the result feedback information is transmitted to the first system.

2. The method according to claim 1, wherein executing the target instruction in the second system to obtain the execution result of the target task comprises: determining, in the second system, a target control associated with the target instruction; Based on the target control, determining a target execution position of the target instruction; Executing the target instruction at the target execution position to obtain an execution result of the target instruction; Based on the execution result of the target instruction, the execution result of the target task is determined.

3. The method according to claim 2, wherein determining the target control related to the target instruction in the second system comprises: Obtaining attribute information corresponding to a plurality of controls in the second system, wherein the attribute information includes a control position, a control type, and a text label; A target control is determined based on the attribute information and the target instruction.

4. The method according to claim 1, further comprising: Obtaining historical input information of the target model; the historical input information includes one or more of historical prompt information and historical desktop image data; Obtain the current desktop image data after executing the target instruction; Based on the historical input information, the current desktop image data and the prompt information corresponding to the target task, a subsequent instruction corresponding to the target instruction is predicted through the target model.

5. The method according to claim 1, further comprising: During the execution of the target instruction, determining process feedback information corresponding to the target instruction; Transmitting the process feedback information and the target task to the first system, so as to display a control interface of the target task in a display area of ​​the first system; A control instruction for the target task sent by the first system is received, where the control instruction is an instruction generated based on input data received by the control interface.

6. The method according to claim 1, further comprising: In the process of executing the target instruction, obtaining desktop image data of the second system; The desktop image data is transmitted to the first system so as to be displayed in a display area of ​​the first system.

7. The method according to claim 1, further comprising: Obtaining process feedback information of the target instruction, wherein the process feedback information includes one or more of execution progress, execution result, and decision information; The process feedback information during the execution of the target instruction is transmitted to the first system, so that the process feedback information is displayed in a display area of ​​the first system.

8. A control method, the method comprising: Obtain the target task; Sending the target task to the second system, wherein the second system inputs the target model through the prompt information corresponding to the target task and the desktop image data of the second system to obtain the target instruction for executing the target task; Receiving result feedback information corresponding to the execution result of the target task sent by the second system; The result feedback information is displayed in a display area of ​​the first system.

9. The method according to claim 8, further comprising: receiving process feedback information and target tasks sent by the second system; Based on the process feedback information and the target task, determining a display control instruction corresponding to the feedback information; Based on the display control instruction, displaying a control interface of the target task in a display area of ​​the first system; receiving input data for the target task from the control interface; A control instruction for the target task is generated based on the input data, and the control instruction is sent to the second system.

10. The method according to claim 8, further comprising: receiving desktop image data sent by the second system; The desktop image data is displayed in a display area of ​​the first system.

11. The method according to claim 8, further comprising: receiving process feedback information sent by the second system; The process feedback information includes one or more of execution progress, execution results, and decision information; The process feedback information is displayed in a display area of ​​the first system.

12. A control device, applied to a second system, comprising: A receiving module, used for receiving a target task sent by the first system; An acquisition module, used for acquiring desktop image data of the second system based on the target task; A generation module, used for generating prompt information corresponding to the target task, and inputting the prompt information and the desktop image data into the target model to obtain a target instruction corresponding to the target task; An execution module, used for executing the target instruction in the second system to obtain an execution result of the target task; A feedback module is used to determine corresponding result feedback information based on the execution result, and transmit the result feedback information to the first system.

13. A control device, applied to a first system, comprising: A task acquisition module is used to obtain the target task; A sending module, used for sending the target task to the second system, wherein the second system obtains a target instruction for executing the target task by inputting the prompt information corresponding to the target task and the desktop image data of the second system into a target model; A feedback receiving module, used for receiving result feedback information corresponding to the execution result of the target task sent by the second system; A display module is used to display the result feedback information in a display area of ​​the first system.

Images