Task execution method and device, electronic equipment and program product

Abstract

The invention provides a task execution method which comprises the steps that a target program is started in a first desktop in response to a received target instruction, the first desktop and a second desktop currently used by a user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; and under the condition that the second desktop is displayed, controlling the target program to execute a target task corresponding to the target instruction through the first desktop.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a task execution method, device, electronic device, and program product. Background Art

[0002] With the rapid development of computer technology, automatic control technology is increasingly being used in electronic devices. For example, on a computer, the mouse and keyboard can be operated automatically so that the application automatically performs the target task without manual operation by the user.

[0003] However, in this case, it will affect the user's ability to perform other operations with the mouse and keyboard, and the open application window will also affect the user's ability to operate other applications. The user can only choose between manual operation and automatic operation, and cannot automatically operate the target application while operating other applications. Summary of the invention

[0004] In view of this, the present disclosure provides a task execution method, device, electronic device medium, and program product.

[0005] One aspect of the present disclosure provides a task execution method, including: in response to receiving target task information, starting a target program in a first desktop, wherein the first desktop and a second desktop currently being used by a user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; when the second desktop is displayed, controlling the target program to execute the target task corresponding to the target task information through the first desktop.

[0006] According to an embodiment of the present disclosure, a target program is controlled to execute a target task corresponding to target task information through a first desktop, including: generating a third desktop based on a second desktop, wherein the third desktop and the second desktop at least partially share the same coordinate system information; synchronizing the screen in the first desktop to the third desktop; and controlling the target program in the first desktop to execute the target task corresponding to the target task information based on a screenshot of the third desktop.

[0007] According to an embodiment of the present disclosure, based on a screenshot of a third desktop, controlling a target program in the first desktop to execute a target task corresponding to target task information includes: obtaining target operation step information for the target task based on the screenshot and the target task information, the target operation step information at least including a target control and a first instruction in the first desktop, the target operation step information corresponding to the target program, the operating system having a first instruction and a second instruction, the first instruction and the second instruction having the same function, and the second instruction being used to execute a specific operation on the second desktop; controlling the target program in the first desktop to execute the target step corresponding to the target task information based on the target operation step information.

[0008] According to an embodiment of the present disclosure, target operation step information for a target task is obtained based on a screenshot and target task information, including: identifying control information in a first desktop; obtaining prompt information corresponding to the target task information; inputting the control information, prompt information and screenshot into a target model; and obtaining at least one target operation step information generated by the target model.

[0009] According to an embodiment of the present disclosure, the target step is the first step, and the task execution method also includes: in response to the completion of the execution of the first step, obtaining an updated screenshot of the third desktop; obtaining second operation step information for the target task based on the updated screenshot and the target task information; and controlling the target program in the first desktop to execute the second step corresponding to the target task information based on the second operation step information.

[0010] According to an embodiment of the present disclosure, task execution further includes: generating inquiry information on the second desktop when the user is required to operate the target program, the inquiry information being used to inquire whether to switch from displaying the second desktop to displaying the first desktop.

[0011] According to an embodiment of the present disclosure, the target task information is information input by the target user on the second desktop.

[0012] Another aspect of the present disclosure provides a task execution device, including: a startup module, used to start a target program in a first desktop in response to receiving target task information, wherein the first desktop and a second desktop currently being used by a user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; and a control module, used to control the target program to execute the target task corresponding to the target task information through the first desktop when the second desktop is displayed.

[0013] Another aspect of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0014] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0015] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 An exemplary operating system to which the task execution method according to an embodiment of the present disclosure can be applied is schematically shown;

[0018] Figure 2 A flowchart of a task execution method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 3 A flowchart of controlling a target program in a task execution method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 4 A flowchart of controlling a target program according to a screenshot in a task execution method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5 The effect of executing instructions on the desktop according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 6 Schematically shows a flow chart of obtaining operation step information in a task execution method according to an embodiment of the present disclosure;

[0023] Figure 7 Another flowchart of the task execution method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown;

[0025] Fig. 9 A block diagram schematically shows a method of generating inquiry information in a second desktop according to an embodiment of the present disclosure;

[0026] Fig.10 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown;

[0027] Fig.11 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown;

[0028] Fig.12 A block diagram schematically shows a task execution device according to an embodiment of the present disclosure; and

[0029] Fig.13 A block diagram of an electronic device suitable for implementing a task execution method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] An embodiment of the present disclosure provides a task execution method, comprising: in response to receiving target task information, starting a target program in a first desktop, wherein the first desktop and a second desktop currently being used by a user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; when the second desktop is displayed, controlling the target program to execute the target task corresponding to the target task information through the first desktop.

[0035] Figure 1 An exemplary operating system to which the task execution method according to an embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 The examples shown are merely examples of operating systems to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1As shown, the operating system 100 according to this embodiment may include a first desktop 101 , a second desktop 102 , at least one application 103 , and at least one interface 104 .

[0037] The first desktop 101 and the second desktop 102 provide a visual and interactive environment for the user, so that the user can conveniently manage and access applications, files and system functions. For example, the application window and icon are displayed, prompt information is displayed, and in response to the hardware data received by the operating system, the operation of the corresponding hardware is displayed, for example, the position and action of the mouse are displayed, and the input content of the keyboard is displayed. In the operating system, only one of the first desktop 101 and the second desktop 102 can be displayed as the current desktop, and the desktop that is not displayed is a hidden desktop. The user cannot directly operate the hidden desktop, and needs to switch the hidden desktop to the desktop that is not currently displayed before operating the content in the desktop.

[0038] The application 103 can display the program page on the desktop in the form of a visual window. When the visual window of the application 103 is displayed on the current desktop, the user can operate the application through hardware peripherals, such as a mouse and keyboard, etc. When the visual window of the application 103 is displayed on the hidden desktop, the window is invisible to the user, and the user cannot directly operate the application through hardware peripherals.

[0039] The operating system can control the application 103 to perform a task or a target operation, or, in response to a signal or data transmitted from a hardware device such as a mouse or keyboard received through the interface 104, control the application 103 to perform a task or a target operation. The operating system can also control the switching between the first desktop and the second desktop, or, in response to a signal or data transmitted from a hardware device such as a mouse or keyboard received through the interface 104, control the switching between the first desktop and the second desktop.

[0040] It should be noted that the task execution method provided in the embodiment of the present disclosure can generally be executed by an operating system. The task execution method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the operating system and can communicate with the operating system.

[0041] It should be understood that Figure 1 The number of desktops, applications, and interfaces in the embodiment is only illustrative. Any number of desktops, applications, and interfaces may be provided as required.

[0042] Figure 2 The flowchart of the task execution method according to the embodiment of the present disclosure is schematically shown.

[0043] like Figure 2As shown, the method includes operations S210 to S220.

[0044] In operation S210, in response to receiving the target task information, a target program is started in a first desktop, wherein the first desktop and a second desktop currently being used by the user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time.

[0045] According to an embodiment of the present disclosure, the target task information may include a natural language description of the target task, at least one instruction / symbol / code / label according to a specific rule, or at least one of other types of data that can express the content of completing the target task.

[0046] The natural language description can be a text description or a recorded description, such as "I want to use my mailbox to send an email to A@***.com, the email subject is B, and the email body is C"; at least one instruction / symbol / code / label according to a specific rule, such as a programming language that can express certain steps, such as python, C++, assembly language, etc., or a user-defined instruction / symbol / code / label or combination of content that can express the content of the target task. Other types of data content, such as action and gesture information obtained through a camera, for example, different gestures can represent different steps or operations, so as to fully express the content of the target task.

[0047] The first desktop and the second desktop are different desktops belonging to the same operating system, and the first desktop and the second desktop are not displayed at the same time. That is, only one of the first desktop and the second desktop can use the display coordinate system content of the operating system to display the desktop content on the display device. And, the desktop that the user can directly operate is the displayed desktop. The desktop that is not displayed is a hidden desktop. When it is not displayed, the user cannot directly operate the content in the hidden desktop. The first desktop and the second desktop can be switched with each other. The first desktop and the second desktop are not displayed at the same time, which means that the first desktop and the second desktop themselves are not displayed at the same time. The thumbnail of the first desktop and the thumbnail of the second desktop will be displayed simultaneously in the desktop management interface, but it does not belong to the first desktop and the second desktop not being displayed at the same time referred to in this application.

[0048] For example, application 1 is running on the first desktop, and an interactive window of application 1 is displayed. When switching from the first desktop to the second desktop, the screen of the first desktop will not be visible, and the content of the second desktop will be displayed. The user can operate the content displayed in the second desktop, such as the application, through operating devices such as a mouse and keyboard, but cannot directly operate application 1 in the first desktop through a mouse or keyboard.

[0049] The target program is at least one application program required to execute the target task in the target task information. By executing the operation steps on the target application program, the function corresponding to the target application program is realized to complete the target task in the target task information.

[0050] For example, if the target task information is: "I want to use my mailbox to send an email to A@***.com, the email subject is B, and the email body is C", then the target program is the mailbox program. For example, if the target task information is: "Check my mailbox, if I receive a work email, write the email content to the note", then the target application programs are the mailbox program and the note program.

[0051] In response to receiving the target task information, the target program is started in the first desktop. At this time, the desktop currently being displayed can be the first desktop or the second desktop. In some embodiments, if the desktop currently being displayed is the first desktop, after the target program is started in the first desktop, it can automatically switch to the second desktop to facilitate the user to perform other operations without being affected by the target program in the first desktop.

[0052] In some embodiments, the target task information may be information input by the target user on the second desktop. Specifically, when the displayed desktop is the second desktop and the first desktop is a hidden desktop, there may be a preset program or process on the second desktop. After the program or process is run, an input interface may appear on the second desktop. The target user may use a mouse, keyboard, or other device to input the target task information through the input interface.

[0053] In operation S220, when the second desktop is displayed, the target program is controlled to execute the target task corresponding to the target task information through the first desktop.

[0054] Specifically, when the second desktop is displayed, the first desktop is hidden, and the target program is controlled to execute the target task corresponding to the target task information through the hidden first desktop. In this case, the target task is automatically executed, and its interactive window is hidden and invisible, which will not affect the display content on the second desktop. Controlling the target program to execute the target task corresponding to the target task information through the hidden first desktop is automated control, and the user can interact with the visual content on the second desktop without affecting the operation of the target program.

[0055] Figure 3 The flowchart of controlling the target program in the task execution method according to the embodiment of the present disclosure is schematically shown.

[0056] like Figure 3 As shown, based on the foregoing embodiment, operation S220 may include operations S310 to S330.

[0057] In operation S310, a third desktop is generated based on the second desktop, and the third desktop and the second desktop at least partially share the same coordinate system information.

[0058] Specifically, based on the second desktop currently being displayed, a third desktop is generated. The third desktop can be an extension of the second desktop and at least partially shares the same coordinate system information with the second desktop. In addition, the screen content of the third desktop does not cover / block any content of the second desktop.

[0059] For example, a display adapter can be used to construct a virtual extended desktop as a third desktop based on the second desktop currently being displayed. The third desktop constructed by the display adapter can be similarly understood as the dual-screen mode commonly used by computers, where one screen displays the second desktop and the other screen displays the third desktop. The difference is that the virtual extended desktop constructed by the display adapter here is not connected to the display to display the screen.

[0060] In the coordinate information of the system, the origin (0, 0) of the coordinate system is usually located at the upper left corner of the main display screen. When a virtual third desktop is expanded based on the main display screen, that is, the second desktop currently displayed in the present disclosure, the third desktop will be assigned a coordinate range. This coordinate range is determined relative to the coordinate system of the main screen, which is equivalent to the third desktop and the second desktop at least partially sharing the same coordinate system information.

[0061] For example, the display resolution of the second desktop is 1920×1080, and the third desktop is a right horizontally extended desktop of the second desktop, then the coordinates of the upper left corner of the third desktop may start at (1920, 0).

[0062] When the second and third desktops share at least some coordinate system information, the second and third desktops can be considered as a continuous workspace. This means that although each desktop can have its own resolution and screen settings (such as orientation and zoom level), they together constitute a large display area in which all desktops at least partially share a unified coordinate system. This enables the operating system to know which desktop a window or application should appear on at any given point in time, and its position in the entire display area.

[0063] At this time, the display screen of the operating system only displays the content within the coordinate system range corresponding to the second desktop, and does not display the content within the coordinate system range of the third desktop.

[0064] In operation S320, the screen in the first desktop is synchronized to the third desktop. Since the first desktop is hidden, the screen content in the first desktop does not have coordinate system information. After the screen in the first desktop is synchronized to the third desktop, the third desktop has the same screen as the hidden first desktop. When the screen in the first desktop changes, the screen in the third desktop also changes synchronously. Moreover, since the third desktop has coordinate system information, it is equivalent to giving the screen in the first desktop corresponding coordinate system information.

[0065] In operation S330, according to the screenshot of the third desktop, the target program in the first desktop is controlled to execute the target task corresponding to the target task information.

[0066] Specifically, since the third desktop has coordinate system information, the third desktop can be screenshoted. Through the screenshot of the third desktop, the interactive interface and the corresponding interactive position of the target program in the first desktop can be obtained. Based on the interactive interface and the corresponding interactive position of the target program, the target program in the first desktop can be controlled to execute the target task corresponding to the target task information.

[0067] Figure 4 The flowchart of controlling the target program according to the screenshot in the task execution method according to the embodiment of the present disclosure is schematically shown.

[0068] like Figure 4 As shown, based on the foregoing embodiment, operation S330 may include operation S410 and operation S420.

[0069] In operation S410, target operation step information for the target task is obtained based on the screenshot and the target task information. The target operation step information includes at least a target control and a first instruction in the first desktop. The target operation step information corresponds to the target program. The operating system has a first instruction and a second instruction. The first instruction and the second instruction have the same function. The second instruction is used to perform a specific operation on the second desktop.

[0070] In operation S420, the target program in the first desktop is controlled to execute the target step corresponding to the target task information according to the target operation step information.

[0071] Specifically, the target operation information represents at least one step and operation of executing the target task. For example, the target task information is: "I want to use the mailbox to send an email to A@***.com, the email subject is B, and the email body is C". At this time, the screenshot of the first desktop has an interactive window of the email program, so the screenshot in the third desktop also has the same picture. Then, according to the screenshot and the target task information, a target operation step information for the target task can be obtained, such as "control the mouse cursor through instruction 1 to click the input box corresponding to the recipient". It should be noted that the example here is for those skilled in the art to know the specific content represented by the target operation step information, and the natural language example given. In actual implementation, the target operation step information here is structured content, and the format is text format or other formats, such as jason string, etc. In order to facilitate the description of the scheme below, the target operation step will continue to be described using natural language such as "control the mouse cursor through instruction 1 to click the input box corresponding to the recipient" When implementing this scheme, those skilled in the art can use the corresponding structured content that can represent the natural language to implement it.

[0072] The target control may be, for example, a part of the interface that requires interaction, such as a button, an input box, etc. The first instruction may be a specific operation instruction executed on the target control, such as click, input, etc.

[0073] For example, the content represented by the target operation step information is "control the mouse cursor to click the input box corresponding to the recipient through instruction 1", then the target control is the input box corresponding to the recipient, and the first instruction is the instruction corresponding to the click operation of controlling the mouse cursor to click the position of the input box.

[0074] According to the embodiments of the present disclosure, the execution of instructions sometimes requires coordinate system information. For example, the first instruction is "click at the location of the input box corresponding to the recipient", then it is necessary to know the location of the input box corresponding to the recipient. Since the first desktop is a hidden desktop and has no coordinate system information, it is necessary to obtain the location of the input box corresponding to the recipient through a screenshot of the third desktop to generate the first instruction.

[0075] It should be noted that, in the disclosed embodiment, the operating system has a first instruction and a second instruction, the first instruction and the second instruction have the same function, the second instruction is used to perform a specific operation on the second desktop, and the first instruction and the second instruction are different. Among them, the second instruction can be automatically generated and executed without user intervention. Specifically, the second instruction can be an operation on the interactive content displayed on the second desktop, such as clicking, inputting, etc. The first instruction is to perform a specific operation on the same interactive content on the invisible first desktop. When the contents of the first desktop and the second desktop are exactly the same, the effect of executing the first instruction on the first desktop is the same as the effect of executing the second instruction on the second desktop.

[0076] Figure 5 The effect of executing instructions on the desktop according to an embodiment of the present disclosure is schematically shown.

[0077] For example, Figure 5 As shown, in the first desktop, the mailbox application is opened, and the first instruction represents "a click operation at the location of the input box corresponding to the recipient". After the first instruction is executed, the effect achieved is that the mouse cursor is moved to the location of the input box corresponding to the recipient and a click operation is performed. However, in fact, the movement or click of the hardware mouse does not affect the execution of the first instruction. Correspondingly, there is also a corresponding second instruction in the system. In the second desktop, the mailbox application is opened, and the second instruction represents "a click operation at the location of the input box corresponding to the recipient". After the second instruction is executed, the effect achieved is also that the mouse cursor is moved to the location of the input box corresponding to the recipient and a click operation is performed.

[0078] According to an embodiment of the present disclosure, the first instruction may be a virtual operation instruction, such as click, input, and other instructions generated by the invoke class. The invoke class represents a collection of methods / functions / classes in classes related to graphical user interface (GUI) components that are used to ensure thread-safe updates of UI elements or perform UI-related operations. These methods / functions / classes can run to safely call UI threads from non-UI threads to update UI elements and perform UI-related operations. For example, bypassing the control of the keyboard and directly entering the corresponding content in the input box. For another example, the user can simulate the action of clicking a button without using a mouse.

[0079] For example, Figure 5 Taking the content shown as an example, the first instruction can be a virtual operation instruction that simulates a user clicking at the input box, such as an invoke instruction. The second instruction is an instruction generated by the user operating the mouse to click at the input box to enable the operating system to complete the corresponding operation.

[0080] On this basis, by controlling the target program in the first desktop to execute the target step corresponding to the target task information according to the target operation step information, it is possible to control the target control in the hidden first desktop according to the first instruction so that the target program executes the target step corresponding to the target task information, while the second desktop currently being displayed is not interfered with by the first desktop.

[0081] Figure 6 The flowchart for obtaining operation step information in the task execution method according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 6 As shown, based on any of the foregoing embodiments, operation S410 may include operations S610 to S640.

[0083] In operation S610, control information in the first desktop is identified. The control information in the first desktop may include interactive elements in the first desktop such as buttons, drop-down boxes, input boxes, and the like.

[0084] In operation S620, prompt information corresponding to the target task information is obtained. The prompt information corresponding to the target task information may be description information related to the step in the target task information, or may be instruction / symbol / code / label information corresponding to the step content.

[0085] In operation S630, the control information, the prompt information, and the screenshot are input into the target model.

[0086] In operation S640, operation step information of target module generation is obtained.

[0087] The target model can be a pre-trained model that can generate operation step information based on control information, prompt information and screenshots. For example, a deep learning model, a machine learning model, or a whole formed by a combination of multiple different types of models. Specifically, it can be a combination of one or more models of a feedforward neural network, a convolutional neural network, a recurrent neural network, a generative adversarial network, a self-attention and transformer model, a supervised learning model, an unsupervised learning model, and a reinforcement learning model. The target model can be a large language model or a multimodal large model. The number of parameters of the target model that can adjust the parameters is greater than 500 million. The target model generates multiple output text units in sequence according to the prompt information, and the multiple output text units are used to form the output text. Furthermore, the target model includes a target module, which can convert the first feature sequence into a second feature sequence according to the association information between the feature elements in the first feature sequence. Optionally, the target module is an attention layer module. Furthermore, the first feature sequence has a first feature element, and the second feature sequence has a second feature element corresponding to the first feature element. The second feature element is determined based on the association information between the first feature element and the target feature element. The target feature element is arranged before the first feature element in the first feature sequence. Optionally, the target module is a mask attention layer module.

[0088] Figure 7 Another flowchart of the task execution method according to an embodiment of the present disclosure is schematically shown.

[0089] like Figure 7 As shown, based on any of the foregoing embodiments, the target step is the first step, and the task execution method may further include operations S710 to S730.

[0090] In operation S710, in response to completion of the execution of the first step, an updated screenshot of the third desktop is obtained.

[0091] In operation S720, second operation step information for the target task is obtained according to the updated screenshot and the target task information.

[0092] In operation S730, the target program in the first desktop is controlled to execute the second step corresponding to the target task information according to the second operation step information.

[0093] Generally, the target program needs more than one step to execute the target task corresponding to the target task information. After the previous step is completed, the window or screen of the target program may change. At this time, the screen corresponding to the first desktop is updated, and this update is synchronized to the third desktop. The screen with coordinate system information on the third desktop changes synchronously. At this time, the updated screenshot of the third interface is obtained to obtain the position information corresponding to the updated screen in the first desktop. For example, after clicking the [Recipient] button in the mailbox program, a list of candidate recipients may pop up, and the candidate list will at least partially cover a part of the original interface of the mailbox program. If the next step is to select the target candidate in the list of candidate recipients, then it is necessary to determine the position of the target candidate displayed in the updated screen. Therefore, after the previous instruction (clicking the recipient) is completed, it is necessary to re-acquire the screenshot of the third interface to be able to or the position of the target recipient in the screen, so that the second operation step information of selecting the target candidate can be generated.

[0094] It should be noted that the above first step and second step are only examples, and do not mean that the target task corresponding to the target task information executed by the target program only has the first step and the second step. Among the multiple steps required for the target program to execute the target task corresponding to the target task information, any step can be the first step, and the second step is the step immediately following the first step. That is, after any step is completed, it is necessary to re-acquire a screenshot of the third desktop until the last step is completed. Any operation step information is generated based on the screenshot of the third interface, the control information in the first desktop, and the prompt information corresponding to the target task information after the previous step is completed.

[0095] Figure 8 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 8 As shown, based on any of the foregoing embodiments, the task control method may further include operation S810.

[0097] In operation S810, when the user needs to operate the target program, query information is generated on the second desktop, and the query information is used to inquire whether to switch from displaying the second desktop to displaying the first desktop.

[0098] According to an embodiment of the present disclosure, when the automated program does not have permission to operate some specific steps and requires the user to execute them manually, or when some specific steps cannot be completed and require the user to operate the target program, an inquiry message can be generated in the second desktop currently being used by the user to ask the user whether to switch from displaying the second desktop to displaying the first desktop, so that the user can complete the specific steps in the displayed first desktop.

[0099] In some embodiments, the inquiry information may be a preset fixed inquiry content, for example: "You need manual operation to hide the program on the desktop. Do you want to switch the desktop?".

[0100] In some embodiments, the query information can be generated based on at least one of the target task information, the target task information, the target application, and the screenshot of the third desktop. For example, the query information is generated based on at least one of the target task information, the target task information, the target application, and the screenshot of the third desktop through a pre-trained query model.

[0101] Fig. 9 A block diagram for generating inquiry information in the second desktop is schematically shown.

[0102] For example, Fig. 9 As shown, in the first desktop, it is necessary to input a verification code received by the user's mobile phone. At this time, since the specific content of the verification code cannot be known, the target task cannot be automatically executed. Then, on the second desktop being displayed, according to the target task information, target task information, target application, and at least one of the screenshots of the third desktop, an inquiry model can be used to generate an inquiry message: "Program A requires input of a verification code, do you want to switch desktops?" to prompt the user whether it is necessary to switch desktops, and the operations to be performed after switching desktops.

[0103] Fig.10 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown.

[0104] like Fig.10 As shown, based on any of the foregoing embodiments, operation S1010 may be further included before operation S210.

[0105] In operation S1010, in response to receiving a target instruction, it is determined whether the first desktop exists, and if the first desktop does not exist, the first desktop is created.

[0106] In some cases, the user uses only one desktop in the operating system. In this case, when there is only the current desktop in the system, a first desktop is created to control the target application to execute the target task.

[0107] Fig.11 Another flowchart of the task control method according to an embodiment of the present disclosure is schematically shown.

[0108] like Fig.11 As shown, based on any of the foregoing embodiments, the task execution method may further include operations S1110 to S1120.

[0109] In operation S1110, in response to the user switching the current desktop to the first desktop, the target program is controlled to execute the target task corresponding to the target instruction through the first desktop.

[0110] In operation S1120, when the target task is not completed and the user switches the current desktop from the first desktop to the second desktop, the target program is controlled to execute the target task corresponding to the target instruction through the first desktop.

[0111] Specifically, when the second desktop is the currently displayed desktop and the target application is being controlled to execute the target task through the hidden first desktop, the user may switch the desktop from the second desktop to the first desktop to execute other operations in the first desktop, or the user may decide that he should manually operate the target application, or the automatic control cannot be completed at this time and the user needs to intervene manually. At this time, the control of the target program in the first desktop will interfere with the user's operation, so the control of the target program to execute the target task corresponding to the target instruction can be interrupted at this time.

[0112] When the target task is not completed and the user switches the current desktop from the first desktop to the second desktop, the target program is controlled to execute the target task corresponding to the target instruction through the first desktop. Specifically, the operation step information to be executed can be re-determined based on the screenshot of the third desktop, and the target program in the first desktop is controlled to execute the target step corresponding to the target task information based on the operation step information.

[0113] Fig.12 The block diagram schematically shows a task execution device according to an embodiment of the present disclosure.

[0114] like Fig.12 As shown, the task execution device 1200 includes a starting module 1210 and a control module 1220 .

[0115] The launch module 1210 is used to launch the target program in the first desktop in response to receiving the target task information, wherein the first desktop and the second desktop currently being used by the user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time. In some embodiments, the launch module 1210 is used to perform the aforementioned operation S210, which is not described in detail here.

[0116] The control module 1220 is used to control the target program to execute the target task corresponding to the target task information through the first desktop when the second desktop is displayed. In some embodiments, the control module 1220 is used to execute the above operation S220, which will not be described in detail here.

[0117] According to an embodiment of the present disclosure, the control module 1220 may further include a first generation module 1310 , a synchronization module 1320 , and a second control module 1330 .

[0118] The first generating module 1310 is used to generate a third desktop based on the second desktop, and the third desktop and the second desktop at least partially share the same coordinate system information. In some embodiments, the control module 1310 is used to perform the above operation S310, which will not be described in detail here.

[0119] The synchronization module 1320 is used to synchronize the screen in the first desktop to the third desktop. In some embodiments, the synchronization module 1320 is used to perform the aforementioned operation S320, which will not be described in detail here.

[0120] The second control module 1330 is used to control the target program in the first desktop to execute the target task corresponding to the target task information according to the screenshot of the third desktop. In some embodiments, the second control module 1330 is used to execute the aforementioned operation S330, which will not be described in detail here.

[0121] According to an embodiment of the present disclosure, the second control module 1330 may include a first obtaining module 1410 and a second control module 1420 .

[0122] The first acquisition module 1410 is used to obtain target operation step information for the target task based on the screenshot and the target task information, the target operation step information at least includes the target control and the first instruction in the first desktop, the target operation step information corresponds to the target program, the operating system has the first instruction and the second instruction, the first instruction and the second instruction have the same function, and the second instruction is used to perform a specific operation on the second desktop. In some embodiments, the first acquisition module 1410 is used to perform the aforementioned operation S410, which is not described in detail here.

[0123] The second control module 1420 is used to control the target program in the first desktop to execute the target step corresponding to the target task information according to the target operation step information. In some embodiments, the second control module 1420 is used to execute the above operation S420, which will not be described in detail here.

[0124] According to an embodiment of the present disclosure, the second control module 1410 may include an identification module 1510 , a first obtaining module 1520 , an input module 1530 , and a second obtaining module 1540 .

[0125] The identification module 1510 is used to identify the control information in the first desktop. In some embodiments, the identification module 1510 is used to perform the aforementioned operation S610, which will not be described in detail here.

[0126] The first obtaining module 1420 is used to generate a third desktop based on the second desktop, and the third desktop and the second desktop at least partially share the same coordinate system information. In some embodiments, the first obtaining module 1520 is used to perform the above operation S620, which is not described in detail here.

[0127] The input module 1530 is used to generate a third desktop based on the second desktop, and the third desktop and the second desktop at least partially share the same coordinate system information. In some embodiments, the input module 1530 is used to perform the aforementioned operation S630, which is not described in detail here.

[0128] The second obtaining module 1540 is used to generate a third desktop based on the second desktop, and the third desktop and the second desktop at least partially share the same coordinate system information. In some embodiments, the second obtaining module 1540 is used to perform the above-mentioned operation S640, which is not described in detail here.

[0129] According to the embodiment of the present disclosure, the target step is the first step, and the task execution device 1200 may further include a third acquisition module 1610 , a fourth acquisition module 1620 , and a third control module 1630 .

[0130] The third acquisition module 1610 is used to acquire the updated screenshot of the third desktop in response to the completion of the first step. In some embodiments, the third acquisition module 1610 is used to perform the aforementioned operation S710, which will not be described in detail herein.

[0131] The fourth obtaining module 1620 is used to obtain the second operation step information for the target task according to the updated screenshot and the target task information. In some embodiments, the fourth obtaining module 1620 is used to perform the above-mentioned operation S720, which will not be described in detail here.

[0132] The third control module 1630 is used to control the target program in the first desktop to execute the second step corresponding to the target task information according to the second operation step information. In some embodiments, the third control module 1630 is used to execute the aforementioned operation S730, which will not be described in detail here.

[0133] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0134] For example, any of the start module 1210 and the control module 1220, and / or the first generation module 1310, the synchronization module 1320, the second control module 1330, and / or the first acquisition module 1410 and the second control module 1420, and / or the identification module 1510, the first acquisition module 1520, the input module 1530, the second acquisition module 1540, and / or the third acquisition module 1610, the fourth acquisition module 1620, and the third control module 1630 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the starting module 1210 and the control module 1220, and / or the first generation module 1310, the synchronization module 1320, the second control module 1330, and / or the first acquisition module 1410 and the second control module 1420, and / or the identification module 1510, the first acquisition module 1520, the input module 1530, the second acquisition module 1540, and / or the third acquisition module 1610, the fourth acquisition module 1620, and the third control module 1630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, the startup module 1210 and the control module 1220, and / or the first generation module 1310, the synchronization module 1320, the second control module 1330, and / or the first acquisition module 1410 and the second control module 1420, and / or the identification module 1510, the first acquisition module 1520, the input module 1530, the second acquisition module 1540, and / or at least one of the third acquisition module 1610, the fourth acquisition module 1620, and the third control module 1630 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0135] It should be noted that the data processing system part in the embodiments of the present disclosure corresponds to the data processing method part in the embodiments of the present disclosure. The description of the data processing system part specifically refers to the data processing method part, which will not be repeated here.

[0136] Fig.13A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0137] like Fig.13 As shown, the electronic device 1700 according to an embodiment of the present disclosure includes a processor 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage part 1708 to a random access memory (RAM) 1703. The processor 1701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1701 may also include an onboard memory for caching purposes. The processor 1701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0138] In RAM 1703, various programs and data required for the operation of electronic device 1700 are stored. Processor 1701, ROM 1702 and RAM 1703 are connected to each other through bus 1704. Processor 1701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1702 and / or RAM 1703. It should be noted that the program can also be stored in one or more memories other than ROM 1702 and RAM 1703. Processor 1701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0139] According to an embodiment of the present disclosure, the electronic device 1700 may further include an input / output (I / O) interface 1705, which is also connected to the bus 1704. The electronic device 1700 may further include one or more of the following components connected to the input / output (I / O) interface 1705: an input portion 1706 including a keyboard, a mouse, etc.; an output portion 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1708 including a hard disk, etc.; and a communication portion 1709 including a network interface card such as a LAN card, a modem, etc. The communication portion 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the input / output (I / O) interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed, so that a computer program read therefrom is installed into the storage portion 1708 as needed.

[0140] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1709, and / or installed from the removable medium 1711. When the computer program is executed by the processor 1701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0142] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0143] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1702 and / or the RAM 1703 described above and / or one or more memories other than the ROM 1702 and the RAM 1703 .

[0144] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the task execution method provided by the embodiment of the present disclosure.

[0145] When the computer program is executed by the processor 1701, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0146] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1709, and / or installed from a removable medium 1711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0147] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0149] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A task execution method, comprising: In response to receiving the target task information, starting the target program in a first desktop, wherein the first desktop and a second desktop currently being used by the user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; When the second desktop is displayed, the target program is controlled to execute the target task corresponding to the target task information through the first desktop.

2. The method according to claim 1, wherein, through the first desktop, controlling the target program to execute the target task corresponding to the target task information comprises: Based on the second desktop, generate a third desktop, wherein the third desktop and the second desktop at least partially share the same coordinate system information; Synchronize the screen in the first desktop to the third desktop; According to the screenshot of the third desktop, the target program in the first desktop is controlled to execute the target task corresponding to the target task information.

3. The method according to claim 2, wherein, according to the screenshot of the third desktop, controlling the target program in the first desktop to execute the target task corresponding to the target task information comprises: According to the screenshot and the target task information, target operation step information for the target task is obtained, wherein the target operation step information at least includes a target control and a first instruction in the first desktop, the target operation step information corresponds to the target program, the operating system has the first instruction and the second instruction, the first instruction and the second instruction have the same function, and the second instruction is used to perform a specific operation in the second desktop; According to the target operation step information, the target program in the first desktop is controlled to execute the target step corresponding to the target task information.

4. The method according to claim 3, wherein obtaining target operation step information for the target task based on the screenshot and the target task information comprises: Identify control information in the first desktop; Obtaining prompt information corresponding to the target task information; Inputting the control information, the prompt information and the screenshot into a target model; At least one target operation step information generated by the target model is obtained.

5. The method according to claim 3, wherein the target step is the first step, and the method further comprises: In response to completion of the execution of the first step, obtaining an updated screenshot of the third desktop; Obtaining second operation step information for the target task according to the updated screenshot and the target task information; According to the second operation step information, the target program in the first desktop is controlled to execute the second step corresponding to the target task information.

6. The method according to claim 1, further comprising: In the case where the user is required to operate the target program, query information is generated on the second desktop, where the query information is used to inquire whether to switch from displaying the second desktop to displaying the first desktop. 7 . The method according to claim 1 , wherein the target task information is information input by the target user on the second desktop.

8. A task execution device, comprising: a starting module, configured to start a target program in a first desktop in response to receiving target task information, wherein the first desktop and a second desktop currently being used by the user belong to the same operating system, and the first desktop and the second desktop are not displayed at the same time; and A control module is used to control the target program to execute the target task corresponding to the target task information through the first desktop when the second desktop is displayed.

9. An electronic device, comprising: at least one processor; as well as A memory connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images