Operation response method and device, terminal and storage medium

Abstract

The embodiment of the invention discloses an operation response method and device, a terminal and a storage medium, and belongs to the technical field of barrier-free services. The method comprises the steps that under the condition that an execution instruction of an automation task is received, the automation task is executed in a container, the container is independent of a first window, and the automation task is a task simulating real user operation on an interface; in the process of executing the automatic task, under the condition that user operation in the first window is received, responding to the user operation. By adopting the scheme provided by the embodiment of the invention, the user operation can be responded while the automatic task is executed, so that the automatic task and the user operation coexist, and the user operation is prevented from being limited.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of barrier-free service technology, and in particular to an operation response method, device, terminal and storage medium. Background Art

[0002] As an auxiliary function, accessibility services can monitor and respond to changes in the user interface and perform automated tasks such as clicking and sliding that simulate user operations.

[0003] In the related art, the terminal directly executes the automated task in the main window, so that during the execution of the automated task, the terminal can no longer respond to user operations, resulting in limited user operations and affecting user experience. Summary of the invention

[0004] The present application provides an operation response method, device, terminal and storage medium, and the technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides an operation response method, the method comprising:

[0006] When receiving an execution instruction of an automated task, executing the automated task in a container, the container being independent of the first window, the automated task being a task simulating a real user operation on an interface;

[0007] During the execution of the automated task, upon receiving a user operation in the first window, respond to the user operation.

[0008] On the other hand, an embodiment of the present application provides an operation response device, the device comprising:

[0009] A task execution module, configured to execute the automation task in a container when receiving an execution instruction of the automation task, wherein the container is independent of the first window, and the automation task is a task that simulates a real user operation on the interface;

[0010] The operation response module is used to respond to the user operation when receiving the user operation in the first window during the execution of the automation task.

[0011] On the other hand, an embodiment of the present application provides a terminal, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the operation response method described in the above aspects.

[0012] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer instruction is stored. The at least one computer instruction is loaded and executed by a processor to implement the operation response method described in the above aspects.

[0013] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the operation response method provided in the above aspect.

[0014] Different from the related art, the automated task is directly executed in the main window, which results in the terminal being unable to respond to user operations during the execution of the automated task. In the embodiment of the present application, when an execution instruction of the automated task is received, a container independent of the first window is created, and the automated task is executed in the container, so that when a user operation in the first window is received, the user operation can still be responded to, thereby achieving the coexistence of executing the automated task and responding to the user operation, avoiding the restriction of the user operation when executing the automated task, improving the flexibility of the operation response, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of an operation response method provided by an exemplary embodiment of the present application is shown;

[0017] Figure 2 A flowchart of an operation response method provided by another exemplary embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of an interface for executing an automated task in a second window provided by an exemplary embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram of an interface in which an automated task is interrupted provided by an exemplary embodiment of the present application is shown;

[0020] Figure 5 A schematic diagram of an interface in which an automated task is not interrupted provided by an exemplary embodiment of the present application is shown;

[0021] Figure 6 A flowchart of an operation response method provided by another exemplary embodiment of the present application is shown;

[0022] Figure 7 A schematic diagram of an interface for executing an automated task in a second window provided by another exemplary embodiment of the present application is shown;

[0023] Figure 8 A schematic diagram of an interface for executing an automated task in a virtual screen provided by an exemplary embodiment of the present application is shown;

[0024] Fig. 9 An interactive timing diagram of an operation response provided by an exemplary embodiment of the present application is shown;

[0025] Fig.10 A structural block diagram of an operation response device provided by an exemplary embodiment of the present application is shown;

[0026] Fig.11 A structural block diagram of a terminal provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] First, the nouns involved in the embodiments of the present application are introduced:

[0029] Automated tasks: refers to tasks that simulate real user operations on the interface. That is, in the process of executing automated tasks, it is allowed to listen to and respond to interface events, as well as simulate user operations, so as to help users use the device better.

[0030] Container: It can be understood as an interface component. A container contains multiple interface elements, such as buttons, text boxes, labels, etc. The function of a container is to organize and manage interface elements. The container can be in the form of a window, panel, frame, virtual screen, etc.

[0031] Window: refers to the user interface window, which is the main entry point for users to interact with the application. Optionally, a window usually contains navigation elements, menus, toolbars, and other user interface components. In the case of multiple windows, there can be a main window and multiple sub-windows, and the main window may also be responsible for managing other sub-windows.

[0032] Please refer to Figure 1 , which shows a flowchart of an operation response method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a terminal as an example. The method includes the following steps:

[0033] Step 101, upon receiving an execution instruction of an automated task, executing the automated task in a container, the container being independent of the first window, the automated task being a task simulating a real user operation on an interface.

[0034] Optionally, the automated task is a task that simulates real user operations on the interface. That is, in the process of executing the automated task, it is allowed to listen to and respond to interface events, and simulate user operations, so as to help users better use the device.

[0035] Optionally, the execution instruction of the automated task may be a voice instruction, such as "open application A to send a message to B", "open application C to play video D", etc., or it may be a scheduled task instruction, such as sending a message to A at a certain time, playing a weather forecast at a certain time, etc., or it may be other instruction forms, which are not limited to the embodiments of the present application.

[0036] Different from the related art, when receiving the execution instruction of the automated task, the terminal directly executes the automated task in the main window, so that it cannot respond to user operations during the execution of the automated task, resulting in limited user operations. In the embodiment of the present application, when receiving the execution instruction of the automated task, the terminal does not directly execute the automated task in the first window, but creates a container independent of the first window.

[0037] Optionally, the first window may be a main window; in the case of multiple windows, it may also be any window in the multiple windows. Optionally, the main window refers to the main user interface window, which is the main entry point for users to interact with the application. Optionally, the main window usually contains navigation elements, menus, toolbars and other user interface components. In the case of multiple windows, the main window may also be responsible for managing other sub-windows.

[0038] Optionally, a container can be understood as an interface component, which contains multiple interface elements, such as buttons, text boxes, labels, etc. The function of a container is to organize and manage interface elements, and the container can be in the form of a window, panel, frame, virtual screen, etc.

[0039] In a possible implementation, the terminal can determine the interface elements contained in the container according to the application corresponding to the automation task, and set the container parameters, such as the container size, display type, position, etc., so as to create a container according to the container parameters and the interface elements, and execute the automation task in the container. That is, since the container is independent of the first window, the first window will not be occupied during the execution of the automation task.

[0040] Regarding the process of executing automated tasks, in one possible implementation, the terminal can call a specific API (Application Programming Interface) to listen to and obtain the interface content in the container, thereby simulating user operations through the API, interacting with the interface content, and executing automated tasks.

[0041] Schematically, taking the automated task of "opening application A to send a message to B" as an example, the terminal first opens application A in the container, and finds the dialogue entry corresponding to B from the application interface, thereby simulating the user's click operation, clicking the dialogue entry, entering the dialogue interface corresponding to B, and then simulating user input in the dialogue interface, entering the corresponding message content, and simulating clicking the send button to send a message to B.

[0042] In a possible implementation, when receiving an execution instruction of an automated task, an application is running in the first window, and the application is also an application corresponding to the automated task. The terminal may first close the application running in the first window, and then start executing the automated task in the container.

[0043] Illustratively, the first window currently runs application A. When receiving the automated task of "open application A to send a message to B", the terminal first closes application A in the first window, and then executes the automated task in the container to open application A.

[0044] Step 102: during the process of executing the automated task, upon receiving a user operation in the first window, responding to the user operation.

[0045] In the embodiment of the present application, since the automated task is executed in a container independent of the first window, the terminal can still respond to user operations in the first window during the execution of the automated task. In some embodiments, during the execution of the automated task, when a user operation in the first window is received, the terminal responds to the user operation.

[0046] Optionally, the user operation may be a click operation, a slide operation, a double-click operation, a long press operation, etc., which is not limited in the embodiments of the present application.

[0047] Illustratively, in the process of executing the automated task of "opening application A to send a message to application B" in the container, upon receiving a click operation by the user on the application identifier corresponding to application C in the first window, the terminal responds to the click operation, opens application C in the first window, and displays the application interface corresponding to application C, and then continues to respond to the user's operations in application C.

[0048] Optionally, in order to ensure the coexistence of executing automated tasks and responding to user operations, the terminal may also start multiple threads to respectively execute automated tasks in the container and respond to user operations in the first window, thereby improving the efficiency of task execution and operation response.

[0049] Optionally, when the user operation indicates to open a certain application, the application corresponding to the user operation may not be the same as the application involved in the automation task. In a possible implementation, when starting to execute the automation task, the terminal first determines each application indicated in the automation task and closes the running permission of each application in the first window. After the automation task is executed, the terminal then opens the running permission of each application in the first window.

[0050] That is, during the execution of the automation task, the terminal cannot respond to the opening operation of the application involved in the automation task in the first window, and the same application cannot be run in the container and the first window at the same time.

[0051] In an illustrative manner, the execution of an automated task requires the sequential launch of Application A and Application B. Therefore, when the automated task is started, the terminal first closes the running permissions of Application A and Application B in the first window, and then executes the automated task. That is, during the execution of the automated task, the user cannot open Application A and Application B in the first window. Then, when the automated task is completed, the terminal reopens the running permissions of Application A and Application B in the first window.

[0052] To summarize, in an embodiment of the present application, when an execution instruction of an automated task is received, a container independent of the first window is created and the automated task is executed in the container. Thus, when a user operation in the first window is received, the user operation can still be responded to, thereby achieving the coexistence of executing automated tasks and responding to user operations, avoiding restrictions on user operations when executing automated tasks, improving the flexibility of operation responses, and enhancing user experience.

[0053] In some embodiments, during the execution of an automated task, in order to display the task execution process to the user so that the user can view the execution status of the automated task in real time and ensure the security of user information, the container can be a second window displayed independently of the first window.

[0054] Optionally, when the first window is the main window, the second window may be a floating window. Optionally, a floating window refers to a floating window that is displayed above the main window. Optionally, the window size of the floating window is smaller than the window size of the main window, and the window size of the floating window may also be adjusted based on user operations.

[0055] Optionally, the user can drag the floating window to adjust the display position of the floating window. Optionally, the user can trigger a floating window closing control corresponding to the floating window to close the floating window.

[0056] In some embodiments, in the case of multiple windows, in order to ensure the orderliness of operation response, an interactive focus is usually set, and the window where the interactive focus is located is determined as the active window. In the single-focus interaction mode, the terminal can directly respond to user operations in the active window, while for the inactive window, it is necessary to first seize the interactive focus from the active window and convert it to the active window before the terminal can respond to the user operation.

[0057] Optionally, the interaction focus (Focus) refers to the element currently being interacted with, and the interaction focus determines which element or window can respond to the user operation. Optionally, the terminal can determine the window where the current interaction focus is located according to the Focus event.

[0058] In a possible implementation, when a second window is created to execute an automated task, there are at least two windows, the second window and the first window. In order to achieve coexistence of executing automated tasks and responding to user operations, it is necessary to time-share the interactive focus between the two windows.

[0059] The process of executing the automation task through the second window will be described below through a specific embodiment.

[0060] Please refer to Figure 2 , which shows a flowchart of an operation response method provided by another exemplary embodiment of the present application. This embodiment is described by taking the method used in a terminal as an example. The method includes the following steps:

[0061] Step 201: upon receiving an execution instruction of an automation task, create a second window.

[0062] In a possible implementation, upon receiving an execution instruction of an automated task, the terminal starts the application according to the application indicated by the automated task and obtains the interface content corresponding to the application, thereby creating a second window with first attributes according to the interface content and window parameters.

[0063] Optionally, the second window may be displayed in a split screen with the first window, and the window size of the second window may be the same as or different from the window size of the first window.

[0064] Optionally, the second window can also be displayed floating above the first window, the window size of the second window is smaller than the window size of the first window, and the window size of the second window can be adjusted based on user operations. Optionally, the user can adjust the display position of the second window based on a drag operation, and can also close the second window based on a window closing operation.

[0065] The second window with the first attribute needs to actively seize the interaction focus from other windows during the process of executing the automation task in the terminal.

[0066] Optionally, in order to execute the automated task in the second window with the first attribute, the second window needs to obtain the interactive focus first, and in order to ensure the coexistence of executing the automated task and responding to the user operation, the first window also needs to obtain the interactive focus first when responding to the user operation in the first window. Therefore, in the process of executing the automated task, the second window and the first window need to seize the interactive focus from each other. That is, in order to execute the automated task in the second window with the first attribute, the second window needs to actively seize the interactive focus.

[0067] Step 202: Execute the automation task in the second window.

[0068] In a possible implementation, after receiving an execution instruction of an automated task and creating a second window with a first attribute, the second window may first seize the interactive focus, so that when the interactive focus is located in the second window, the terminal executes the automated task in the second window.

[0069] Optionally, in order for the terminal to immediately execute the automated task after the second window with the first attribute is created, it can be set that when the second window is created and opened, the second window automatically seizes the interactive focus. That is, when the second window with the first attribute is created, if no user operation in other windows is received, the interactive focus is directly placed on the second window.

[0070] Optionally, when the interaction focus is on the second window, the terminal can execute the automation task in the second window according to the interface content in the second window and the operation sequence corresponding to the automation task, and the interface content in the second window changes with the execution of the automation task.

[0071] Optionally, the terminal may execute the automation task in the second window through an accessibility service. Optionally, in the process of executing the automation task, the accessibility service may stably access and operate each control in the second window.

[0072] Optionally, the terminal may execute the automation script corresponding to the automation task in the second window through the automation script execution engine.

[0073] Indicatively, Figure 3As shown, taking the automated task of "opening application A to send a message to B", the first window is the main window, and the second window is the floating window as an example, when receiving the execution instruction of the automated task, the terminal creates a second window 302, which is displayed independently of the first window 301, and executes the automated task in the second window 302.

[0074] Step 203: When the interaction focus is located in the second window and a user operation in the first window is received, the interaction focus is transferred from the second window to the first window.

[0075] In a possible implementation, in order to respond to user operations in the first window, when the interaction focus is in the second window and a user operation in the first window is received, the terminal needs to transfer the interaction focus from the second window to the first window, that is, to convert the first window from an inactive state to an active state.

[0076] Illustratively, when the interaction focus is located in the second window and a click operation is received in the first window, the terminal transfers the interaction focus from the second window to the first window, and the first window becomes the active window.

[0077] Step 204: When the first window obtains the interactive focus, respond to the user operation in the first window.

[0078] In a possible implementation manner, when the first window obtains the interactive focus, that is, the first window is an active window, the terminal can directly respond to the user operation in the first window.

[0079] Taking the user clicking a control in the first window as an example, the process of the terminal responding to the click operation can be divided into two steps. The first step is to convert the first window into an active window according to the first click operation; the second step is to respond to the second click operation on the target control.

[0080] Indicatively, Figure 3 As shown, during the process of the terminal executing the automated task in the second window 302, upon receiving a click operation from the user on the photo-taking application in the first window 301, the terminal transfers the interactive focus from the second window 302 to the first window 301, and opens the photo-taking application in response to the user's click operation on the photo-taking application icon.

[0081] Optionally, in order to prompt the user that the interactive focus has shifted, the terminal may also set different display effects for the first window and the second window. For example, when the first window obtains the interactive focus, the window brightness of the first window is higher than the window brightness of the second window, or the window edge of the first window displays an animation effect, etc., which is not limited in the embodiments of the present application.

[0082] As the interactive focus shifts, the first window becomes the active window and the second window becomes the inactive window (i.e., the second window loses focus). At this time, the second window cannot respond to the interactive operations in the automated task. However, considering that different interactive operations are implemented in different ways, for example, some interactive operations are atomic (will not be interrupted during the execution of the interactive operation), the automated task in the second window can continue to execute even if the second window is out of focus.

[0083] In a possible implementation, when the interaction focus is transferred from the second window to the first window and the interaction operation is simulated in the second window in the first manner, the terminal suspends the interaction operation in the second window.

[0084] Optionally, the interactive operation simulated by the first method is interrupted when the second window loses the interactive focus. Optionally, the first method may be a coordinate click method.

[0085] Optionally, the coordinate click method refers to first determining the screen coordinates corresponding to the interactive operation, and then simulating the interactive operation by clicking the screen coordinates. For example, in the process of executing an automated task, when the coordinate click method is used to simulate the click control operation, the terminal can first determine the screen coordinates where the control is located, and click the screen coordinates where the control is located through dispatchGesture, thereby simulating the click control operation.

[0086] Schematically, when the second window is out of focus, the process of clicking the control needs to be divided into two steps. The first step is to activate the second window, and the second step is to pass the click event to the target control. However, there is a response time between the first step and the second step. Between the first step and the second step, once the user operation in the first window is received, the process will be interrupted and the second window will not be able to obtain the interactive focus. At this time, the interactive focus is still obtained by the first window.

[0087] Furthermore, after the interactive operation simulated by the first method is interrupted, in order to regain the interactive focus so as to continue to execute the automated task in the second window with the first attribute, the terminal also needs to determine in real time whether a user operation is received. In a possible implementation, when the interactive focus is located in the first window and no user operation in the first window is received, the terminal transfers the interactive focus from the first window to the second window, thereby continuing to simulate the interactive operation in the first method in the second window.

[0088] Indicative, such as Figure 4As shown, taking the first window as the main window, the second window as the floating window, and the automated task as "open application A to send a message to B" as an example, when receiving the execution instruction of the automated task, the terminal creates a second window 402, which is displayed independently of the first window 401, and executes the automated task in the second window 402. When receiving the user's click operation on the camera application in the first window 401, the terminal transfers the interactive focus from the second window 402 to the first window 401, and at this time, the terminal simulates the execution of the control trigger operation by coordinate clicking in the second window 402 to send a message to the friend, but because the interactive focus is transferred to the first window 401, the terminal can only suspend the execution of the automated task.

[0089] In another possible implementation, when the interaction focus is transferred from the second window to the first window and the interaction operation is simulated in the second window in the second manner, the terminal can continue to perform the interaction operation in the second window.

[0090] Optionally, the interactive operation simulated by the second mode is not interrupted when the second window loses the interactive focus. Optionally, the second mode refers to a mode of calling an atomic operation interface.

[0091] Optionally, during the execution of the automated task, the terminal can perform interactive operations by calling the atomic operation interface performAction. For example, performAction(ACTION_CLICK) to perform a click operation, performAction(ACTION_LONG_CLICK) to perform a long press operation, performAction(SCROLL_FORWARD) to perform a forward scroll operation, performAction(ACTION_SET_TEXT) to perform a text input operation, and so on.

[0092] That is, when an atomic operation interface is used to perform interactive operations, even if a user operation is received in the first window and the interactive focus is transferred from the second window to the first window, the currently executed interactive operation will not be interrupted by the transfer of the interactive focus.

[0093] Indicatively, Figure 5As shown, taking the first window as the main window, the second window as the floating window, and the automated task as "open application A to send a message to B" as an example, when receiving the execution instruction of the automated task, the terminal creates a second window 502, which is displayed independently of the first window 501, and executes the automated task in the second window 502. When receiving the user's click operation on the camera application in the first window 501, the terminal transfers the interactive focus from the second window 502 to the first window 501, but at this time, the terminal simulates the execution of the control trigger operation (performAction(ACTION_CLICK)) in the second window 502 by calling the atomic operation interface to send a message to the friend. Therefore, even if the interactive focus is transferred to the first window 501, the terminal can still continue to execute the automated task and simulate the user inputting the message text in the second window 502.

[0094] In the above embodiment, by executing the automated task in the second window, the execution process of the automated task can be displayed to the user in the foreground, and the second window with the first attribute can respond to user operations in the first window while executing the automated task in the second window by competing with the first window for interaction focus, thereby improving the flexibility of operation response.

[0095] In some embodiments, in order to prevent the dynamic preemption of the interactive focus from affecting the execution process of the automated task, the window attributes of the second window can be directly modified to create a second window with the second attribute, so that the terminal can still execute the automated task in the second window without preempting the interactive focus. That is, in the process of executing the automated task, the first window does not need to preempt the interactive focus from the second window and directly responds to the user operation.

[0096] Please refer to Figure 6 , which shows a flowchart of an operation response method provided by another exemplary embodiment of the present application. This embodiment is described by taking the method used in a terminal as an example. The method includes the following steps:

[0097] Step 601: upon receiving an execution instruction of an automation task, create a second window.

[0098] In a possible implementation, upon receiving an execution instruction of an automated task, the terminal starts the application according to the application indicated by the automated task and obtains the interface content corresponding to the application, thereby creating a second window with second attributes according to the interface content and window parameters.

[0099] Optionally, the second window may be displayed in a split screen with the first window, and the window size of the second window may be the same as or different from the window size of the first window.

[0100] Optionally, the second window can also be displayed floating above the first window, the window size of the second window is smaller than the window size of the first window, and the window size of the second window can be adjusted based on user operations. Optionally, the user can adjust the display position of the second window based on a drag operation, and can also close the second window based on a window closing operation.

[0101] Optionally, the second window with the second attribute does not actively seize the interactive focus during the execution of the automated task, that is, the terminal can directly execute the automated task in the second window when the second window does not have the interactive focus.

[0102] It should be noted that, although the second window with the second attribute does not actively seize the interaction focus from other windows during the execution of the automated task, upon receiving a trigger operation on the second window by the user, the terminal can also transfer the interaction focus from other windows to the second window.

[0103] In order to implement the execution of automated tasks in the second window without interactive focus, the terminal can directly create a second window with a PIP (Picture In Picture) attribute, wherein the PIP attribute supports the second window to execute automated tasks without obtaining interactive focus.

[0104] Optionally, the terminal may modify the Android Framework layer code to create a second window with a PIP attribute, so that when the automated task is executed in the second window, there is no need to seize the interactive focus.

[0105] Step 602, executing the automation task in the second window.

[0106] Different from the case where the second window with the first attribute is created, the second window automatically seizes the interactive focus when it is started. In a possible implementation, after receiving the execution instruction of the automated task and creating the second window with the second attribute, the terminal can directly execute the automated task in the second window according to the interface content in the second window and the operation sequence corresponding to the automated task, and the interface content in the second window changes with the execution of the automated task. That is, during the execution of the automated task in the second window, the terminal does not need to consider whether the second window obtains the interactive focus.

[0107] Optionally, the terminal may execute the automated task in the second window through the barrier-free service. Optionally, in the process of executing the automated task, the barrier-free service may stably access and operate each control in the second window.

[0108] Step 603: during the process of executing the automated task, upon receiving a user operation in the first window, respond to the user operation.

[0109] In a possible implementation, during the execution of the automated task, since the interaction focus is not occupied by the second window, that is, the interaction focus is still located in the first window, when a user operation in the first window is received, the terminal can also directly respond to the user operation.

[0110] Indicatively, Figure 7 As shown, taking the first window as the main window, the second window as the floating window, and the automated task as "open application A to send a message to application B" as an example, when receiving the execution instruction of the automated task, the terminal creates a second window 702, which has a PIP attribute and is displayed independently of the first window 701. And the second window 702 does not need to seize the interactive focus, and the terminal can execute the automated task in the second window 702. Then, when receiving the user's click operation on the camera application in the first window 701, the terminal directly responds to the click operation in the first window 701 and opens the camera application.

[0111] At the same time, the terminal continues to execute the automated task in the second window 702 based on the operation sequence corresponding to the automated task, including searching for friend B in application interface A, clicking to enter friend B's chat interface, simulating the user to input message text in the chat interface, and sending the message text to friend B.

[0112] In the above embodiment, by executing the automated task in the second window, the execution process of the automated task can be displayed to the user in the foreground, and by setting the second window to the PIP attribute, the automated task can be executed in the second window without occupying the interactive focus, thereby avoiding mutual interference between the automated task and the user operation, allowing the user to smoothly perform the user operation in the first window, further optimizing the user experience, and improving the responsiveness and interactivity of the first window.

[0113] In some embodiments, considering that when the second window is displayed in the foreground, it may block part of the interface content in the first window, or occupy the display area of ​​the first window, thereby affecting the user's operation in the first window, in order to further optimize the response to the user's operation in the first window, the terminal can also set the container in the form of a virtual screen.

[0114] In a possible implementation, when receiving an execution instruction of an automation task, the terminal creates a virtual screen, which is independent of the first window and is in an invisible state, and then the terminal executes the automation task in the virtual screen.

[0115] Optionally, the virtual screen refers to a virtual interface created in the background. Optionally, the terminal starts the application in the background according to the application corresponding to the automation task, and obtains the interface content corresponding to the application, thereby creating a virtual screen according to the interface content and the virtual screen parameters.

[0116] In a possible implementation, considering that when executing an automated task in a virtual screen, the task execution process is not visible to the user, which may pose a potential information security risk, in order to facilitate the user to timely view the execution progress of the automated task, during the execution of the automated task, the terminal may also display a task execution prompt, which is used to prompt the user that the automated task is currently being executed. Furthermore, when the automated task is completed, the terminal cancels the display of the task execution prompt.

[0117] Optionally, the task execution prompt may include the application corresponding to the automation task, the task content, and the task execution progress, etc. Optionally, the task execution prompt may include a program icon of the application, a task progress bar, etc., which is not limited in the embodiment of the present application.

[0118] Indicatively, Figure 8 As shown, taking the automated task of "opening application A to send a message to B" as an example, upon receiving the execution instruction of the automated task, the terminal creates a virtual screen 802, which is independent of the first window 801 and runs in the background in an invisible state.

[0119] Then, the terminal executes automated tasks in the virtual screen 802, including searching for friend B in the A application interface, clicking to enter the chat interface of friend B, simulating the user to input message text in the chat interface, and sending the message text to friend B.

[0120] At the same time, when receiving a click operation of the user on the photo taking application in the first window 801, the terminal directly responds to the click operation and starts the photo taking application in the first window 801. In order to remind the user that the current automated task is being executed, the terminal displays a task execution prompt 803 in the first window 801, so that the user can understand the execution progress of the automated task at any time.

[0121] In the above embodiment, by executing the automated task in the virtual screen, the automated task is directly executed in the background without affecting the first window displayed in the foreground, thereby optimizing the user operation response process in the first window. In addition, during the process of executing the automated task in the background, by displaying the task execution prompt, it is convenient for the user to directly understand the execution status of the automated task, thereby ensuring the execution security of the background automated task.

[0122] Regarding the process of executing automated tasks in a container, in some embodiments, upon receiving an execution instruction for an automated task, the terminal first obtains an operation sequence corresponding to the automated task, where the operation sequence includes at least one interactive operation and an operation order of different interactive operations. Then, the terminal can execute the automated task in the container according to the interface content and the operation sequence in the container.

[0123] Illustratively, taking the automated task of "opening application A to send a message to application B" as an example, the operation sequence includes starting application A, finding the chat box corresponding to application B, entering the chat interface corresponding to application B, entering the message text, and sending the message text.

[0124] In a possible implementation, when the current interaction operation is a control interaction operation, the terminal first determines an interaction control corresponding to the control interaction operation, and then when it is determined that the interaction control is located in the container, the terminal triggers the interaction control.

[0125] That is, for a control interaction operation, the terminal first needs to determine the interface position corresponding to the interactive control, and then trigger the interactive control based on the interface position of the interactive control.

[0126] Optionally, in the process of searching for interactive controls, the terminal may start searching for interactive controls from the window where the interactive focus is located.

[0127] In a possible implementation, when the current interaction operation is a control interaction operation and the interaction focus is in the container, the terminal directly searches for and determines the interaction control corresponding to the control interaction operation in the container.

[0128] In another possible implementation, when the current interactive operation is a control interactive operation and the interactive focus is not in the container, the terminal needs to first search for the interactive control in the window where the interactive focus is located, but obviously the interactive control will not be found. Therefore, when the interactive control is not found in the window where the interactive focus is located, the terminal needs to expand the control search range and search for the interactive control globally, so as to search and determine the interactive control corresponding to the control interactive operation in the container.

[0129] Optionally, in order to improve the efficiency of control search and avoid global search, the terminal may not use the window where the interactive focus is located as the starting point, but directly search for interactive controls in the container.

[0130] In combination with the above embodiment, when the automated task is executed in the second window, the terminal directly searches for the interactive control in the second window; when the automated task is executed on the virtual screen, the terminal directly searches for the interactive control on the virtual screen.

[0131] In the above embodiment, in the process of executing an automated task, when the automated task involves control interaction operations, based on the control search strategy, the window where the interaction focus is located is used as the search starting point to search for interactive controls, and based on the control search situation, the search is expanded to a global search, thereby ensuring the orderliness and accuracy of the control search.

[0132] In addition, without using the window where the interactive focus is located as the search starting point, the interactive control is directly searched from the container as the starting point, which further improves the efficiency of control search.

[0133] Please refer to Fig. 9 , which shows an interactive sequence diagram of an operation response provided by an exemplary embodiment of the present application. This embodiment takes the container as a second window (floating window) with a first attribute, and uses an automated script execution engine to execute an automated task as an example to illustrate the process of the operation response.

[0134] like Fig. 9 As shown, when receiving an execution instruction of an automated task issued by a user, the terminal sends the execution instruction to the automated script execution engine, and the automated script execution engine requests the window management to create a second window with a first attribute, and the second window is displayed independently of the first window. Therefore, when the second window is successfully created, the second window is displayed floating on the upper layer of the first window.

[0135] Optionally, the window size of the second window is smaller than the window size of the first window, and the window size of the second window can be adjusted based on user operations. Optionally, the user can adjust the display position of the second window based on a drag operation, or close the second window based on a window closing operation.

[0136] Optionally, the interface content in the second window is determined based on a program interface of an application program indicated by the automation task.

[0137] After creating the second window with the first attribute, the terminal transfers the interactive focus to the second window and executes the automation task in the second window through the automation script execution engine.

[0138] Optionally, the automated script execution engine can call the accessibility service to obtain the interface content in the second window, and search for the interactive control corresponding to the control interaction operation according to the automated script corresponding to the automated task. Thus, when the interactive control in the second window is determined, the automated script execution engine calls the accessibility service, performs the interactive operation through the atomic operation interface, and returns the corresponding interactive operation result.

[0139] During the execution of the automated task, upon receiving a user operation in the first window, the terminal transfers the interaction focus from the second window to the first window, and responds to the user operation in the first window when the first window obtains the interaction focus.

[0140] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0141] Please refer to Fig.10 , which shows a structural block diagram of an operation response device provided by an exemplary embodiment of the present application, the device includes:

[0142] The task execution module 1001 is used to execute the automation task in a container when receiving an execution instruction of the automation task, wherein the container is independent of the first window, and the automation task is a task that simulates a real user operation on the interface;

[0143] The operation response module 1002 is used to respond to the user operation when receiving the user operation in the first window during the execution of the automation task.

[0144] Optionally, the task execution module 1001 includes:

[0145] a window creation unit, configured to create a second window upon receiving the execution instruction of the automation task, wherein the second window is displayed independently of the first window;

[0146] The first task execution unit is used to execute the automation task in the second window.

[0147] Optionally, the window creation unit is used to:

[0148] creating the second window with the first attribute, wherein the second window with the first attribute actively seizes the interactive focus during the execution of the automated task; or,

[0149] The second window having a second attribute is created, wherein the second window having the second attribute does not actively seize the interactive focus during the execution of the automated task.

[0150] Optionally, when creating the second window having the first attribute, the first task execution unit is configured to:

[0151] In case the interaction focus is located in the second window, the automated task is performed in the second window.

[0152] Optionally, the operation response module 1002 is used to:

[0153] When the interaction focus is located in the second window and the user operation in the first window is received, transferring the interaction focus from the second window to the first window;

[0154] When the first window obtains the interactive focus, respond to the user operation in the first window.

[0155] Optionally, the device further comprises:

[0156] An operation suspension module is used to suspend the execution of the interactive operation in the second window when the interactive focus is transferred from the second window to the first window and the interactive operation is simulated in the second window using a first method, wherein the interactive operation simulated in the first method is interrupted when the second window loses the interactive focus.

[0157] Optionally, the device further comprises:

[0158] a focus transfer module, configured to transfer the interaction focus from the first window to the second window when the interaction focus is located in the first window and the user operation in the first window is not received;

[0159] The first operation execution module is used to continue to simulate and execute the interactive operation in the first manner in the second window.

[0160] Optionally, the device further comprises:

[0161] A second operation execution module is used to continue executing the interactive operation in the second window when the interactive focus is transferred from the second window to the first window and the interactive operation is simulated in the second window using a second method, wherein the interactive operation simulated in the second method is not interrupted when the second window loses the interactive focus.

[0162] Optionally, the window creation unit is further used to:

[0163] The second window is created with a PIP attribute, wherein the PIP attribute enables the second window to perform the automated task without obtaining the interactive focus.

[0164] Optionally, the task execution module 1001 includes:

[0165] A virtual screen creating unit, configured to create a virtual screen upon receiving the execution instruction of the automation task, wherein the virtual screen is in an invisible state;

[0166] The second task execution unit is used to execute the automation task in the virtual screen.

[0167] Optionally, the device further comprises:

[0168] A prompt display module, used for displaying a task execution prompt during the execution of the automation task, wherein the task execution prompt is used to prompt that the automation task is currently being executed;

[0169] The prompt display module is used to cancel the display of the task execution prompt when the automation task is completed.

[0170] Optionally, the task execution module 1001 includes:

[0171] A sequence acquisition unit, configured to acquire an operation sequence corresponding to the automation task upon receiving the execution instruction of the automation task, wherein the operation sequence includes at least one interactive operation and an operation sequence of different interactive operations;

[0172] The third task execution unit is used to execute the automation task in the container based on the interface content in the container and the operation sequence.

[0173] Optionally, the third task execution unit is used to:

[0174] In the case where the current interaction operation is a control interaction operation, determining an interaction control corresponding to the control interaction operation;

[0175] When the interactive control is located in the container, the interactive control is triggered.

[0176] Optionally, the third task execution unit is used to:

[0177] When the current interaction operation is the control interaction operation and the interaction focus is within the container, searching and determining the interaction control corresponding to the control interaction operation within the container;

[0178] When the current interaction operation is the control interaction operation, and the interaction focus is not in the container, and the interaction control is not found in the window where the interaction focus is located, the interaction control corresponding to the control interaction operation is searched and determined in the container.

[0179] To summarize, in an embodiment of the present application, when an execution instruction of an automated task is received, a container independent of the first window is created and the automated task is executed in the container. Thus, when a user operation in the first window is received, the user operation can still be responded to, thereby achieving the coexistence of executing automated tasks and responding to user operations, avoiding restrictions on user operations when executing automated tasks, improving the flexibility of operation responses, and enhancing user experience.

[0180] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0181] Please refer to Fig.11 , Fig.11 The terminal may include one or more of the following components: a processor 1101 and a memory 1102 .

[0182] The processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect the various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1102, and calling data stored in the memory 1102. Optionally, the processor 1101 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1101 can integrate one or more combinations of central processing unit (Central Processing Unit, CPU), image processor (Graphics Processing Unit, GPU), neural network processor (Neural-network Processing Unit, NPU) and modem. Among them, the CPU mainly processes the operating system, user interface and application program; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (Artificial Intelligence, AI) functions; the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 1101, but may be implemented by a separate chip.

[0183] The memory 1102 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1102 includes a non-transitory computer-readable storage medium. The memory 1102 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the terminal, etc.

[0184] In addition, those skilled in the art will appreciate that the structure of the terminal shown in the above drawings does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the terminal also includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, a sensor, an audio circuit, a WiFi module, a power supply, a Bluetooth module and other components, which will not be described in detail here.

[0185] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor to implement the operation response method described in the above embodiments.

[0186] According to one aspect of the present application, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the operation response method provided in various optional implementations of the above aspects.

[0187] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0188] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An operation response method, characterized in that: The method comprises: When receiving an execution instruction of an automated task, executing the automated task in a container, the container being independent of the first window, the automated task being a task simulating a real user operation on an interface; During the execution of the automated task, upon receiving a user operation in the first window, respond to the user operation.

2. The method according to claim 1, characterized in that The step of executing the automation task in the container upon receiving the execution instruction of the automation task includes: Upon receiving the execution instruction of the automation task, creating a second window, the second window being displayed independently of the first window; The automated task is executed within the second window.

3. The method according to claim 2, characterized in that The step of creating the second window comprises: creating the second window with the first attribute, wherein the second window with the first attribute actively seizes the interactive focus during the execution of the automated task; or, The second window having a second attribute is created, wherein the second window having the second attribute does not actively seize the interactive focus during the execution of the automated task.

4. The method according to claim 3, characterized in that In case of creating the second window having the first attribute, executing the automation task in the second window includes: In case the interaction focus is located in the second window, the automated task is performed in the second window.

5. The method according to claim 4, characterized in that In the process of executing the automation task, in the case of receiving a user operation in the first window, responding to the user operation includes: When the interaction focus is located in the second window and the user operation in the first window is received, transferring the interaction focus from the second window to the first window; When the first window obtains the interactive focus, respond to the user operation in the first window.

6. The method according to claim 5, characterized in that The method further comprises: When the interaction focus is transferred from the second window to the first window and the interaction operation is simulated in the second window in a first manner, the interaction operation in the second window is suspended, wherein the interaction operation simulated in the first manner is interrupted when the second window loses the interaction focus.

7. The method according to claim 6, characterized in that The method further comprises: When the interaction focus is located in the first window and the user operation in the first window is not received, transferring the interaction focus from the first window to the second window; The interactive operation is continued to be simulated and executed in the first manner in the second window.

8. The method according to claim 5, characterized in that The method further comprises: When the interaction focus is transferred from the second window to the first window and the interaction operation is simulated in the second window using a second method, the interaction operation continues to be performed in the second window, wherein the interaction operation simulated in the second method is not interrupted when the second window loses the interaction focus.

9. The method according to claim 3, characterized in that: The step of creating the second window having the second attribute comprises: The second window is created with a PIP attribute, wherein the PIP attribute enables the second window to perform the automated task without obtaining the interactive focus.

10. The method according to claim 1, characterized in that The step of executing the automation task in the container upon receiving the execution instruction of the automation task includes: In case of receiving the execution instruction of the automation task, creating a virtual screen, wherein the virtual screen is in an invisible state; The automation task is performed in the virtual screen.

11. The method according to claim 10, characterized in that The method further comprises: During the execution of the automation task, a task execution prompt is displayed, wherein the task execution prompt is used to prompt that the automation task is currently being executed; When the automated task is completed, the task execution prompt is canceled.

12. The method according to claim 1, characterized in that The step of executing the automation task in the container upon receiving the execution instruction of the automation task includes: When receiving the execution instruction of the automation task, acquiring an operation sequence corresponding to the automation task, the operation sequence including at least one interactive operation and an operation sequence of different interactive operations; The automation task is executed in the container based on the interface content in the container and the operation sequence.

13. The method according to claim 12, characterized in that The executing the automation task in the container based on the interface content in the container and the operation sequence includes: In the case where the current interaction operation is a control interaction operation, determining an interaction control corresponding to the control interaction operation; When the interactive control is located in the container, the interactive control is triggered.

14. The method according to claim 13, characterized in that In the case where the current interaction operation is a control interaction operation, determining the interaction control corresponding to the control interaction operation includes: When the current interaction operation is the control interaction operation and the interaction focus is within the container, searching and determining the interaction control corresponding to the control interaction operation within the container; When the current interaction operation is the control interaction operation, and the interaction focus is not in the container, and the interaction control is not found in the window where the interaction focus is located, the interaction control corresponding to the control interaction operation is searched and determined in the container.

15. An operation response device, characterized in that: The device comprises: A task execution module, configured to execute the automation task in a container when receiving an execution instruction of the automation task, wherein the container is independent of the first window, and the automation task is a task that simulates a real user operation on the interface; The operation response module is used to respond to the user operation when receiving the user operation in the first window during the execution of the automation task.

16. A terminal, characterized in that: The terminal includes a processor and a memory; the memory stores at least one computer instruction, and the at least one computer instruction is used to be executed by the processor to implement the operation response method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the operation response method as described in any one of claims 1 to 14.

18. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor of the terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal implements the operation response method as described in any one of claims 1 to 14.

Images