Automatic execution method for UI task, and electronic device

Abstract

The present application relates to the technical field of terminals, and provides an automatic execution method for a UI task, and an electronic device. In the method, a task can be automatically executed in response to a user instruction (e.g., querying auto-renewal, sending a file to someone, disabling an application permission, optimizing overall performance, disabling application notification, and ordering takeout); and in the process of automatically executing the task, automatic execution can be suspended if a situation such as identity verification, requiring a user to confirm order information, or a pop-up window is encountered, and upon receiving the operation of the user, the task continues to be executed, thereby avoiding interruption or termination of the automatic execution process, and improving the success rate and stability of task execution. The method can help users to automatically execute tasks, and the users do not need to operate manually and only need to input instructions, thereby improving user experience and operation efficiency.

Description

UI task automatic execution method and electronic device

[0001] This application claims priority to the Chinese patent application No. 202411244923.1, filed on September 5, 2024, and entitled "Application Business Processing Method, Electronic Device and Storage Medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminals, and in particular to a UI task automatic execution method and an electronic device. BACKGROUND

[0003] In the daily use of a smart phone, a user often needs to manually perform a series of user interface (UI) operations to complete certain specific tasks, such as ordering takeout, closing automatic renewal of a service, sending a file, closing application permissions, closing application notifications, optimizing the entire machine, or other possible tasks. However, manually operating each step of these tasks is neither convenient nor error-free, and even the operation path of some tasks is too complex, and the user operation is tedious and time-consuming. Moreover, in the process of performing these tasks, various special situations such as page loading delay, pop-up ads, operation conflicts, etc. may be encountered, making it difficult for the user to successfully complete the operation and affecting the user experience.

[0004] SUMMARY

[0005] The present application provides a UI task automatic execution method and an electronic device, which can help the user to automatically perform tasks, and the user only needs to input an instruction without manual operation, thereby improving the user experience and operation efficiency.

[0006] In a first aspect, an embodiment of the present application provides a UI task automatic execution method, which comprises: when a first interface is displayed, a first prompt window is displayed, and a first user instruction is received; in response to the first user instruction, after M second interfaces are displayed in sequence, a third interface is displayed; the third interface comprises first information, and the first information is used to prompt a user to take over; M is a positive integer; in response to a first operation input by the user on the third interface, after N fourth interfaces are displayed in sequence, a fifth interface is displayed; the fifth interface is an interface associated with the first user instruction, and N is a positive integer.

[0007] Through the method provided in the embodiments of the present application, a task can be automatically executed in response to a user instruction (for example, inquiring about automatic renewal, sending a file to someone, closing application permissions, optimizing overall machine performance, closing application notifications, ordering takeout), and in the process of automatically executing the task, the automatic execution can be paused when encountering a situation such as identity verification, needing a user to confirm order information, or a pop-up window, and the task execution continues after a user operation is received, thereby avoiding interruption or termination of the automatic execution process and improving the success rate and stability of task execution. Through the scheme, a user can be helped to automatically execute a task, and the user only needs to input an instruction, thereby improving user experience and operation efficiency.

[0008] In some possible implementation manners, the third interface is an identity verification interface, a password input interface, an information confirmation interface, a permission request interface, a single selection or multiple selection interface, or a pop-up window that needs to be taken over by the user. Through the above scheme, the scenario that needs to be taken over by the user can be more accurately identified in the process of automatically executing the task, thereby improving the success rate and stability of task execution.

[0009] Through the above scheme, in the process of automatically executing the task, the automatic execution can be paused when encountering a situation such as identity verification, needing a user to confirm order information, or a pop-up window requiring permission, and the user takes over the process, thereby improving the success rate and stability of task execution.

[0010] In some possible implementation manners, the detection that the user needs to take over includes: when multiple results are obtained during automatic execution and the user needs to select, it is determined that the user needs to take over.

[0011] In some possible implementation manners, the prompting of the user interaction operation in the case that the user needs to take over includes: pausing the execution of the first operation sequence in the case that the user needs to take over; and prompting the user interaction operation by means of display information, vibration prompting, pop-up window prompting, and / or sound prompting.

[0012] Through the above scheme, in the process of automatically executing the task, when the scenario that needs to be taken over by the user is identified, the user is prompted to perform an interaction operation, thereby improving the success rate and stability of task execution.

[0013] In some possible implementation manners, the method further includes: when the M second interfaces are displayed in sequence, displaying a cursor at each second interface, and the cursor is displayed at a target control, and the target control is used to trigger display of a next interface of a current interface.

[0014] In some possible implementation manners, the displaying of the M second interfaces in sequence includes: displaying one second interface in a full-screen window, and displaying a first control for triggering a reduced window in the full-screen window.

[0015] The method further includes: in response to a user operation on the first control, switching the full-screen window to a floating window, the floating window being displayed on the first interface, and M-1 second interfaces being sequentially displayed in the floating window.

[0016] In some possible implementation manners, the method further includes: displaying task execution progress information in the first window while sequentially displaying the M second interfaces through the first window; and / or, displaying a light effect layer in the first window; wherein the first window includes a full-screen window or a floating window.

[0017] In some possible implementation manners, the method further includes: in the full-screen window state, if a preset type of application is started, switching the full-screen window to the floating window; wherein the preset type of application includes a video application or a game application.

[0018] In some possible implementation manners, the method further includes: drawing the light effect layer on a system customized dynamic effect layer level; and when the light effect layer is displayed in the first window, if a touch operation of a user is received, the touch operation is transparently transmitted to a next level of the dynamic effect layer level, and the touch operation is not responded.

[0019] In some possible implementation manners, the method further includes: updating the light effect layer displayed in the first window according to a task execution state change; the task execution state includes an executing state, a user takeover state, and an execution end state.

[0020] The updating the light effect layer displayed in the first window according to the task execution state change includes: displaying a first light effect layer in the first window in the executing state; displaying a second light effect layer in the first window in the user takeover state; and hiding the light effect layer displayed in the first window in the execution end state.

[0021] In some possible implementation manners, the method further includes: in the process of sequentially displaying the M second interfaces, in response to a user operation on any second interface, displaying second information, the second information including prompt information that does not support operation when automatically executed, an exit control, and a continue execution control.

[0022] In some possible implementation manners, the method further includes: in response to the first user instruction, determining a first operation sequence according to a first task graph, and sequentially displaying the M second interfaces according to the first operation sequence; wherein the first task graph includes a start node, a plurality of intermediate nodes, and an end node, the start node being a first interface of the M second interfaces, and the fifth interface being the end node.

[0023] In some possible implementation manners, after the third interface is displayed, the method further includes: in response to a first operation input by a user on the third interface, determining a node position of a currently displayed interface in a first task graph, and determining a second operation sequence according to the node position and the first task graph; the second operation sequence includes the N fourth interfaces.

[0024] In some possible implementation manners, the first task graph is used to represent page states and bidirectional conversion relationships.

[0025] The first task graph is represented as G=(V, E), where V is a page node set, representing page states; E is an edge set, representing bidirectional conversion relationships between pages; a node v i ∈V represents one page in the page node set, an edge (v i , v j )∈E represents conversion from the page v i to the page v j , and an edge (v j , v i )∈E represents conversion from the page v j to the page v i .

[0026] By the above scheme, a task graph structure and a representation method are provided, which are beneficial to re-planning a route according to a node position of a current page in a task graph, and improve the success rate and stability of task execution.

[0027] By the above scheme, in an automatic task execution process, when encountering situations such as identity verification, needing a user to confirm order information, or a pop-up window, the automatic execution can be paused, and a user takeover process is entered. After the user completes an interactive operation, a page node can be relocated, an optimal path can be planned based on the relocated page node, and the task can be continued to be executed according to the re-planned path, so that the automatic execution process is not interrupted or terminated, and the success rate and stability of task execution are improved. Through the scheme, a user can be helped to automatically execute a task, and the user only needs to input an instruction, without manual operation, thereby improving user experience and operation efficiency.

[0028] In some cases, after the user interactive operation is completed, compared with before the user interactive operation, a page can jump or change, and a current page does not belong to an original operation path (a first operation sequence). In this case, a task graph can be used to relocate a page node, an optimal path can be planned based on the relocated page node, and the task can be continued to be executed according to the re-planned path. The second operation sequence is different from the first operation sequence.

[0029] In some cases, after the user interaction operation is completed, the page may jump or change compared with before the user interaction operation, and the current page belongs to the page node of the original operation path (the first operation sequence). In this case, the second operation sequence can be obtained based on the task graph by relocating the page node and planning an optimal path based on the relocated page node. The second operation sequence is different from the first operation sequence.

[0030] In some cases, after the user interaction operation is completed, the page may jump or change compared with before the user interaction operation, and the current page belongs to the page node of the original operation path (the first operation sequence). In this case, the second operation sequence can be obtained based on the task graph by relocating the page node and planning an optimal path based on the relocated page node. The second operation sequence is different from the first operation sequence.

[0031] In some cases, after the user interaction operation is completed, the page may jump or change compared with before the user interaction operation, and the current page belongs to the page node of the original operation path (the first operation sequence). In this case, the second operation sequence can be obtained based on the task graph by relocating the page node and planning an optimal path based on the relocated page node. The second operation sequence is different from the first operation sequence.

[0032] Through the above scheme, in the case where it is detected that the user interaction operation is completed, the optimal execution route of the remaining operation steps can be re-planned by relocating the current page in combination with the task graph, the automatic execution process is quickly completed, and the user experience and operation efficiency are improved.

[0033] In some possible implementation manners, the method further includes: in the case where the second interface appears an advertisement pop-up window, automatically closing the advertisement pop-up window.

[0034] For example, the pop-up window detection is performed when the page appears a pop-up window; in the case where it is detected that the pop-up window is in a system pop-up window whitelist, it is determined that the user needs to take over; or, in the case where it is detected that the pop-up window is an advertisement pop-up window and the advertisement pop-up window has no closing mark, it is determined that the user needs to take over.

[0035] For example, in some scenarios, some applications may request the user to allow a certain permission, and thus a pop-up window is displayed to request authorization. In this case, the user needs to take over.

[0036] Through the above scheme, when a pop-up window appears during the automatic execution of the task, the scenario in which the user needs to take over can be accurately identified, and the success rate and stability of the task execution are improved.

[0037] In some possible implementation manners, the method further includes: in the case where the second interface does not display a target control, triggering the second interface to display by scrolling; after the second interface displays by scrolling, displaying the target control in the second interface, and displaying a cursor at the target control.

[0038] In some possible implementation manners, the method further includes displaying a continue execution control in the third interface; after the third interface is displayed, the method further includes, in response to receiving a user operation on the continue execution control, sequentially displaying the N fourth interfaces.

[0039] By the above scheme, in the process of automatically executing a task, after a scenario requiring user takeover is identified, a user interaction operation is detected to be completed, and then the task is continued to be executed, thereby improving the success rate and stability of task execution.

[0040] In some possible implementation manners, the electronic device includes a system assistant and a UI agent module; the method further includes: the system assistant determining, according to the first user instruction, intent information and slot information, and sending the intent information and the slot information to the UI agent module; the intent information indicates a first business type; the UI agent module determines first robot process automation (RPA) configuration information according to the intent information, and a business type identifier of the first RPA configuration information corresponds to the first business type; the UI agent module generates a user interface (UI) task operation step according to the slot information and the first RPA configuration information, and the UI task operation step includes sequentially displaying the M second interfaces and displaying the fifth interface.

[0041] In some possible implementation manners, the first RPA configuration information includes a task identifier (ID), a tool name, and one or more tool parameters, and one tool parameter corresponds to one slot. The generating of the UI task operation step according to the slot information and the first RPA configuration information includes: filling the slots in the first RPA configuration information according to the slot information to generate the UI task operation step, and the UI task operation step includes the task ID, the tool name, and the one or more tool parameters, and each tool parameter has a slot value.

[0042] In some possible implementation manners, the UI task operation step further includes a target control and a UI task, and the automatic execution process includes: identifying the target control in each interface and determining a position coordinate of the target control; performing the UI task on the target control according to the position coordinate of the target control; and jumping from a current display interface to a next interface after the UI task is executed.

[0043] In some possible implementation manners, the target control is at least one of the following: text, button, slide switch, icon; and the UI task is clicking or sliding the target control; and the performing the UI task on the target control comprises: displaying a cursor simulating a clicking operation on the target control, or displaying a track simulating a sliding operation.

[0044] In some possible implementation manners, the target control is a search box; and the UI task is inputting a keyword in the search box; and the performing the UI task on the target control comprises: displaying an action simulating inputting a keyword in the search box.

[0045] In some possible implementation manners, the identifying the target control comprises: detecting a target control in a text form based on page structure features; wherein the page structure features comprise extensible markup language (XML) structure information and / or a document object model (DOM); detecting a target control in an icon form based on page visual features; in a case where the target control is an icon control and has associated text, detecting the target control based on the page visual features and the page structure features; in a case where the target control is a first text and a plurality of the first texts are included in a page, detecting the target control based on page layout features; and in a case where the target control is not detected based on the page visual features and / or the page structure features, detecting the target control based on the page visual features, the page structure features and the page layout features.

[0046] In some possible implementation manners, in a case where the target control is an icon control and has associated text, the detecting the target control based on the page visual features and the page structure features comprises: in a case where the associated text is a variable parameter, obtaining the page structure features, matching the page structure features with the associated text, and determining position coordinates of the associated text; and based on the page visual features and the position coordinates of the associated text, determining an icon control corresponding to the associated text and position coordinates of the icon control.

[0047] In some possible implementation manners, the determining the user intention information according to the first user instruction comprises: the system assistant determining the user intention information according to the user instruction and user behavior habit data.

[0048] In some possible implementation manners, the user intention information comprises any one of the following: querying automatic renewal, sending a file to someone, closing application permissions, optimizing overall machine performance, closing application notifications, and ordering takeout.

[0049] In some possible implementation manners, the method further includes: in a case where the fifth interface is displayed, displaying a second prompt window in the fifth interface, and displaying service information corresponding to the first user instruction in the second prompt window.

[0050] It should be noted that in the embodiments of the present application, the interface and the interface can be used interchangeably, and no special distinction is made between the two.

[0051] The present application also provides a UI task automatic execution method, the method comprising: receiving a first input of a user, the first input being used to indicate querying automatic renewal service; in response to the first input, starting an automatic execution process corresponding to the querying automatic renewal service. Wherein, the automatic execution process comprises: jumping to a user page and positioning to a service control in the user page, displaying a simulated click cursor at the service control; jumping to an identity verification page, prompting the user to manually operate and input fingerprint / gesture, and performing fingerprint / gesture verification after receiving the user input; after the identity verification is passed, jumping to a service page and positioning to a wallet control in the service page, displaying a simulated click cursor at the wallet control; jumping to a payment setting page and positioning to an automatic renewal control in the payment setting page, displaying a simulated click cursor at the automatic renewal; jumping to an automatic renewal page, obtaining information about the automatic renewal service in the automatic renewal page; displaying the information about the automatic renewal service and a cancel control through a system assistant.

[0052] In some possible implementation manners, the method further includes: in response to the operation of the cancel control by the user, canceling the automatic renewal service.

[0053] Through the UI task automatic execution method provided in the embodiments of the present application, the automatic renewal service of a certain application or each application on the electronic device can be automatically searched in response to the user instruction, and the search result about the automatic renewal service can be quickly displayed for the user, and a quick control for triggering the cancellation of one or more automatic renewal services or a quick control for confirming the cancellation of the automatic renewal service can be provided, so that the user can quickly cancel the automatic renewal service by only confirming, and the operation is convenient and fast. In this way, the user does not need to manually search the automatic renewal service of each application one by one, and therefore the user experience is greatly improved through the present application.

[0054] The application further provides a UI task automatic execution method, which comprises the following steps: receiving a second input of a user, wherein the second input is used for indicating sending a first file to a first object; and starting an automatic execution process corresponding to the sending file in response to the second input. The automatic execution process comprises the following steps: jumping to a file management page and locating the first file, displaying a simulated click cursor at the first file; jumping to a contact page and locating an icon of the first object, displaying a simulated click cursor at the icon of the first object; displaying a file sharing page and locating a sharing control, displaying a simulated click cursor at the sharing control; and displaying a task of sending the first file to the first object by a system assistant.

[0055] The application further provides a UI task automatic execution method, which comprises the following steps: receiving a third input of a user, wherein the third input is used for indicating closing a first function permission of a first application; and starting an automatic execution process corresponding to the closing application permission in response to the third input. The automatic execution process comprises the following steps: jumping to a permission management page and locating the first function permission, displaying a simulated click cursor at the first function permission; jumping to a first function permission setting page and locating an icon of the first application, displaying a simulated click cursor at the icon of the first application; displaying a setting page of the first application using the first function permission and locating an inhibition control, displaying a simulated click cursor at the inhibition control; and displaying a task of closing the first function permission of the first application by a system assistant.

[0056] The application further provides a UI task automatic execution method, which comprises the following steps: receiving a fourth input of a user, wherein the fourth input is used for indicating optimizing overall machine performance; and starting an automatic execution process corresponding to the optimizing overall machine performance in response to the fourth input. The automatic execution process comprises the following steps: jumping to a system manager page and locating a one-key optimization control, displaying a simulated click cursor at the one-key optimization; jumping to the one-key optimization page, displaying performance optimization data; jumping to an optimization completion page and locating a completion control, displaying a simulated click cursor at the completion control; and displaying a task of optimizing the overall machine performance by a system assistant.

[0057] The application further provides a UI task automatic execution method, which comprises the following steps: receiving a fifth input of a user, the fifth input being used for indicating to close a notification of a first application; and starting an automatic execution process corresponding to the closing of the application notification in response to the fifth input. The automatic execution process comprises the following steps: jumping to a settings page and locating a notification control, and displaying a simulated click cursor at the notification control; jumping to the notification settings page and locating a search box, and simulating the input of a first application in the search box; jumping to a notification management page of the first application and locating a notification switch icon, displaying a simulated sliding track at the notification switch icon when the notification switch icon is in an open state, and displaying that the notification switch icon is in a closed state; and displaying, by a system assistant, a task of closing the notification of the first application that has been completed.

[0058] The application further provides a UI task automatic execution method, which comprises the following steps: receiving a sixth input of a user, the sixth input being used for indicating to order takeout; and starting an automatic execution process corresponding to the ordering of takeout in response to the sixth input. The automatic execution process comprises the following steps: jumping to a takeout page and locating a search box, and simulating the input of merchant information in the search box; jumping to a merchant page and locating a search box of the merchant page, and simulating the input of product information in the search box of the merchant page; jumping to a specification selection page and locating a product specification control, displaying a simulated click cursor at the product specification control, and locating a shopping cart control, and displaying a simulated click cursor at the shopping cart control; jumping to a shopping cart page, and prompting the user to click a payment button; and displaying, by a system assistant, a task of ordering takeout that has been completed after the payment is completed.

[0059] In a second aspect, the application provides a UI task automatic execution device, which comprises units for executing the method in the first aspect. The device can correspond to the method described in the first aspect, and the related description of the units in the device is referred to the description of the first aspect. For brevity, the description is not repeated here.

[0060] The method described in the first aspect can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software comprises one or more modules or units corresponding to the above functions. For example, processing modules or units, display modules or units, etc.

[0061] In a third aspect, the application provides an electronic device, which comprises a processor, a processor and a computer program or instructions stored in the memory. The processor is used to execute the computer program or instructions, so that the method in the first aspect is executed.

[0062] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program (which can also be referred to as an instruction or code) for implementing the method in the first aspect. For example, the computer program is executed by a computer, so that the computer can perform the method in the first aspect.

[0063] In a fifth aspect, the present application provides a chip including a processor. The processor is configured to read and execute a computer program stored in a memory, so as to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip further includes the memory, and the memory is connected to the processor through a circuit or a wire.

[0064] In a sixth aspect, the present application provides a chip system including a processor. The processor is configured to read and execute a computer program stored in a memory, so as to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip system further includes the memory, and the memory is connected to the processor through a circuit or a wire.

[0065] In a seventh aspect, the present application provides a computer program product, which includes a computer program (which can also be referred to as an instruction or code). The computer program is executed by an electronic device, so that the electronic device implements the method in the first aspect.

[0066] It can be understood that the beneficial effects of the above-mentioned second aspect to seventh aspect can be referred to the related description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0067] FIG. 1A is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0068] FIG. 1B is a software architecture schematic diagram of an electronic device according to an embodiment of the present application;

[0069] FIG. 2A and FIG. 2B are flow schematic diagrams of a UI task automatic execution method according to an embodiment of the present application;

[0070] FIG. 2C is an interface schematic diagram of system assistant wake-up, reply and result display of a UI task automatic execution method according to an embodiment of the present application;

[0071] FIG. 3A to FIG. 3C are interface schematic diagrams of a UI task automatic execution method applied to query all application renewal services according to an embodiment of the present application;

[0072] FIG. 3D is an interface schematic diagram of a UI task automatic execution method applied to query a single application renewal service according to an embodiment of the present application;

[0073] FIG. 3E is an interface schematic diagram of a UI task automatic execution method applied to query multiple application renewal services according to an embodiment of the present application;

[0074] FIG. 3F is a schematic diagram of an interface of the UI task automatic execution method provided by the embodiment of the present application applied to cancel application renewal service;

[0075] FIGS. 4A to 4C are a schematic diagram of an interface and a functional module interaction flow of dynamic light effect reminding in the UI task automatic execution method provided by the embodiment of the present application;

[0076] FIG. 5A is a schematic diagram of an interface of small window full screen switching in the UI task automatic execution method provided by the embodiment of the present application;

[0077] FIG. 5B is a schematic diagram of an interface of automatic small window display in the immersive scene in the UI task automatic execution method provided by the embodiment of the present application;

[0078] FIGS. 6A to 6D are schematic diagrams of exit mode and abnormal situation processing in the UI task automatic execution method provided by the embodiment of the present application;

[0079] FIGS. 7A to 7D are schematic diagrams of the UI task automatic execution method applied to send files provided by the embodiment of the present application;

[0080] FIG. 8 is a schematic diagram of an interface of the UI task automatic execution method applied to close application permission provided by the embodiment of the present application;

[0081] FIG. 9 is a schematic diagram of an interface of the UI task automatic execution method applied to optimize the whole machine performance provided by the embodiment of the present application;

[0082] FIGS. 10A to 10B are schematic diagrams of an interface of the UI task automatic execution method applied to close application notification provided by the embodiment of the present application;

[0083] FIGS. 11A to 11D are schematic diagrams of an interface of the UI task automatic execution method applied to point drinks provided by the embodiment of the present application;

[0084] FIGS. 12A to 12C are schematic diagrams of the whole flow of the UI task automatic execution method provided by the embodiment of the present application;

[0085] FIGS. 13A to 13E are a flow and a schematic diagram of detecting target control in the UI task automatic execution method provided by the embodiment of the present application;

[0086] FIGS. 14A to 14I are a flow and a schematic diagram of user takeover required in the UI task automatic execution method provided by the embodiment of the present application;

[0087] FIGS. 15A to 15C are a flow and a schematic diagram of identifying pop-up window and automatically closing advertisement pop-up window in the UI task automatic execution method provided by the embodiment of the present application;

[0088] FIGS. 16A-16B are schematic diagrams of interfaces for attempting to search for a target control again in a case where the target control is not found in a UI task automatic execution method according to an embodiment of the present application;

[0089] FIGS. 17A-17G are schematic diagrams of a task agent model used in a UI task automatic execution method according to an embodiment of the present application and a model training method. DETAILED DESCRIPTION

[0090] The embodiments of the present application are further described below in conjunction with the specific embodiments and with reference to the drawings.

[0091] Hereinafter, the terms "first" and "second" are used only for the purpose of description and should not be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0092] In the embodiments of the present application, the term UI is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes conversion between an internal form of information and a form acceptable by the user. The user interface is source code written in a specific computer language such as Java and extensible markup language (XML), and the interface source code is parsed, rendered, and finally presented as content recognizable by the user on an electronic device. A commonly used form of user interface is a graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. displayed in the display screen of the electronic device.

[0093] In the embodiments of the present application, the operation can include one or more of a sliding operation, a multi-finger sliding operation, a single screen operation, a multi-screen operation, a multi-finger single screen operation, a multi-finger multi-screen operation, and a long press operation performed by the user in the first user interface. One operation can also be composed of multiple sub-operations, and the present application does not make specific limitations.

[0094] In the daily use of a smart phone, a user often needs to frequently perform a series of operations to complete certain specific tasks, such as ordering takeout, closing an automatic renewal service, sending a file, closing an application permission, closing an application notification, optimizing the whole machine, or other possible tasks. However, manually operating each step of these tasks is not only inconvenient but also prone to errors, and even the operation path of some tasks is too complex for a user to find and complete the operation smoothly, affecting the user experience.

[0095] In addition, in the process of performing these tasks, various special situations (or abnormal situations) such as page loading delay, pop-up ads, operation conflict, etc. may be encountered, making it difficult for the user to complete the operation smoothly and affecting the user experience.

[0096] To solve the above problems, the embodiments of the present application provide a UI task automatic execution method and an electronic device. Through the UI task automatic execution method, a series of preset task operations can be automatically executed according to the user's intention, helping the user save time and effort. The embodiments of the present application provide a more intelligent and perfect automatic execution technical framework and automatic execution algorithm to realize the UI task automatic execution method.

[0097] It should be noted that the UI task automatic execution method provided by the embodiments of the present application not only can automatically execute a series of preset task operations, but also can handle complex operation sequences, can realize the full-flow page jump of the UI task, and can accurately identify the target control in each page after the jump and simulate the click event on the target control to realize the automatic execution of the task; can timely prompt the user when user interaction is needed; and can handle various special situations such as page loading delay, pop-up ads, operation conflict, etc. that occur during execution, so that the automatic execution process can be completed smoothly.

[0098] In actual application, many business operation steps provided by the electronic device are multiple, or the operation path is deep, or there are a large amount of specific information that needs to be repeatedly filled in by hand. The UI task automatic execution method provided by the embodiments of the present application can improve the above problems and provide great convenience for the user.

[0099] The present application scheme can provide the user with sufficient sense of trust and sense of control, and be intuitive, visual, convenient and controllable in strategy. The present application scheme has the following improvements and technical effects:

[0100] 1) State visual: edge light in the execution process, clearly showing the control state;

[0101] 2) Execution visual: full-screen and small window execution interface display scheme, full-process visual;

[0102] 3) Step notification: there is a step prompt window to clearly show the current execution progress;

[0103] 4) Artificial takeover: special cases are taken over by users, safe and controllable;

[0104] 5) Interrupt at any time: clear exit button, pause at any time.

[0105] Next, the electronic device to which the UI task automatic execution method provided by the embodiments of the present application is applied will be described with reference to the accompanying drawings. Illustratively, the electronic device in the embodiments of the present application can be a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet), and the like, which has a voice wake-up function. The embodiments of the present application do not specially limit the specific form of the electronic device.

[0106] In some examples, the user can input an instruction through the mobile phone to trigger the mobile phone to complete automatic execution of a task.

[0107] In some other examples, the user can quickly input an instruction through a wearable device and forward the instruction to the mobile phone, and the mobile phone completes automatic execution of a task, and then the mobile phone returns a task completion result to the wearable device, and the wearable device displays the task completion result.

[0108] FIG. 1A shows a schematic diagram of the hardware structure of an electronic device provided by the embodiments of the present application. Illustratively, taking the electronic device as a mobile phone as an example, as shown in FIG. 1A, the electronic device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver (i.e., an earpiece) 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like.

[0109] The sensor module can include a pressure sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0110] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated into one or more processors.

[0111] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0112] The memory in the processor 110 can also be provided to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0113] It can be understood that the interface connection relationship between the modules shown in the embodiments is only illustrative and does not constitute a structural limitation on the electronic device. In other embodiments, the electronic device can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0114] In the embodiments of the present application, the electronic device can implement the method provided by the embodiments of the present application through the processor 110.

[0115] The above is a specific description of the embodiments of the present application taking the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation of the electronic device 100. The electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0116] In addition, on the above hardware, an operating system is running. The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or windows operating system, etc. Application programs can be installed and run on the operating system.

[0117] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0118] FIG. 1B is a software structure block diagram of the electronic device 100 of the embodiments of the present application.

[0119] The layered architecture divides the software into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android TM system is divided into five layers, from top to bottom, the application layer, the application framework layer, the Android TM runtime and system library layer, the hardware abstraction layer (HAL), and the kernel layer.

[0120] The application layer can include a series of application packages. For example, the application layer can include game App, map App, short video App, social App, shopping App, etc.

[0121] In the embodiments of the present application, the application layer can also include a smart perception service module (system assistant application), which includes a scene perception module, a business logic processing module, and a business presentation module, etc. The scene perception module, the business logic processing module, and the business presentation module can be independent Apps, or can be integrated in different Apps, or can be integrated in the same App, which is not limited by the present application.

[0122] The context awareness module, which is running in the background or in a low power mode, has the ability to perceive external facts or environment. The context awareness module can detect relevant events and obtain the state of the events from other applications in the application layer or the application framework layer or the system layer or the kernel layer through an application programming interface (API), such as detecting a Bluetooth connection, a network connection, monitoring user short messages, customizing a timer, and the like. In the embodiments of the present application, the main function of the context awareness module is to listen to user input events such as querying an automatic renewal service, canceling an automatic renewal service, and the like. The context awareness module can notify the business logic processing module of the user input events that are listened to.

[0123] The business logic processing module (such as a computing engine) has a business logic processing capability, and is configured to perform data processing or logic judgment according to the user input events listened to by the context awareness module and a preset logic algorithm, to determine to query an automatic renewal service in a first application and to obtain an automatic execution step or path, to start an automatic execution process, to perform page jumping according to the automatic execution step or path, and to jump to an automatic renewal interface, thereby automatically querying an automatic renewal service of one or more applications.

[0124] The business presentation module is configured to display automatic renewal service information and a cancel control on a screen of the electronic device according to an indication of the business logic processing module.

[0125] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some pre-defined functions. In the embodiments of the present application, the application framework layer can include a windows manager service (WMS), an input manager service (IMS), a virtual event processor, a view system, and the like, and the present application does not make any limitation in this regard.

[0126] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, and the like. The view system can be used to build an application. A display interface can be composed of one or more views.

[0127] For example, in the embodiments of the present application, the view system can be used to display a dynamic light effect prompt pattern in the automatic execution window to prompt that the current page is a window for automatic execution of a UI task. The view system can also be used to display prompt information in the automatic execution window to prompt the specific action or step of the UI task being executed by the current page. The view system can also be used to display a zoom-out control (also referred to as a zoom-out to small window button) in the automatic execution window for triggering the automatic execution window to be switched from full screen mode to small window mode for display. The view system can also be used to display a stop control in the automatic execution window for triggering the stop of execution of the UI task.

[0128] The input management service (IMS) is used to manage input events. For example, in the embodiments of the present application, the input events can include virtual click events, virtual sliding events, and virtual input events, etc. The input events can also include user click events, user sliding events, and user input events, etc.

[0129] The virtual event processor is used to process virtual events.

[0130] For example, in the embodiments of the present application, during automatic execution of a UI task, the virtual event processor can implement the following virtual events:

[0131] The virtual click event, that is, the system automatically clicks a target control related to the UI task in the automatic execution window.

[0132] The virtual sliding event, that is, the system automatically triggers a sliding operation in a certain area related to the UI task in the automatic execution window.

[0133] The virtual input event, that is, the system automatically inputs a specific character or symbol, etc. related to the UI task in the input box of the automatic execution window. The notification manager enables an application program to display notification information in the status bar, which can be used to convey a message of the notification type and can automatically disappear after a short stay without user interaction.

[0134] The window management service (WMS) is used to manage window programs. For example, the window manager can control the automatic execution window to be displayed in full screen mode, can control the automatic execution window to be displayed in small window mode, and can control the automatic execution window to be switched from full screen mode (or full screen window mode) display to small window mode display, or from small window mode display to full screen mode display.

[0135] The Android runtime is responsible for the scheduling and management of the Android system. The system library layer can also be referred to as the native layer (Native layer), and can also be referred to as the native service layer or the native framework layer. This layer is a native code platform library, which can provide native services and link libraries, etc. for hardware operations for the upper layer.

[0136] The HAL layer is a wrapper for the Linux kernel driver, providing an interface upwards and shielding the implementation details of the lower layer hardware.

[0137] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, audio driver, and sensor driver.

[0138] For ease of illustration, the hardware layer interacting with the above software structure is also embodied in the figures. For example, the hardware layer can include display screen and sensor, etc.

[0139] It should be noted that, although the embodiments of the present application are described taking the Android system as an example, the basic principles are also applicable to electronic devices based on iOS or Windows operating system.

[0140] The execution subject of the UI task automatic execution method provided by the embodiments of the present application can be the electronic device described above, or a functional module and / or functional entity capable of implementing the UI task automatic execution method in the electronic device, and the present application scheme can be implemented by hardware and / or software, and the specific implementation can be determined according to actual use requirements, which is not limited by the embodiments of the present application. The UI task automatic execution method provided by the embodiments of the present application will be described exemplarily below taking the electronic device as an example in combination with the drawings.

[0141] The embodiments of the present application will be described below by way of example in combination with the drawings. The methods in the following embodiments can be implemented in the electronic device having the hardware structure and software architecture described above. The hardware structure diagram of the electronic device can be as shown in FIG. 1A, and the software structure block diagram of the electronic device can be as shown in FIG. 1B, but the embodiments of the present application are not limited thereto. For ease of illustration, the electronic device in the embodiments of the present application is taken as a mobile phone as an example.

[0142] The UI task automatic execution method provided by the embodiments of the present application will be described below in combination with specific embodiments.

[0143] FIG. 2A shows a flowchart of the UI task automatic execution method provided by the embodiments of the present application.

[0144] S201, receiving a user instruction, and determining user intention information according to the user instruction.

[0145] In the embodiments of the present application, when the user wants to query automatic renewal, send files, order takeout, etc. through the electronic device, the user can input an instruction to the electronic device, and the electronic device can determine the user intention information according to the user instruction after receiving the user instruction.

[0146] The user instruction can be voice input and / or text input.

[0147] In the embodiments of the present application, the electronic device receives a user instruction, performs semantic recognition processing on the user instruction, obtains a keyword, and then determines user intent information according to the keyword.

[0148] For example, in some scenarios, the user instruction is “Please help me check what application is automatically deducted”, or “Please help me check whether XX application / service is automatically deducted”, or “Help me close all automatic deduction services”, or “Help me close the automatic deduction service of XX application / service”. The electronic device can determine the user intent information as “querying automatic deduction” according to the semantic recognition of the user instruction.

[0149] For example, in some scenarios, the user instruction is “Help me send a certain file to someone”, or “Help me send the ticket photo received by WeChat to someone”, or “Help me send a photo to someone”. The electronic device can determine the user intent information as “sending a file to someone” by performing semantic recognition on the user instruction. TM For example, in some scenarios, the user instruction is “Help me send a certain file to someone”, or “Help me send the ticket photo received by WeChat to someone”, or “Help me send a photo to someone”. The electronic device can determine the user intent information as “sending a file to someone” by performing semantic recognition on the user instruction.

[0150] For example, in some scenarios, the user instruction is “Help me close the XX permission (microphone permission) of a certain application”. The electronic device can determine the user intent information as “closing a certain permission of an application” by performing semantic recognition on the user instruction.

[0151] For example, in some scenarios, the user instruction is “Help me optimize my phone”. The electronic device can determine the user intent information as “optimizing the overall performance of the phone” by performing semantic recognition on the user instruction.

[0152] For example, in some scenarios, the user instruction is “Help me close the notification of App1” or “Help me close the notification of App1 and App2”. By semantic recognition, the user intent information is “closing the application notification”.

[0153] For example, in some scenarios, the user instruction is “Help me order a large ice Americano coffee” or “Help me order an ice coffee and a hot milk tea”. By semantic recognition, the user intent information is “ordering takeout”.

[0154] Taking the user instruction as a voice input as an example, the user instruction can be a voice input for waking up a smart assistant application of the electronic device. The smart assistant application is a system assistant application or other assistant application. The system assistant application is a system application program in the electronic device.

[0155] For example, the system assistant can be awakened in various ways, including voice awakening, power key awakening, shortcut key awakening, screen sliding awakening, gesture awakening, search awakening, and breath awakening.

[0156] Voice wake-up is the most common way, users can directly say "Hello, YOYO" to the phone to activate the system assistant.

[0157] Power key wake-up allows users to press and hold the power key until the system assistant is awakened.

[0158] Shortcut key wake-up is by adding a shortcut to the system assistant on the desktop, users just need to tap to quickly wake up.

[0159] Slide screen wake-up and gesture wake-up provide more personalized ways of operation, users can set specific sliding actions or gestures to activate the system assistant.

[0160] Search wake-up is by searching "voice assistant" on the screen to wake up the system assistant.

[0161] Breath wake-up is a convenient way to wake up the system assistant by breath, users can hold the phone close to their mouth and directly speak voice commands without using specific wake-up words. This way is more natural and convenient. For ease of illustration, the following embodiments are described by taking breath wake-up as an example.

[0162] In some embodiments, when the electronic device displays a desktop, the electronic device can receive a user input instruction, wake up the system assistant, and determine user intent information according to the user instruction.

[0163] In other embodiments, when the electronic device displays an application page, the electronic device can receive a user input instruction, wake up the system assistant, and determine user intent information according to the user instruction.

[0164] In the embodiments of the present application, the electronic device can identify the key words in the user instruction according to the user instruction, and determine the user intent information corresponding to the key words according to the preset correspondence between the key words and the user intent information.

[0165] In the embodiments of the present application, the electronic device is preset with multiple user intent information, and each user intent information corresponds to one or more key words. Table 1 schematically shows the correspondence between the preset user intent information and the key words.

[0166] Table 1

[0167] It should be noted that the user intent information in Table 1 is illustrative, and other user intent information can also be preset in actual implementation, such as opening App1 to send a red envelope to someone, and the specific content of the preset user intent information is not limited in the embodiments of the present application.

[0168] Similarly, the keywords in Table 1 are illustratively exemplified, and in actual implementation, the keywords can be set according to actual needs, and the embodiments of the application are not limited.

[0169] S202, determining a UI task operation step according to the user intention information, each step in the UI task operation step including a page, a target control in the page, a UI task, and whether the user takes over.

[0170] It should be noted that the system assistant can trigger the electronic device to automatically execute the UI task in response to the user instruction, wherein the RPA (robotic process automation) technology is adopted to automatically simulate clicking and jumping pages to achieve the purpose of automatic execution of the UI task. Exemplarily, the scenario of automatic execution of the UI task can include closing permission notification, closing automatic renewal, helping the user to order multiple specifications of coffee, and / or ordering takeout, etc.

[0171] The embodiments of the application adopt the RPA technology to achieve automatic execution of the UI task flow. Wherein, the RPA technology can simulate user operations on the electronic device by means of a preset program, such as keyboard input, mouse click, etc., and can automatically execute repetitive, rule-based and predictable business processes or tasks, thereby achieving business process automation.

[0172] Wherein, the automatic execution information of a single step includes a page, a target control in the page, a UI task, and whether the user takes over. For ease of description, it is denoted as step i (page, target control, UI task, whether the user takes over). Step i represents the i-th step. i is a positive integer.

[0173] Wherein, each user intention information corresponds to a UI task operation step, and each UI task operation step includes multiple steps.

[0174] In the embodiments of the application, the electronic device pre-sets the correspondence between the user intention information and the UI task operation step. Table 2A illustratively shows the pre-set correspondence between the user intention information and the UI task operation step.

[0175] Table 2A

[0176] In some embodiments, the target control in the page is at least one of the following: an icon, a button (such as a round button), a sliding switch, and text. The corresponding UI task includes automatically clicking, sliding, or long-pressing the target control by simulating the operation.

[0177] Exemplarily, the user intention information is "optimizing the overall performance of the machine", and the corresponding UI task operation steps include the following two steps. Step 1 (system manager page, one-key optimization option, clicking the one-key optimization option, without user intervention); and step 2 (one-key optimization page, completion option, clicking the completion option, without user intervention).

[0178] In some embodiments, when user intervention is needed, the user operation content is embodied in the step. If a step does not need user intervention, the step can not embody the user intervention.

[0179] Exemplarily, the user intention information is "turning off the notification of App1", and the corresponding UI task operation steps include the following three steps.

[0180] Step 1 (setting main page, notification option, clicking the notification option);

[0181] Step 2 (notification setting page, notification management option, clicking the notification management option);

[0182] Step 3 (notification management page, App1 switch option, sliding the App1 switch option).

[0183] Exemplarily, the user intention information is "turning off the microphone permission of the browser", and the corresponding UI task operation steps include the following five steps.

[0184] Step 1 (setting main page, application option, clicking the application option);

[0185] Step 2 (application setting page, permission management option, clicking the permission management option);

[0186] Step 3 (permission management page, microphone option, clicking the microphone option);

[0187] Step 4 (microphone permission setting page, browser option, clicking the browser option);

[0188] Step 5 (browser microphone permission setting page, prohibit option, clicking the prohibit option).

[0189] It should be noted that the above UI task operation steps are exemplarily described, and the UI task operation steps in the embodiments of the present application are not limited to the above cases. In actual implementation, the number of steps of the UI task operation steps can be set according to actual use requirements, and the page information, target control, UI task and / or whether user intervention is needed in each step can be determined according to actual use requirements.

[0190] For example, the user intent information is "turn off the microphone permission of the browser", and the corresponding UI task operation steps can include the following 3 steps.

[0191] Step 1 (permission management page, microphone option, click microphone option);

[0192] Step 2 (microphone permission setting page, browser option, click browser option);

[0193] Step 3 (browser microphone permission setting page, disable option, click disable option).

[0194] For example, the user intent information is "turn off the microphone permission of the browser", and the corresponding UI task operation steps can include the following 1 step. Step 1 (browser microphone permission setting page, disable option, click disable option).

[0195] That is, after starting the automatic execution process, the electronic device directly jumps to the "browser microphone permission setting page", identifies the target control "disable option" in the setting page and determines the position coordinates of the disable option, and then selects the disable option by simulating a click operation. In this way, the task of turning off the microphone permission of the browser is completed.

[0196] In some other embodiments, the target control in the page is a search box, and the corresponding UI task includes automatically inputting information in the search box by simulating an operation.

[0197] For example, for user intent information such as sending a certain file to someone, turning off a certain permission of an application, turning off the notification of an application, etc., information such as application name, person name, file name, or permission name needs to be input in the search box to query specific objects such as applications, specific persons, specific files, or specific permissions.

[0198] Table 2B shows the case where the UI task operation steps include a search box and inputting information in the search box.

[0199] Table 2B

[0200] For example, the user intent information is "turn off the notification of App1", and the corresponding UI task operation steps include the following 3 steps.

[0201] Step 1 (setting main page, notification option, click notification option);

[0202] Step 2 (notification setting page, search box, automatically input App1 in search);

[0203] Step 3 (notification management page, App1 switch option, identify switch state and slide App1 switch option in switch on state).

[0204] It should be noted that the above UI task operation steps are illustrative. It can be understood that, in actual implementation, the UI task operation steps corresponding to each user intent information can be determined according to actual use requirements, and the embodiments of the present application are not limited.

[0205] S203, according to the UI task operation steps, start the UI task automatic execution process, and the automatic execution process includes identifying the target control in each page and determining the position of the target control, simulating the click operation of the target control or inputting information in the search box, and jumping to the page after the click operation or inputting information in the search box.

[0206] In the embodiments of the present application, the electronic device can simulate the click operation, the sliding operation or the input operation according to the UI task operation steps, and automatically execute the UI task.

[0207] For example, the electronic device can automatically jump to a page of the UI task according to the UI task operation steps, identify the target control in the page and determine the position of the target control according to the UI task corresponding to the page, then complete the click operation of the target control by simulating the click operation, and then automatically jump to the next page.

[0208] For example, the electronic device can automatically jump to a page of the UI task according to the UI task operation steps, identify the search box in the page and determine the position of the search box according to the UI task corresponding to the page, then input the slot information of the UI task in the search box by simulating the input operation, and then automatically jump to the next page.

[0209] In some embodiments, in the automatic execution process, when the electronic device simulates the click operation, or simulates the input operation, or simulates the sliding operation in the page, the cursor can be displayed at the click position, the input position or the sliding position of the page, so that each execution step of the automatic execution process is very intuitive, and the user experience can be improved.

[0210] In the embodiments of the present application, the display size, display form and display color of the cursor are not limited, and can be determined according to actual use requirements.

[0211] S204, after the automatic execution process is completed, the automatic execution result is displayed.

[0212] In the following, the user interaction interface and internal implementation algorithm of S203 will be described in detail through specific embodiments.

[0213] FIG. 2B shows another flowchart of the UI task automatic execution method according to an embodiment of the present application. As shown in FIG. 2B, the electronic device is woken up after receiving the user input, for example, the system assistant is woken up, then the system assistant replies, then the automatic execution process is started, and the automatic execution process is displayed, and finally the automatic query result is displayed.

[0214] FIG. 2C shows an interface diagram of the wake-up state instruction reply in the UI task automatic execution method according to an embodiment of the present application. The present application provides two states of the system assistant, the wake-up state and the reply state. As shown in (a) of FIG. 2C, when the user holds the bottom of the mobile phone close to the mouth and says the voice instruction "help me cancel all application automatic deduction", in response to the user voice instruction, the electronic device wakes up the system assistant and displays the prompt information "help me cancel all application automatic deduction". As shown in (b) of FIG. 2C, after being woken up, the system assistant replies to the user voice instruction and interacts with the user, for example, the electronic device displays the prompt information "OK, will automatically query and cancel all application automatic deduction", then the electronic device automatically queries the automatic deduction service, as shown in (c) of FIG. 2C, the electronic device shows the query result to the user, and after receiving the user confirmation operation, the electronic device cancels the automatic deduction of all applications.

[0215] The UI task automatic execution method provided by the present application can be applied to various user scenarios, for example, the user scenario can be that the user requests to query the application automatic deduction service, or the user requests to send a file to someone, or the user requests to close the application permission, or the user requests to optimize the overall performance, or the user requests to close the application notification, or the user requests to order take-out (such as coffee or milk tea) through a shopping application. The UI task automatic execution method provided by the present application will be described in detail with respect to each user scenario.

[0216] First implementation: automatic execution process of querying automatic deduction service

[0217] At present, a large number of application programs (application, App) are installed on the electronic device, such as entertainment type App (for example, video type application, audio type application), consumption type App (such as electronic supermarket) and life payment type App (such as paying for calls) provide automatic deduction service (or called automatic deduction service), which will automatically deduct and provide corresponding App monthly service after being turned on. In actual application, due to the complex setting path of the automatic deduction service of each application, it is difficult for the user to find the setting entrance of the automatic deduction service of each application. Therefore, there is a problem of difficult application automatic deduction service finding and tedious setting operation.

[0218] To solve the problem, the UI task automatic execution method provided in the first embodiment of the present application can realize automatic query of the automatic renewal service. The user only needs to input a query instruction, and the electronic device side can automatically execute the query task of the automatic renewal service and complete the full-process jump of querying the automatic renewal service through the user interface, display the whole query process and the final query result to the user, and solve the problem of tedious manual query operation of the user. Therefore, the scheme improves the user experience.

[0219] In the first implementation, in response to a user instruction, it can be automatically found whether an automatic renewal service exists in a certain application or each application on the electronic device, and each page jump of the first application is performed according to a query path until an automatic renewal service sub-interface is jumped to. The automatic renewal service sub-interface includes automatic renewal conditions of each application based on the first application.

[0220] It should be noted that there can be various user needs for how to query the automatic renewal service, for example, the user requests to query the automatic renewal service of all applications, or the user requests to query the automatic renewal service of a single application, or the user requests to query the automatic renewal service of two or more applications, or the user requests to cancel the automatic renewal service of a certain application.

[0221] Table 3 schematically shows different query modes and user instructions corresponding to querying the automatic renewal.

[0222] Table 3

[0223] The embodiments of the present application provide corresponding UI task automatic execution methods for various user needs. The following will be described in detail through various embodiments.

[0224] Method 1-automatic execution process of querying all application renewal

[0225] Through the UI task automatic execution method provided in the embodiments of the present application, in response to a user instruction, it can be automatically found whether an automatic renewal service exists in all applications on the electronic device, and the query result of the automatic renewal service of all applications can be quickly displayed to the user.

[0226] Table 4 schematically shows the UI task operation steps in the automatic execution process of querying all application renewal. The UI task operation steps include jump page information, target controls in each page, and UI tasks corresponding to each page and whether user operation is needed.

[0227] Table 4

[0228] By automatically executing the process, the electronic device can obtain the App1 renewal information and the App2 renewal information, and then display all the renewal information for the user to view. The interface implementation of the automatic execution process is described below in conjunction with the accompanying drawings.

[0229] FIG. 3A shows a schematic diagram of an interface of a UI task automatic execution method provided by an embodiment of the present application applied to querying all application automatic renewal business.

[0230] As shown in (a) of FIG. 3A, the electronic device displays a desktop 301. When the electronic device displays the desktop 301, an operation of waking up the intelligent assistant (or system assistant) is received, and the system assistant of the electronic device is woken up to receive the instruction of the user. After receiving the instruction of the user, as shown in (b) of FIG. 3A, the system assistant displays a prompt box 302 on the desktop 301, and the prompt box 302 includes the instruction of the user “Help me check which application is automatically deducted”. The instruction operation includes any one of the following modes: user voice input instruction, keyboard input instruction, or operation of a pre-defined instruction shortcut (not shown in the figure).

[0231] After receiving the instruction of the user, as shown in (c) of FIG. 3A, the system assistant updates the prompt box 302 to a prompt box 303, and the prompt box 303 includes the request for querying information “Help me check which application is automatically deducted”, and the response information “OK, now jump to App and App to find all application automatic renewal”.

[0232] Among them, App and App are collectively referred to as “business affiliated App”. The “business affiliated App” refers to the application that depends on the automatic renewal business. It can be understood that after the automatic renewal business of a certain application (for example, a video application) is opened, the video application will automatically deduct the fee from the “business affiliated App”.

[0233] It should be noted that the embodiments of the present application take App and App as “business affiliated App” to query the automatic renewal of all applications on the electronic device as an example for illustrative description. In actual implementation, it can be determined according to actual use requirements which “business affiliated App” is used for business query.

[0234] For example, in the case where the electronic device does not install App, the automatic renewal of all applications on the electronic device can be queried only through App.

[0235] For another example, in a case where the electronic device is installed with at least three service affiliated Apps, the electronic device can further query the automatic renewal of all applications on the electronic device through the at least three service affiliated Apps.

[0236] After (c) in FIG. 3A, the electronic device automatically jumps to the page of the App to query the automatic renewal service, as shown in (d) of FIG. 3A, the electronic device automatically jumps from the desktop 301 to the main page 304 of the App. The main interface 304 displays a prompt information “WeChat is querying for you” 305. The prompt information is used to indicate the current state by a preset manner, for example, at most 6 Chinese characters are displayed in the display state, and a super long marquee form is used for scrolling display.

[0237] It should be noted that the main page 304 is in a page locked state, and the interface in the page locked state does not support user input, or in other words, ignores user input, that is, does not respond to user input. In the locked state, the electronic device displays a light effect prompt pattern 306 around the current page.

[0238] It should be further noted that during the automatic query process of the electronic device, each jump page is in a page locked state and displays a light effect prompt pattern. After the automatic query of the electronic device ends, the page locked state is cancelled and the light effect prompt pattern is hidden.

[0239] The electronic device can identify the target control “I” in the main page 304 and determine the position of the target control “I”, and then simulate clicking the target control “I”. As shown by the simulated click cursor in (d) of FIG. 3A, the electronic device automatically clicks the target control “I” through the simulated click operation, and in response to the click operation, the electronic device continues to jump the page to query the automatic renewal service, as shown in (e) of FIG. 3A, the electronic device automatically jumps from the main page 304 of the App to the user page 307. The electronic device can identify the target control “I” in the main page 304 and determine the position of the target control “I”, and then simulate clicking the target control “I”. As shown by the simulated click cursor in (d) of FIG. 3A, the electronic device automatically clicks the target control “I” through the simulated click operation, and in response to the click operation, the electronic device continues to jump the page to query the automatic renewal service, as shown in (e) of FIG. 3A, the electronic device automatically jumps from the main page 304 of the App to the user page 307.

[0240] After (e) in FIG. 3A, as shown in (f) of FIG. 3A, the electronic device automatically jumps from the user page 307 of the App to the identity verification page 308. The identity verification page 308 is used for user identity verification. The identity verification page 308 displays a prompt information “Please manually operate” 309, and further displays a “fingerprint / gesture verification” text prompt content 310 and a gesture dynamic prompt pattern.

[0241] ​​It should be noted that in this case, the electronic device switches from the automatic execution state to the user operation state, and in the case that the electronic device receives the user input and the identity verification is successful, the electronic device switches from the user operation state back to the automatic execution state, and continues to automatically jump the page to query the automatic renewal service.

[0242] After (f) in FIG. 3A, as shown in (g) in FIG. 3A, the electronic device automatically jumps from the identity verification page 308 of the App to the service page 311 of the App. After (g) in FIG. 3A, as shown in (h) in FIG. 3A, the electronic device automatically jumps from the service page 311 of the App to the wallet page 312 of the App, and identifies the target control "wallet" and determines the position of the target control "wallet", and then simulates clicking the target control "wallet". After (g) in FIG. 3A, as shown in (h) in FIG. 3A, the electronic device automatically jumps from the service page 311 of the App to the wallet page 312 of the App, and identifies the target control "wallet" and determines the position of the target control "wallet", and then simulates clicking the target control "wallet". After (i) in FIG. 3A, as shown in (j) in FIG. 3A, the electronic device automatically jumps from the payment setting page 313 of the App to the automatic renewal page 314 of the App. After (i) in FIG. 3A, as shown in (j) in FIG. 3A, the electronic device automatically jumps from the payment setting page 313 of the App to the automatic renewal page 314 of the App.

[0243] Thus, the electronic device automatically queries the automatic renewal page of the App, and obtains the service information and App information based on the automatic renewal service that the App has started. Thus, the electronic device automatically queries the automatic renewal page of the App, and obtains the service information and App information based on the automatic renewal service that the App has started. Thus, the electronic device automatically queries the automatic renewal page of the App, and obtains the service information and App information based on the automatic renewal service that the App has started.

[0244] FIG. 3B shows a second interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to query the automatic renewal service of all applications. After (j) in FIG. 3A, the electronic device automatically jumps to the page of the App to continue to query the automatic renewal service, as shown in (a) to (e) in FIG. 3B, the electronic device automatically jumps to the main page 315 of the App. After (i) in FIG. 3A, as shown in (j) in FIG. 3A, the electronic device automatically jumps from the payment setting page 313 of the App to the automatic renewal page 314 of the App. Thus, the electronic device automatically queries the automatic renewal page of the App, and obtains the service information and App information based on the automatic renewal service that the App has started. Thus, the electronic device automatically queries the automatic renewal page of the App, and obtains the service information and App information based on the automatic renewal service that the App has started.

[0245] ​Thus, it is queried that the electronic device has started the APP1 package month automatic renewal and the APP2 fee automatic deduction two services (i.e., automatic execution result).

[0246] As shown in (f) of FIG. 3B, the electronic device displays prompt information 320 "Query the following automatic renewal services. Do you want to cancel? Renewal service (2), cancel all control 321; App1 package month automatic renewal, cancel control; App1 fee automatic deduction, cancel control".

[0247] It should be noted that the above embodiments are exemplary descriptions of the pages jumped by the electronic device during automatic execution, and the present application is not limited thereto. The specific page content and quantity can be set according to actual use requirements.

[0248] It should be further noted that the above embodiments are exemplary descriptions of the prompt information displayed by the electronic device during automatic execution, and the present application is not limited thereto. The specific prompt information can be set according to actual use requirements.

[0249] It should be noted that after the electronic device displays the automatic execution result, the automatic execution process ends, and at this time the electronic device can prompt the user for interaction (user takeover). The user can view the automatic execution result and perform interactive operations according to the requirements.

[0250] FIG. 3C shows an interface schematic diagram of the UI task automatic execution method provided by the present application applied to cancel automatic renewal. As shown in (a) and (b) of FIG. 3C, in response to the operation of the user clicking the cancel all control 321, the electronic device displays prompt information, and the prompt information includes user indication information "cancel all renewal services" 323 and response information "OK, the system assistant will automatically execute and cancel all renewal services" 324.

[0251] The electronic device automatically cancels all renewal services through the system assistant. After the electronic device automatically cancels all renewal services, the electronic device can prompt that all renewal services have been canceled.

[0252] After the electronic device prompts that all renewal services have been canceled, the user can request to query the automatic renewal service again to confirm whether the automatic renewal service has been canceled by the electronic device. As shown in (c) to (e) of FIG. 3C, in response to the user voice input "help me check what application is automatically deducted", the electronic device performs an automatic query process (similar to the above FIG. 3A to FIG. 3B), and after the query is completed, the electronic device displays prompt information 325 "no application is automatically deducted".

[0253] The UI task automatic execution method provided in the embodiments of the present application can automatically find whether there is an automatic renewal service for a certain application or each application on the electronic device in response to a user instruction, and can quickly display the finding result of the automatic renewal service for the user and provide a shortcut control for triggering cancellation of one or more automatic renewal services. The user can quickly cancel the automatic renewal service by one-key clicking, which is convenient and fast. In this way, the user does not need to manually find the automatic renewal service of each application one by one, and therefore the user experience is greatly improved by the solution of the present application.

[0254] It should be further explained that in the automatic execution process, when jumping to a payment or verification (such as password verification or face verification) page, the electronic device will prompt the user to interact, and the user manually completes the payment or verification, thereby improving the user information security and ensuring the smooth completion of the automatic execution process.

[0255] Method 2-automatic execution process for querying automatic renewal of a single application

[0256] The UI task automatic execution method provided in the embodiments of the present application can automatically find whether there is an automatic renewal service for a certain application or each application on the electronic device in response to a user instruction, and can quickly display the finding result of the automatic renewal service for the user and provide a shortcut control for triggering cancellation of one or more automatic renewal services. The user can quickly cancel the automatic renewal service by one-key clicking, which is convenient and fast. In this way, the user does not need to manually find the automatic renewal service of each application one by one, and therefore the user experience is greatly improved by the solution of the present application.

[0257] FIG. 3D shows an interface diagram of the UI task automatic execution method provided in the embodiments of the present application applied to querying automatic renewal of a single application. As shown in (a) to (d) of FIG. 3D, in response to the user voice input 330 "Help me check whether XX application is automatically debited", the electronic device displays a prompt information 331 "OK, now jump to App and App to find the automatic renewal of X application", the electronic device performs an automatic query process (similar to the automatic query process shown in FIGS. 3A to 3B), and after the query is completed, the electronic device displays a prompt information 332 "XX video is found to be automatically renewed for the following services, do you want to cancel?" "X video App monthly automatic renewal", and a cancel control. In response to the operation of the user clicking the cancel control, the electronic device automatically cancels the monthly automatic renewal service of the X video App.

[0258] After the electronic device prompts that the renewal service has been canceled, the user can request to query the automatic renewal service again to confirm whether the automatic renewal service has been canceled by the electronic device. As shown in (e) and (f) in FIG. 3D, in response to the user voice input 333 "Help me check if X video is automatically charged", the electronic device performs an automatic query process, and after the query is completed, the electronic device displays prompt information 334 "No automatic charge of X video is found", and "You have the following services being automatically charged: renewal service (2), cancel all controls; supermarket App monthly renewal, cancel controls; and call App automatic deduction, cancel controls". Through this scheme, the electronic device can not only query the automatic renewal service of a certain application, but also query the automatic renewal service of other applications, thereby improving the user experience.

[0259] It should be noted that the implementation manner of automatically finding whether an application on the electronic device has the automatic renewal service is similar to the implementation manner of automatically finding whether all applications on the electronic device have the automatic renewal service in the above embodiment, and the difference between the two is that:

[0260] (1) Different user inputs. For example, "Help me check what application is automatically charged" can trigger the electronic device to automatically query which applications have the automatic renewal service among all applications, and "Help me check if XX application is automatically charged" can trigger the electronic device to automatically query whether a certain specified application has the automatic renewal service.

[0261] (2) Different query results. For example, the query result of "Help me check what application is automatically charged" is the automatic renewal service query result for all applications, and the query result of "Help me check if XX application is automatically charged" is the automatic renewal service query result for the application specified by the user.

[0262] Method 3: Automatic execution process of querying automatic renewal of multiple applications

[0263] Through the UI task automatic execution method provided in the embodiments of the present application, it can be automatically found whether two or more applications on the electronic device have the automatic renewal service in response to the user instruction, and the finding result about the automatic renewal service of the two or more applications can be quickly displayed for the user.

[0264] FIG. 3E shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to query the automatic renewal of multiple applications.

[0265] Figures 3E (a) and (b) show a schematic diagram of querying a single result. In response to the user voice input "Help me check if X video App and Y video App are automatically debited", the electronic device performs an automatic query process, and after the query is completed, the electronic device displays prompt information "Querying X video App is automatically debited for the following services, Y video App has no automatic renewal, do you want to cancel the service that X video App is renewing?" and "X video App VIP automatic renewal, cancel control".

[0266] Figures 3E (c) to (e) show a schematic diagram of querying multiple results. In response to the user voice input "Help me check if X video App and Y video App are automatically debited", the electronic device performs an automatic query process.

[0267] In some cases, after the query is completed, the electronic device displays prompt information "Querying X video App and Y video App are automatically debited for the following services, do you want to cancel?" and "Renewal service (2), cancel all controls"; "X video App VIP automatic renewal, cancel control"; "Y video App automatic renewal, cancel control".

[0268] In other cases, after the query is completed, the electronic device displays prompt information "Querying X video App and Y video App are automatically debited for the following services, do you want to cancel?" and "Renewal service (2), cancel all controls"; "X video App VIP automatic renewal, cancel control"; "X video App SVIP automatic renewal, cancel control"; "Y video App automatic renewal, cancel control".

[0269] Through the above scheme, when responding to user demand, querying the automatic renewal service of two applications, the electronic device can show the user the automatic renewal service of the two applications, and the user can cancel the automatic renewal service of the two applications according to actual use demand, improving the user experience.

[0270] Method 4 - automatic execution process for canceling automatic renewal

[0271] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can automatically find whether one or more applications or all applications on the electronic device have automatic renewal service in response to user instructions, and can quickly show the user the query result about the automatic renewal service of the application; in the case where a certain application has an automatic renewal service, the electronic device can automatically cancel the automatic renewal service of the application in response to user operation confirmation. Therefore, the automatic query function provided by the embodiments of the present application provides very convenient service for the user.

[0272] In some examples, in response to the user voice input "help me cancel the automatic deduction of an application", the electronic device performs an automatic query process, and after querying the automatic deduction service of the application, the automatic deduction service of the application is automatically canceled.

[0273] In some examples, in response to the user voice input "help me cancel the automatic deduction of an application", the electronic device performs an automatic query process, and after querying that the application has multiple automatic deduction services, the electronic device prompts the user to confirm whether to cancel one or all of the multiple automatic deduction services. After receiving the user confirmation operation, the electronic device automatically cancels one or all of the automatic deduction services of the application. In this way, the user experience can be improved by avoiding accidental cancellation.

[0274] FIG. 3F shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to canceling application renewal service. As shown in (a), (b) and (c) of FIG. 3F, in response to the user voice input "help me cancel the automatic deduction of X video App", the electronic device performs an automatic query process, and after the query is completed, the electronic device displays a prompt information "X video App is found to be automatically renewed for the following services, please confirm the cancellation: X video App VIP automatic renewal, cancel control". In response to the user instruction to confirm the cancellation, the electronic device automatically cancels the renewal service.

[0275] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can automatically find whether an application or each application on the electronic device has an automatic renewal service in response to a user instruction, and can quickly display the query result of the automatic renewal service to the user, and provide a shortcut control for triggering the cancellation of one or more automatic renewal services, or provide a shortcut control for confirming the cancellation of the automatic renewal service. The user can quickly cancel the automatic renewal service by only confirming, and the operation is convenient and fast. In this way, the user does not need to manually find the automatic renewal service of each application one by one, and therefore the user experience is greatly improved through the present application.

[0276] In some embodiments, the electronic device can display a smart recommendation card in the desktop. The smart recommendation card can prompt to query the automatic renewal service. In response to the user operation on the smart recommendation card, the electronic device can trigger to start the automatic query of the automatic renewal service.

[0277] Dynamic light effect reminder in automatic execution process

[0278] The dynamic light effect reminder provided by the embodiments of the present application is a new dynamic effect interaction form, which is a smart light effect scheme throughout the system as a whole, and can be applied to the automatic execution process to prompt the user that the electronic device is currently in the automatic execution process.

[0279] The following will be described by taking the application of the dynamic light effect prompt in the automatic query automatic recharge service process as an example.

[0280] In the embodiments of the present application, in the process of automatically querying the automatic recharge service of the application by the electronic device in response to the user operation, the electronic device performs the page automatic jump stage. In the page automatic jump stage, each page is in the page locked state, and the electronic device displays the light effect prompt pattern in the periphery of the current page. After the query ends, the electronic device releases the page locked state, and the electronic device cancels the display of the light effect prompt pattern.

[0281] In the embodiments of the present application, in the locked state, the electronic device displays the light effect prompt pattern in the periphery of the current page.

[0282] The light effect prompt pattern can be used to prompt that the current page is in the page locked state.

[0283] The light effect prompt pattern can also be used to prompt that the electronic device is currently performing the service query action.

[0284] In some embodiments, in the page locked state, or in the case that the electronic device displays the light effect prompt pattern in the periphery of the current page, if the electronic device receives the click operation of the user on the page, the electronic device can ignore the user input and not respond to the user input.

[0285] In some embodiments, in the page locked state, or in the case that the electronic device displays the light effect prompt pattern in the periphery of the current page, if the electronic device receives the click operation of the user on the page, the electronic device can prompt that the service query action is currently being performed.

[0286] FIG. 4A shows an interface schematic diagram of the dynamic light effect prompt in the UI task automatic execution method provided by the embodiments of the present application.

[0287] As shown in (a) of FIG. 4A, when the electronic device receives the query instruction of the user, the electronic device is in the page unlocked state.

[0288] As shown in (b) of FIG. 4A, when the electronic device performs the automatic query process, the electronic device is in the page locked state, and the electronic device displays the light effect prompt pattern in the periphery of the current page, prompting the user that the electronic device is currently performing the service query action. In the automatic query process of the electronic device, each jump page is in the page locked state and displays the light effect prompt pattern.

[0289] As shown in (c) of FIG. 4A, when the electronic device completes the query process and displays the query result, the electronic device cancels the page locked state and cancels the display of the light effect prompt pattern, prompting the user that the electronic device has completed the service query action.

[0290] It should be noted that the display form (for example, display color, display area, display change effect) of the light effect prompt pattern is not limited in the embodiments of the present application, and can be set according to actual use requirements.

[0291] For different business scenarios, the dynamic light effect (smart light effect) provided in the embodiments of the present application can appear following the target application area of the instruction interaction, or appear a start dynamic effect prompt when the system assistant starts the application, or appear a water ripple light effect when a certain application is invoked.

[0292] That is, the dynamic light effect prompt can be a dynamic effect prompt when the application starts, or a dynamic effect prompt when the business is executed, or a water ripple light effect when invoked.

[0293] The loading and display principle of the dynamic light effect provided in the embodiments of the present application will be described below.

[0294] In the embodiments of the present application, a level (referred to as an Overlay layer) can be customized in a three-party process. Only system applications can use the level, and the system applications can draw some customized effects on the level.

[0295] For ease of description, the level is referred to as a system customized dynamic effect level.

[0296] In the embodiments of the present application, the customized animation effect can be mounted to the system customized dynamic effect level through a smart service.

[0297] In the embodiments of the present application, the system customized dynamic effect level is set as follows: any click event is not processed; that is, in the case of receiving a click event of a user, the response of the click event can be directly ignored and transmitted to the next layer of the system customized dynamic effect level, that is, in the automatic execution process, each page is in a locked state, thereby ensuring that the action of the electronic device being executed is not affected by the click event.

[0298] In the embodiments of the present application, the system customized dynamic effect level can follow the task switching of the target application (for example, an App), and has good concomitance and display effect.

[0299] For example, the dynamic light effect is realized through the system customized dynamic effect level, and the display area of the dynamic light effect is strongly associated with the display task (Task) instance of the target application (for example, an App), and the dynamic light effect display effect can be refreshed following the display area of the target application (for example, an App) operated by the electronic device.

[0300] FIG. 4B shows a module interaction diagram for mounting and displaying a dynamic light effect provided in the embodiments of the present application.

[0301] Step 1: The intelligent perception service requests the window management service WMS to mount the system customized light effect layer.

[0302] The intelligent perception service has the ability to create, layout, draw and update the system customized light effect layer.

[0303] The intelligent perception service has the ability to receive, process and forward touch screen events.

[0304] The intelligent perception service has the ability to inject virtual events.

[0305] Step 2: The window management service WMS mounts the system customized light effect layer, and notifies the update state change when the target task state changes.

[0306] For example, the target task state change can refer to the window of the target application being switched from a full-screen window to a small window.

[0307] The update state change can include window form switching configuration change, focus change and / or visibility change.

[0308] Step 3: The intelligent perception service sends the touch screen event or the virtual event of the update state change to the input management service IMS, and requests the touch screen event or the virtual event corresponding to the update state change to be forwarded to the target application.

[0309] The input management service IMS has the ability to forward or dispatch touch screen events.

[0310] The input management service IMS has the ability to forward or dispatch virtual events.

[0311] Step 4: The input management service IMS dispatches the touch screen event or the virtual event corresponding to the update state change to the system customized light effect layer of the intelligent perception service.

[0312] When the target task state changes, the system customized light effect layer can change with the changes of the window size and the window position of the target application, has good follow-up to the target application, the display mode is more flexible, guarantees the coverage of all scenarios and all needs, and does not need to do business logic and complex customization in the original system framework, and does not damage the original display architecture.

[0313] Step 5: The input management service IMS forwards the touch screen event or the virtual event to the target application.

[0314] The present application does not limit the execution order of step 4 and step 5.

[0315] The dynamic light effect reminder provided by the embodiment of the present application can be applied to various interactive strong reminder businesses, can enhance the user's perception of the intelligent capability of the device, and improve the user experience.

[0316] In the embodiment of the present application, the light effect layer can be displayed by associating the light effect layer with a task (referred to as a Task Overlay mode), and the click event (i.e., the above-mentioned touch screen event) dispatching can also be implemented, and the virtual event injection can also be implemented. FIG. 4C shows an interaction timing diagram of light effect layer display, click event dispatching, and virtual event injection provided by the embodiment of the present application.

[0317] Light effect layer display

[0318] In the embodiment of the present application, the drawing management of the light effect layer is implemented by the wisdom perception service according to business requirements, and does not affect the interface drawing of the target task (Task) itself. The window management service plays a mediating role to associate the light effect layer with the target task, and the Overlay level can follow the task switching of the application, and is not simply displayed on the full screen or in a fixed area, and has good accompanying nature and display effect.

[0319] The specific interaction process of the light effect layer display includes:

[0320] Step 1-1. The wisdom perception service creates a light effect layer and performs light effect drawing.

[0321] Step 1-2. The wisdom perception service requests the window management service to attach the light effect layer as an Overlay layer to the target Task, which is one level higher than the top Activity level of the target Task.

[0322] Step 1-3. In scenarios such as horizontal and vertical screen rotation, window form switching between full screen, split screen, and floating window, target Task size change, focus change, and visibility change, the window management service notifies the wisdom perception service through a callback interface.

[0323] Step 1-4. The wisdom perception service updates the light effect layer display effect according to the target Task change.

[0324] Click event dispatching

[0325] When the wisdom perception service creates the light effect layer, the event handling mode in the Overlay state can be specified, and the handling mode has three types of receiving, transparent transmission, and double sending. When set to the receiving mode, the Input management service sends the click event to the light effect layer, which is handled by the wisdom perception service; when set to the transparent transmission mode, the click event is sent to the Activity of the target Task for processing, to ensure that the application response conforms to the original expectation; and when set to the double sending mode, the click event is sent to the light effect layer and the Activity of the target Task simultaneously.

[0326] The specific interaction process of the click event dispatching includes:

[0327] Step 2-1. When the user clicks the screen, the input management service MS calculates the target window being automatically executed.

[0328] Step 2-2. The input management service MS sets a flag according to the Overlay layer, determines to send the click event to the Overlay layer of the intelligent perception service, or transparently transmit to the Activity of the target task.

[0329] Step 2-3. The input management service MS dispatches the click event to the Overlay layer of the intelligent perception service.

[0330] Step 2-4. The input management service MS transparently transmits the click event to the Activity of the target task.

[0331] Step 2-5. The target application can respond to the click event and execute the target task.

[0332] Virtual event injection

[0333] The intelligent perception service can automatically generate a virtual event or event sequence, and send the virtual event to the Activity of the target task through the input management service MS to achieve automatic interaction control.

[0334] The specific interaction process of virtual event injection includes:

[0335] Step 3-1. The intelligent perception service requests the input management service MS to inject a virtual event.

[0336] Step 3-2. The input management service MS determines to transparently transmit the virtual event to the Activity of the target task.

[0337] Step 3-3. The input management service MS transparently transmits the virtual event to the Activity of the target task.

[0338] Step 3-4. The target application responds to the virtual event and automatically executes the target task.

[0339] Small window full screen switching in automatic execution process

[0340] In the embodiments of the present application, in the process of the electronic device responding to the user operation and executing the automatic query process, the electronic device will execute a page automatic jump stage, in the page automatic jump stage, each page is in a page lock state, and by default, the page is displayed in full screen, and the electronic device displays a light effect prompt pattern in the periphery of the page displayed in full screen. In the case of receiving a user operation, the electronic device can switch from page full screen display to page small window display, and in the case of switching to page small window display, the electronic device displays a light effect prompt pattern in the periphery of the page displayed in small window.

[0341] In the embodiments of the present application, the electronic device can switch between the page window display and the page full screen display (referred to as window full screen switching) in response to the user operation, and thus has good flexibility.

[0342] In some embodiments, in the case of displaying the page in full screen, the electronic device can switch from the page full screen display to the page window display in response to the user operation on the zoom-out control (the "zoom out to window" button) displayed in the full screen page.

[0343] In some embodiments, in the case of displaying the page in the window, the window is in a locked state, the window can keep the position unchanged, the window can keep the size unchanged, and the display page in the window can change with the change of the automatic query process.

[0344] The window display can also be referred to as the floating window display.

[0345] In some embodiments, the state text is displayed below the window.

[0346] In some embodiments, when the situation requiring user intervention occurs in the window state, the electronic device can prompt the user by, for example, highlighting the text, and / or vibrating to prompt the user. When the user clicks the window to enlarge the window, the electronic device receives the user operation and detects whether the user interaction is completed. After detecting that the user interaction is completed, the electronic device continues to automatically execute the process.

[0347] In some embodiments, when the fingerprint verification page appears in the window state, the electronic device can directly perform fingerprint verification in the window state.

[0348] In some embodiments, in the case of displaying the page in the window, the desktop of the electronic device is in an unlocked state, and the electronic device can normally respond to the user operation. For example, the electronic device can respond to the user operation in the desktop or the application interface to perform an action triggered by the user operation, such as displaying the page of a social application, playing a video screen of a video application, or playing audio of an audio application.

[0349] In some embodiments, in the case of displaying the page in the window, the electronic device can switch from the page window display to the page full screen display in response to the user operation on the window.

[0350] FIG. 5A shows an interface schematic diagram of window full screen switching in the UI task automatic execution method provided by the embodiments of the present application.

[0351] As shown in (a) of FIG. 5A, the electronic device is in the stage of performing automatic page jump, the current page 401 is a full-screen display page and is in a page locking state, and a light effect prompt pattern is displayed around the full-screen display page. The current page 401 displays a shrinking control 402.

[0352] When the electronic device receives a user operation on the shrinking control 402, as shown in (b) of FIG. 5A, the electronic device switches from the full-screen display page 401 to a desktop, and displays a small window 403 in the desktop, the small window 403 is in a locking state, and a light effect prompt pattern (for example, a dot pattern shown in the figure) is displayed around the small window.

[0353] The small window 403 is another display form of the full-screen display page 401. The display content of the small window 403 can be the same as or similar to that of the full-screen display page 401. The display position of the small window 403 can be a preset position in the display screen. The display size of the small window 403 can be a preset size. It can be understood that the display size of the small window 403 is smaller than that of the full-screen display page 401. It should be noted that the small window is in a locking state, and if the user clicks any position of the small window, the electronic device will switch the small window to a full-screen window, and the content displayed in the small window will not respond when being operated.

[0354] As shown in (c) and (d) of FIG. 5A, in the case of displaying the page in the form of a small window, in response to a user click operation on the small window 403, the electronic device can switch from page small window display to page full-screen display, and the small window 403 changes to a full-screen window 404, which is in a locking state. In the locking state, the electronic device does not respond to the user's operation on the full-screen window 404, for example, if the user clicks the favorite control in the full-screen window 404, the electronic device will not jump to the favorite page of the application. The electronic device can prompt the user that it is currently in an automatic execution state, and the user can choose to exit the automatic execution process.

[0355] It should be noted that in the process of the electronic device responding to the user operation to perform the automatic query process, when the page automatically jumps, an identity verification page or other pages requiring user confirmation will appear, in which case the user needs to manually operate, and at this time the identity verification page and the like will be paused to receive user input.

[0356] In some embodiments, in the case of displaying the identity verification page in the form of a small window, the electronic device prompts the user to switch the small window display to full-screen display, so that the user inputs the fingerprint or gesture in the full-screen display page for identity verification. After the user clicks the small window, the electronic device displays the identity verification page in full screen, and after the electronic device receives the user input and the identity verification is successful, the electronic device automatically switches from full-screen display to small window display, and the electronic device continues to perform page automatic jump and displays the page jump change in the small window.

[0357] In some embodiments, in the case that the identity verification page is displayed in the form of a small window, the electronic device automatically switches to full-screen display without user input, prompting the user to input a fingerprint or a gesture for identity verification. After the electronic device receives the user input and the identity verification is successful, the electronic device automatically switches from full-screen display to small window display, and the electronic device continues to perform page automatic jumping and display page jumping changes in the small window.

[0358] Exemplarily, as shown in (e) of FIG. 5A, the electronic device is in the stage of performing page automatic jumping, the current page 405 is a full-screen display page and is in a page locked state, and a light effect prompt pattern is displayed around the full-screen display page. The current page 405 displays a zoom-out control 406. When the electronic device receives the user operation on the zoom-out control 406, as shown in (f) of FIG. 5A, the electronic device switches from the full-screen display page 405 to the desktop, and displays a small window 407 in the desktop, the small window 407 is in a locked state, and a light effect prompt pattern is displayed around the small window. The display content in the small window 406 is an identity verification page. At this time, the user needs to input a fingerprint or a gesture for identity verification. As shown in (f) to (h) of FIG. 5A, the electronic device can automatically switch from page small window display to page full-screen display, the small window 407 changes to a full-screen window 408, and the full-screen window 408 is in an unlocked state. In the unlocked state, the electronic device can receive user input and perform identity verification according to the user input. After the identity verification is successful, the electronic device automatically switches from full-screen display to small window display 409, and the electronic device continues to perform page automatic jumping and display page jumping changes in the small window.

[0359] Through the scheme of the present application, in the process that the electronic device performs an automatic query process in response to user operation, the electronic device can switch between page small window display and page full-screen display in response to user operation, which is convenient for the user to use the electronic device and has good flexibility.

[0360] Automatic small window display in an immersive scene in an automatically executed process

[0361] In the embodiments of the present application, when the electronic device runs a certain application in the foreground, the electronic device can perform an automatic query process in response to a query instruction of the user. The electronic device can identify the application type of the application, and determine whether to display an automatically executed page in a small window or in full screen according to the application type.

[0362] In some examples, when the electronic device displays a desktop, the electronic device identifies that the foreground application is a desktop application type, and determines to display an automatically executed page in full screen according to the desktop application type.

[0363] In some examples, when the electronic device plays a video picture or displays a game picture, the electronic device identifies that the foreground application is of a preset application type, and determines to display the automatically executed page in a small window according to the preset application type, that is, to display the small window in suspension on the video picture or the game picture, thereby ensuring that the user is not disturbed in the immersive state.

[0364] In some embodiments, in the case where the page is displayed in full screen, the electronic device simulates a click on the "minimize to small window" button, thereby triggering the small window to display the page.

[0365] In the embodiments of the present application, in the process of executing the automatic query process and displaying the automatically executed page in full screen, if the electronic device receives an operation (such as a voice instruction) of the user playing a video picture or displaying a game picture, the electronic device identifies the application type of the application triggered to start, and when it is determined that the application type is a preset application type such as a video or a game, the electronic device can switch the automatically executed page from full screen display to small window display, to ensure that the user is not disturbed in the immersive state.

[0366] Table 5 shows the correspondence between the preset application type and the window display mode during automatic execution. It should be noted that Table 5 is illustrative and in actual implementation, it can be set according to actual needs.

[0367] Table 5

[0368] In some embodiments, after starting the automatic execution process, the electronic device displays the automatically executed page in a full screen window, and if the electronic device receives a video call request, the electronic device can switch to display the automatically executed page in a small window.

[0369] FIG. 5B shows an interface diagram of small window display in an immersive state in the UI task automatic execution method according to an embodiment of the present application. As shown in (a) of FIG. 5B, when the electronic device plays a video picture, the electronic device identifies that the foreground application is a game application, and determines to display the automatically executed page in a small window according to the game application, thereby ensuring that the user is not disturbed in the immersive state. As shown in (b) of FIG. 5B, when the electronic device plays a game picture, the electronic device identifies that the foreground application is a video application, and determines to display the automatically executed page in a small window according to the video application, thereby ensuring that the user is not disturbed in the immersive state.

[0370] Suspension and recovery of capabilities in the automatic execution process

[0371] In the implementation process, in the process of the electronic device responding to the user operation and performing the automatic query process, some situations that need manual intervention to ensure that the process can continue will be encountered, such as: human biometric identification; single selection or multiple selection of specified content; choice scenarios involving money and major choices, etc. For the above situations, the application provides a pause and resume function to ensure stable and reliable automatic execution.

[0372] In the embodiment of the application, in the process of the electronic device responding to the user operation and performing the automatic query process, when the page automatically jumps, pages such as identity verification or user confirmation are displayed, in which case the user needs to manually operate, at which time the page is paused to receive user input, and after the electronic device receives the user input, the electronic device continues to perform the automatic page jump.

[0373] Among them, in the pause state, the page enters the unlocked state. After receiving the user input, the pause ends, and the page enters the locked state.

[0374] Exit mode and abnormal situation in automatic execution process

[0375] In some examples, when the user clicks the stop button, the electronic device controls the automatic execution process to pause, triggers a secondary confirmation pop-up box, and displays an exit button and a cancel button; when the user clicks the exit button, the automatic execution process is stopped, and when the user clicks the cancel button, the automatic execution process continues.

[0376] In some examples, in the automatic execution process, when the user clicks other areas of the page, clicks the home key, triggers exit, triggers three-key navigation, or triggers side return, the electronic device controls the automatic execution process to pause, triggers a secondary confirmation pop-up box, and displays an exit button and a cancel button.

[0377] The implementation of exiting the automatic execution process will be described below with reference to the accompanying drawings.

[0378] FIG. 6A shows an interface schematic diagram of triggering the exit of the automatic execution process in the UI task automatic execution method provided by the embodiment of the application. As shown in (a) of FIG. 6A, in the automatic execution page, the stop control is included in the prompt information. In response to the user operation on the stop control, as shown in (b) of FIG. 6A, the electronic device displays the prompt information: whether to exit the automatic execution? The cancel control and the exit control. In response to the user operation on the exit control, as shown in (c) of FIG. 6A, the electronic device displays the prompt information: the automatic execution has been exited.

[0379] FIG. 6B shows another interface schematic diagram of triggering exit of automatic execution flow in the UI task automatic execution method provided in the embodiments of the present application. As shown in (a) of FIG. 6B, in the automatic execution page, in response to the operation of the user in the blank area, as shown in (b) of FIG. 6B, the electronic device displays prompt information: manual operation is not supported in automatic execution, whether to exit automatic execution? The cancel control and the exit control. In response to the operation of the user on the exit control, as shown in (c) of FIG. 6B, the electronic device displays prompt information: automatic execution has been exited.

[0380] In the automatic execution process, a dynamic light effect prompt pattern is displayed around the page. After the automatic execution exits, the dynamic light effect prompt pattern around the page disappears.

[0381] The implementation of the abnormal flow processing will be described below with reference to the accompanying drawings.

[0382] In some embodiments, when the user manually operates to enter a page of other non-RPA flow at the user takeover, causing the automatic execution to find no execution target, if there is no subsequent automatic execution of other applications, the automatic execution directly exits.

[0383] FIG. 6C shows an interface schematic diagram of abnormal flow processing in the UI task automatic execution method provided in the embodiments of the present application. After displaying the identity verification page, the interface is in a non-locked state, and the electronic device can receive user input. As shown in (a) of FIG. 6C, the identity verification page includes a close control, in response to the operation of the user on the close control, as shown in (b) of FIG. 6C, the electronic device displays prompt information: WeChat execution is abnormal. As shown in (c) of FIG. 6C, the electronic device displays prompt information: execution fails. At this time, the automatic execution finds no execution target (WeChat), and there is no subsequent automatic execution of other applications, and the automatic execution directly exits.

[0384] In other embodiments, when the user manually operates to enter a page of other non-RPA flow at the user takeover, causing the automatic execution to find no execution target, if there is subsequent automatic execution of other applications, the automatic execution of the current application ends, and the task of the next application is continued to be executed.

[0385] FIG. 6D shows another interface schematic diagram of abnormal flow processing in the UI task automatic execution method provided in the embodiments of the present application. As shown in (a) of FIG. 6D, in response to the operation of the user on the close control in the identity verification page, as shown in (b) of FIG. 6D, the electronic device displays prompt information: WeChat execution is abnormal, and will jump to Alipay to continue execution. As shown in (c) of FIG. 6D, the electronic device displays the page of Alipay, and continues to execute the business query flow. At this time, the automatic execution finds no execution target (WeChat), and there is subsequent automatic execution of other applications (Alipay), the automatic execution of the current application ends, and the task of the next application (Alipay) is continued to be executed.

[0386] In still other embodiments, when an exception occurs in automatic execution, resulting in failure to find an execution target, if there is no automatic execution of other applications subsequently, the automatic execution is directly exited; if there is automatic execution of other applications subsequently, the automatic execution of the current application is ended, and the task of the next application is executed.

[0387] Second embodiment: automatic execution process of sending a file to someone

[0388] In the second embodiment, in response to an instruction of user A requesting to send a file to user B, the file can be automatically sent to user B, and the path is jumped between pages, the file is first queried and the file interface is jumped to, then the sharing page is jumped to, and the file is sent to user B.

[0389] Table 6 shows various scenarios of sending a file to someone and exemplary user input instructions.

[0390] Table 6

[0391] In combination with Table 6, for sending a file to someone, there can be various user needs, different search methods, and various results can be found, such as searching for a file according to a file name; or searching for a file according to a source and a file name; or searching for a file according to a file type, requiring the user to select a file; or non-unique files and / or non-unique names appear, requiring the user to select a file and / or a name. The embodiments of the present application provide corresponding UI task automatic execution methods for various user needs. The following will be described in detail through various embodiments.

[0392] Automatic execution process 1 of sending a file: searching for a file according to a file name

[0393] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can automatically search for a file on the electronic device according to a file name in response to a user instruction, and can quickly display the search result to the user, and can automatically send the file to a specified user.

[0394] Table 7 schematically shows the UI task operation steps in the automatic execution process of sending a file, wherein the user input instruction is “help me send the file of design psychology to dad”. The UI task operation steps include jump page information, target controls in each page, and corresponding UI tasks of each page and whether user operation is required.

[0395] Table 7

[0396] The target control can be text, an icon, a button, and / or a search box, or can also be in other forms.

[0397] The interface implementation of the automatic execution flow of the file sending process is described below with reference to the accompanying drawings.

[0398] FIG. 7A shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to a file sending scenario.

[0399] As shown in (a) to (k) in FIG. 7A, the electronic device receives a user input instruction "Help me send the file Design Psychology to Dad", wakes up the system assistant, and in response to the user input instruction, prompts that the UI task will be automatically executed, for example, the reply information is "OK, now jump to the file management to find the file and send it to Dad". The electronic device can identify that the user demand is to send a file according to the user input instruction, and determine that the file sending scenario is to search for a file according to the file name, thereby determining the UI task operation steps and starting the automatic execution flow.

[0400] After starting the automatic execution flow, the electronic device automatically jumps to the file management page, identifies the target control "search box" in the file management page and determines the position coordinates of the search box, selects the search box at the determined position coordinates through simulated click operation, and inputs "Design Psychology" in the search box. The electronic device displays that one file is searched and displays the file name "Design Psychology" and a circle control. The electronic device identifies the target control "circle control" and determines the position coordinates of the circle control, and selects the circle control at the determined position coordinates through simulated click operation.

[0401] Among them, after starting the automatic execution flow, a light effect prompt pattern appears on the full screen, and the current state "Searching for you in file management" is displayed below the page, as well as a stop button and a button to shrink to a small window.

[0402] After automatically selecting the file, the electronic device automatically triggers file sharing and displays an application icon page. The electronic device identifies the target control "App1" in the application icon page and determines the position coordinates of the target control "App1", and selects the target control "App1" at the determined position coordinates through simulated click operation. Then, the electronic device automatically jumps from the file management page to the App1 contact page. Among them, after jumping to the App1 page, the state information becomes "automatically executing" or "sending for you in App1".

[0403] Then, the electronic device identifies the target control "search box" in the contact page and determines the search box position coordinates, selects the search box at the determined position coordinates through a simulated click operation, and inputs "Dad" in the search box. The electronic device displays that "Dad" is searched, and in the case where there is no control for the user to select, the target control is the default text "Dad". The electronic device identifies the target control "Dad" and determines the position coordinates, and selects the text "Dad" at the determined position coordinates through a simulated click operation.

[0404] After selecting the text "Dad", a sharing confirmation window is popped up, the electronic device identifies the target control "share" in the sharing confirmation window and determines the position coordinates of the target control, and selects the target control "share" at the determined position coordinates through a simulated click operation.

[0405] After selecting the text "Dad", a sharing completion window is popped up, the electronic device identifies the target control "return" in the sharing completion window and determines the position coordinates of the target control, and selects the target control "return" at the determined position coordinates through a simulated click operation.

[0406] After selecting the text "return", the automatic execution result is displayed, for example, "Design Psychology has been sent to Dad" is displayed.

[0407] Automatic execution process 2 of sending a file: searching for a file according to a source and a file name

[0408] According to the UI task automatic execution method provided in the embodiments of the present application, the electronic device can automatically search for a file on the electronic device according to a source and a file name in response to a user instruction, and can quickly display the search result to the user, and can automatically send the file to a specified user.

[0409] For example, the user inputs an instruction "Help me send the ticket photo received by WeChat to Dad".

[0410] Table 8 schematically shows the UI task operation steps in the automatic execution process of sending a file, which includes jump page information, target controls in each page, and corresponding UI tasks of each page and whether the user takes over.

[0411] Table 8

[0412] The target control can be a text, an icon, a button, and / or a search box (or an input box). The embodiments of the present application do not limit the form of the target control.

[0413] The interface implementation of the automatic execution process of sending a file is described schematically below with reference to the accompanying drawings.

[0414] FIG. 7B shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to a file sending scenario. As shown in (a) to (1) of FIG. 7B, the electronic device receives a user input instruction "Help me send the ticket photo received by WeChat to Dad", wakes up the system assistant, and starts an automatic execution process corresponding to file sending.

[0415] After starting the automatic execution process, the electronic device automatically jumps to a file management page, identifies a target control "Source" in the file management page and determines the position coordinates of the "Source", and then selects the "Source" by simulating a click operation.

[0416] Then, the electronic device jumps to the source page, identifies the target control "WeChat" and "Select button" in the source page and determines the position coordinates of the "Select button", and selects the "Select button" by simulating a click operation at the determined position coordinates.

[0417] Then, the electronic device jumps to the WeChat search page, identifies the search box in the WeChat search page and determines the position coordinates of the search box, and selects the search box by simulating a click operation at the determined position coordinates, and inputs "ticket" in the search box.

[0418] Then, the electronic device displays that "ticket" is searched, identifies the target control "ticket" and determines the position coordinates of the "ticket", and selects the "ticket" by simulating a click operation.

[0419] After selecting the "ticket", the electronic device automatically triggers file sharing and displays an application icon page. The electronic device identifies the target control "App1" in the application icon page and determines the position coordinates of the target control "App1", and selects the target control "App1" by simulating a click operation at the determined position coordinates.

[0420] Then, the electronic device automatically jumps to the App1 contact page, identifies the target control "search box" in the contact page and determines the position coordinates of the search box, and selects the search box by simulating a click operation, and inputs "Dad" in the search box. Then, the electronic device displays the search result, identifies the target control "Dad" in the search result and determines the position coordinates, and selects the "Dad" by simulating a click operation.

[0421] After selecting the text "Dad", a sharing confirmation window is popped up, the electronic device identifies the target control "share" in the sharing confirmation window and determines the position coordinates of the target control, and selects the target control "share" by simulating a click operation at the determined position coordinates.

[0422] After selecting the target control "share", the automatic execution result is displayed by the system assistant, for example, "The ticket photo has been sent to Dad" is displayed.

[0423] Automatic execution flow 3 of sending files: non-unique files and / or non-unique names

[0424] By the UI task automatic execution method provided in the embodiments of the present application, the electronic device can automatically find files on the electronic device according to the source and file name in response to the user instruction, and can quickly show the user the search result. When non-unique files and / or non-unique names appear, the user takes over the process, needs to select, and then automatically sends the file to the designated user.

[0425] In the embodiments of the present application, after entering the user takeover process, the electronic device can prompt the user to operate. For example, the prompt mode includes at least one of the following: a guide animation, a change in state prompt content, a change in light effect reminder pattern, vibration, and voice broadcast in a voice wake-up scene. Through various prompts, the user can operate in time to complete the automatic execution process.

[0426] Illustratively, the guide animation disappears after the user clicks any position on the screen.

[0427] Illustratively, after the guide animation disappears, the lower state becomes a "continue execution" button.

[0428] Illustratively, when the user operates the screen, the "continue execution" button is temporarily hidden and appears after a preset time length (for example, 1 second).

[0429] Illustratively, if the user clicks the "continue execution" button without selecting the file, the electronic device prompts an execution exception.

[0430] Illustratively, in the case of electronic device execution failure, the light effect disappears and prompts a failure.

[0431] Illustratively, if the user has selected the content and clicked the "continue execution" button, the electronic device starts to continue the automatic execution process.

[0432] The following takes the user input instruction "help me send the file design psychology to dad" as an example.

[0433] Table 9 schematically shows the UI task operation steps in the automatic execution flow of sending files. The UI task operation steps include jump page information, target controls in each page, and corresponding UI tasks of each page and whether the user takes over.

[0434] Table 9

[0435] The following schematically describes the interface implementation of the automatic execution flow of sending files in conjunction with the drawings.

[0436] FIG. 7C shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to a file sending scenario. As shown in (a) to (1) of FIG. 7C, the electronic device receives a user input instruction "Help me send the file design psychology to Dad", wakes up the system assistant, and starts an automatic execution process corresponding to the file sending.

[0437] After starting the automatic execution process, the electronic device automatically jumps to the file management page, identifies the target control "search box" in the file management page and determines the position coordinates of the "search box", and then inputs the text "design psychology" in the "search box" by simulating input operations.

[0438] Then, the electronic device displays the search results, identifies the target control "design psychology" in the search results, and prompts the user to take over when there are two target controls.

[0439] After the user selects one of the target controls, the electronic device continues the automatic execution process, automatically triggers file sharing, and displays an application icon page. The electronic device identifies the target control "App1" in the application icon page and determines the position coordinates of the target control "App1", and selects the target control "App1" at the determined position coordinates by simulating a click operation.

[0440] Then, the electronic device automatically jumps to the App1 contact page, identifies the target control "search box" in the contact page and determines the position coordinates of the search box, selects the search box by simulating a click operation, and inputs "Dad" in the search box. Then, the electronic device displays the search results, and identifies the target control "Dad" in the search results. When there are two target controls, the electronic device prompts the user to take over.

[0441] After the user selects one of the target controls, the electronic device continues the automatic execution process, pops up a sharing confirmation window, and the electronic device identifies the target control "share" in the sharing confirmation window and determines the position coordinates of the target control, and selects the target control "share" at the determined position coordinates by simulating a click operation.

[0442] After selecting the target control "share", the electronic device identifies the target control "back" and determines the position coordinates of the "back", and selects the target control "back" by simulating a click operation.

[0443] In this way, the electronic device combines user operations to complete the automatic execution process of the file sending task.

[0444] Automatic execution process 4 of sending a file: searching for a file according to a file type

[0445] Through the UI task automatic execution method provided in the embodiments of the present application, the electronic device can automatically find files on the electronic device according to file types in response to a user instruction, and can quickly show the user the finding result. When the finding result is multiple files, the user needs to select, and then the electronic device can automatically send the files to a specified user.

[0446] Exemplarily, the user input instruction is "help me send a photo to Dad". Table 10 schematically shows the UI task operation steps in the automatic execution process of sending files, which include jump page information, target controls in each page, and UI tasks corresponding to each page and whether the user takes over.

[0447] Table 10

[0448] The interface implementation of the automatic execution process of sending files is schematically described below with reference to the accompanying drawings.

[0449] FIG. 7D shows an interface diagram of the UI task automatic execution method provided in the embodiments of the present application applied to a file sending scenario. As shown in (a) to (l) in FIG. 7D, the electronic device receives a user input instruction "help me send a photo to Dad", wakes up the system assistant, and starts the automatic execution process corresponding to sending files.

[0450] After starting the automatic execution process, the electronic device automatically jumps to the file management page, identifies the target control "Browse" in the file management page and determines the position coordinates of "Browse", and then selects "Browse" through simulated click operation.

[0451] Then, the electronic device jumps to the browse page, identifies the target control "Picture" in the browse page and determines the position coordinates of "Picture", and selects "Picture" through simulated click operation.

[0452] Then, the electronic device jumps to the picture display page and prompts the user to manually operate, i.e., the user takes over.

[0453] After the user selects the picture, the electronic device continues the automatic execution process, automatically jumps to the App1 contact page, identifies the target control "Search box" in the contact page and determines the position coordinates of the search box, selects the search box through simulated click operation, and inputs "Dad" in the search box. Then, the electronic device displays the search result, identifies the target control "Dad" in the search result and selects "Dad" through simulated click operation.

[0454] Then, the electronic device automatically pops up a sharing confirmation window, the electronic device identifies the target control "Share" in the sharing confirmation window and determines the position coordinates of the target control, and selects "Share" through simulated click operation.

[0455] After selecting the target control "Share", the electronic device identifies the target control "Back" and determines the position coordinates of "Back", and selects the target control "Back" by simulating a click operation.

[0456] After selecting the target control "Back", the system assistant displays the automatic execution result, such as "Photo sent to Dad".

[0457] Therefore, the electronic device combines user operations to complete the automatic execution process of sending a file task.

[0458] It should be noted that the above embodiments are exemplarily described by taking the sharing / sending a file by an application as an example, and the specific implementation can be determined according to actual use requirements, which is not limited in the present application.

[0459] Third implementation: automatic execution process of closing application permissions

[0460] In the third implementation, in response to a user A's instruction to request to close certain permissions of an application, the path is used to jump between pages, jump to the permission page of the application, and automatically close the specified permissions of the application.

[0461] It should be noted that for sending a file to someone, there can be various user needs, and the application embodiments provide corresponding UI task automatic execution methods for various user needs and search results. The following will be described in detail through various embodiments.

[0462] Automatic execution mode 1: closing a single permission of a single application

[0463] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can respond to user instructions to determine an automatic execution path / step, and realize page jumping according to the automatic execution path / step, jump to the application permission setting page, and automatically close a certain permission of a specified application.

[0464] Exemplarily, the user input instruction is "Help me close the microphone permission of the browser".

[0465] Table 11 schematically shows the UI task operation steps in the automatic execution process of closing the application permissions. The UI task operation steps include jump page information, target controls in each page, and corresponding UI tasks of each page and whether the user takes over.

[0466] Table 11

[0467] ​FIG. 8 shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the scenario of closing application permissions. As shown in (a) to (g) of FIG. 8, the user inputs an instruction of "help me close the microphone permission of the browser", and the electronic device starts an automatic execution process corresponding to the closing of the application notification in response to the user input instruction. After the automatic execution process is started, the electronic device displays each page of the automatic execution process through a full-screen window, displays a dynamic light effect pattern around the window, and displays the state information "automatically executing" in each page.

[0468] The electronic device automatically jumps to the settings main page first, identifies the target control "application" in the settings main page and determines the position coordinates of the "application", and then simulates a click operation to select the "application".

[0469] Then, the electronic device automatically jumps to the application page, identifies the target control "permission management" in the application page and determines the position coordinates of the "permission management", and then simulates a click operation to select the "permission management".

[0470] Then, the electronic device automatically jumps to the permission management page, identifies the target control "microphone" in the permission management page and determines the position coordinates of the "microphone", and then simulates a click operation to select the "microphone".

[0471] Then, the electronic device automatically jumps to the microphone page, identifies the target control "browser" in the microphone page and determines the position coordinates of the "browser", and then simulates a click operation to select the "browser".

[0472] Then, the electronic device displays a pop-up window for setting the microphone permission of the browser, identifies the target control "disable" in the window and determines the position coordinates of the "disable", and then simulates a click operation to select the "disable".

[0473] After the electronic device selects the "disable", the automatic execution process ends. After the automatic execution process ends, the electronic device cancels the display of the dynamic light effect pattern and prompts through the system assistant that the microphone permission of the browser has been closed.

[0474] It should be noted that the microphone permission of the browser App is taken as an example for illustrative purposes, and the specific application can be determined according to actual use requirements, which is not limited in the present application.

[0475] Automatic execution mode 2 - closing multiple permissions of a single application

[0476] Through the UI task automatic execution method provided in the embodiments of the present application, the electronic device can automatically find multiple permission setting pages of a certain application on the electronic device in response to a user instruction, and can quickly show the user the finding result and automatically close multiple permissions of the application. It should be noted that the specific implementation process of closing multiple permissions of a single application is similar to the specific implementation process of closing a single permission of a single application, which will not be described here.

[0477] Automatic execution mode 3: closing multiple permissions of multiple applications

[0478] Through the UI task automatic execution method provided in the embodiments of the present application, the electronic device can automatically find multiple permission setting pages of multiple applications on the electronic device in response to a user instruction, and can quickly show the user the finding result and automatically close multiple permissions of the specified multiple applications. It should be noted that the specific implementation process of closing multiple permissions of multiple applications is similar to the specific implementation process of closing a single permission of a single application, which will not be described here.

[0479] It should be noted that the above embodiments take the automatic execution process of closing application permissions as an example for illustrative description, and it can be understood that in actual implementation, the method provided in the embodiments of the present application can also be applied to the automatic execution process of opening application permissions.

[0480] Fourth implementation: automatic execution process of optimizing overall machine performance

[0481] In the fourth implementation, the automatic execution path / steps can be determined in response to a user request to optimize the overall machine performance, and page jumping can be implemented according to the automatic execution path / steps, jumping to a performance optimization page and automatically optimizing the overall machine performance.

[0482] For example, the user inputs an instruction of “Help me optimize the phone”. Table 12 schematically shows the UI task operation steps in the automatic execution process of optimizing the overall machine performance. The UI task operation steps include jumping page information, target controls in each page, and corresponding UI tasks of each page and whether the user takes over.

[0483] Table 12

[0484] FIG. 9 shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the scenario of optimizing the overall performance of a machine, as shown in (a) to (f) of FIG. 9, the user input instruction is "help me optimize the phone", and the electronic device starts the automatic execution process corresponding to the application notification in response to the user input instruction. After the automatic execution process is started, the electronic device displays each page of the automatic execution process through a full-screen window, displays a dynamic light effect pattern around the window, and displays the status information "automatically executing" in each page.

[0485] The electronic device automatically jumps to the system manager page, identifies the target control "one-key optimization" on the system manager page, and determines the position coordinates of the "one-key optimization", and then simulates a click operation to select the "one-key optimization".

[0486] Then, the electronic device automatically jumps to the one-key optimization page, and displays the performance optimization progress on the one-key optimization page.

[0487] Then, the electronic device automatically jumps to the optimization completion page, identifies the target control "complete" on the optimization completion page, and determines the position coordinates of the "complete", and then simulates a click operation to select the "complete".

[0488] After the electronic device selects the "complete", the automatic execution process ends. After the automatic execution process ends, the electronic device cancels the display of the dynamic light effect pattern, and completes the task of optimizing the overall performance of the machine.

[0489] Fifth implementation: automatic execution process of closing application notification

[0490] In the fifth implementation, the automatic execution path / steps can be determined in response to the user's instruction to request to close the application notification, and the page jump can be realized according to the automatic execution path / steps to jump to the notification setting page and automatically close the application notification.

[0491] Automatic execution mode 1: closing the notification of a single application

[0492] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can automatically search for the notification setting page of a certain application on the electronic device in response to the user's instruction to request to close the notification of a single application, and can quickly display the search result for the user and automatically close the notification of a certain application.

[0493] Exemplarily, the user input instruction is "help me close the notification of App1". Table 13 schematically shows the UI task operation steps in the automatic execution process of closing the notification of a single application. The UI task operation steps include jump page information, target controls in each page, and whether the user takes over the corresponding UI task of each page.

[0494] Table 13

[0495] FIG. 10A shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the scenario of closing the notification of a single application. As shown in (a) to (h) of FIG. 10A, the user input instruction is “help me close the notification of App1”, and the electronic device starts the automatic execution process corresponding to the closing of the application notification in response to the user input instruction. After the automatic execution process is started, the electronic device displays each page of the automatic execution process through a full-screen window, displays a dynamic light effect pattern around the window, and displays the state information “automatically executing” in each page.

[0496] The electronic device automatically jumps to the setting main page, identifies the target control “notification and status bar” on the setting main page and determines the position coordinates of the “notification and status bar”, and then simulates a click operation to select the “notification and status bar”.

[0497] Then, the electronic device automatically jumps to the notification setting page, identifies the target control “search box” on the notification setting page and determines the position coordinates of the search box, and then simulates an input operation to input the text “App1” in the search box, achieving the purpose of automatic search.

[0498] Then, the electronic device automatically jumps to the App1 notification management page, identifies the target control “switch icon” of the “notification allowed” on the App1 notification management page and determines the position coordinates of the “switch icon”, and identifies the on state indicated by the “switch icon”. In the case where the “switch icon” indicates the on state, a sliding operation is simulated on the “switch icon” to make the “switch icon” indicate the off state.

[0499] After the electronic device “switch icon” indicates the off state, the automatic execution process ends. After the automatic execution process ends, the electronic device cancels the display of the dynamic light effect pattern, and completes the task of closing the application notification this time.

[0500] Automatic execution mode 2-closing the notification of multiple applications

[0501] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can respond to the user instruction, automatically search the notification setting page of a certain application on the electronic device, and quickly show the user the search result, and automatically close the notification of a certain application.

[0502] Exemplarily, the user input instruction is “help me close the notification of App1 and App2”. Table 14 schematically shows the UI task operation steps in the automatic execution process of closing the notification of multiple applications. The UI task operation steps include jump page information, target controls in each page, and UI tasks corresponding to each page and whether the user takes over.

[0503] Table 14

[0504] FIG. 10B shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the scenario of closing multiple application notifications. As shown in (a) to (k) in FIG. 10B, the user input instruction is “help me close the notifications of App1 and App2”, and the electronic device starts the automatic execution process corresponding to closing multiple application notifications in response to the user input instruction. Among them, the notification of App1 is automatically closed first, and then the notification of App2 is automatically closed. The specific automatic execution process is similar to that of FIG. 10A, which will not be described here.

[0505] It should be noted that the closing of the notifications of App1 and App2 is taken as an example for illustrative purposes, and the specific implementation can be determined according to actual use requirements, which is not limited in the present application.

[0506] It should be noted that the automatic execution process of closing the application notification is taken as an example for illustrative purposes, and it can be understood that the method provided by the embodiments of the present application can also be applied to the automatic execution process of opening the application notification in actual implementation.

[0507] Sixth implementation: automatic execution process of ordering takeout

[0508] In the sixth implementation, the automatic execution path / steps can be determined in response to the user's instruction to order takeout, and page jumping can be realized according to the automatic execution path / steps, automatically jumping to the drink ordering page, and ordering for the user according to the user's operation. For example, the takeout scenario can include ordering drinks such as coffee / milk tea, as well as other ordering scenarios. For ease of illustration, the following will take ordering drinks as an example for illustrative purposes.

[0509] Automatic execution method 1: ordering a cup of drink with detailed instructions

[0510] Through the UI task automatic execution method provided by the embodiments of the present application, the electronic device can automatically search for the takeout page of a certain application on the electronic device in response to the user's instruction (with detailed information), and can quickly show the user the search result, and order for the user according to the user's operation.

[0511] Illustratively, the user input instruction is “help me order a large ice Americano coffee from C store”. Table 15 shows the UI task operation steps in the takeout automatic execution process. The UI task operation steps include page jumping information, target controls in each page, and UI tasks corresponding to each page and whether the user takes over.

[0512] Table 15

[0513] FIG. 11A shows a schematic diagram of a UI task automatic execution method provided by an embodiment of the present application applied to a scenario of ordering takeout. As shown in (a) to (i) of FIG. 11A, the user inputs an instruction of “Help me order a large iced Americano coffee from C store”, and the electronic device starts an automatic execution process corresponding to ordering takeout in response to the user input instruction. After the automatic execution process is started, the electronic device displays each page of the automatic execution process through a full-screen window, displays a dynamic light effect pattern around the window, and displays status information “Automatically executing” in each page.

[0514] The electronic device automatically jumps to a takeout page, identifies a target control “search box” in the takeout page and determines the position coordinates of the search box, then simulates an input operation to input the text “C store” into the search box, achieving the purpose of automatic search.

[0515] Then, the electronic device automatically jumps to a C store page, identifies a target control “search box” in the C store page and determines the position coordinates of the search box, then simulates an input operation to input the text “American coffee” into the search box, achieving the purpose of automatic search.

[0516] Then, the electronic device automatically jumps to a product page, identifies multiple specifications such as cup type: large cup, temperature: ice, etc. in the product page, and determines the position coordinates of target controls such as “large cup” and “ice”, then simulates a click operation to select the target controls such as “large cup” and “ice”.

[0517] After completing the automatic selection of specifications, the electronic device identifies a target control “add to cart” in the product page and determines the position coordinates of the “add to cart”, then simulates a click operation to select the “add to cart”.

[0518] Then, the electronic device automatically jumps to a shopping cart page, and the automatic execution process ends.

[0519] After the automatic execution process ends, the electronic device cancels the display of the dynamic light effect pattern, and updates the status information to “Added to cart, please settle the payment”, indicating that the user takes over. The user can click a settlement button on the shopping cart page according to actual needs, and then click an immediate payment button on an order payment page to complete the current ordering of drinks task.

[0520] Automatic execution mode 2 - ordering a cup of drinks without detailed instructions

[0521] In some scenarios, the user input instruction is relatively simple, and the electronic device can more accurately determine the user's intention, predict the user's selected product information, and confirm or adjust it by the user, so that the automatic execution process can better meet the user's needs in combination with user behavior habit data.

[0522] In some embodiments, if the merchant is included in the user instruction, the automatic check is performed according to the user instruction and habit, and the reply such as "OK, the relevant specifications are selected according to your habits. Please confirm your order of the goods and specifications in xx (merchant name):" is displayed, and the goods and specification cards are displayed.

[0523] In some embodiments, if the merchant is not included in the user instruction, the automatic check is performed according to the user instruction and habit, and the reply such as "OK, the relevant specifications are selected according to your habits. Please confirm your order of the goods and specifications:" is displayed, and the merchant, goods and specification cards are displayed.

[0524] In some embodiments, three pieces of merchant information are displayed when the merchant information bar is not expanded. The merchant information bar can be automatically expanded to display more merchant information.

[0525] In some embodiments, the goods are displayed in a maximum of four rows of options, and the electronic device can simulate a click on "view more" to jump to an external App / miniprogram page to select goods according to user habits.

[0526] In some embodiments, the specification card displays all obtained goods specifications.

[0527] In some embodiments, the automatic execution of the text specification on the card can be: order xx (number) cups of xx (merchant name) xx (specification description, separated by commas) xx (goods name). For example, order one cup of C store large cup, ice, American coffee.

[0528] Exemplarily, the user inputs an instruction "help me order a cup of coffee". Table 16 schematically shows the UI task operation steps in the automatic execution process of ordering takeout. The UI task operation steps include jump page information, target controls in each page, and corresponding UI tasks of each page and whether the user takes over.

[0529] Table 16

[0530] FIG. 11B shows an interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the ordering takeout scene. As shown in (a) to (i) of FIG. 11B, the user inputs an instruction "help me order a cup of coffee", and the electronic device responds to the user input instruction to start the automatic execution process corresponding to ordering takeout.

[0531] The difference from FIG. 11A is that, as shown in (a)-(c) of FIG. 11B, since the user input instruction contains less information, the electronic device predicts the user's information for ordering the goods according to the user input instruction and in combination with the user behavior habit data at the beginning of the automatic execution process, and displays the goods information such as (merchant: C store; goods: American coffee; cup type: large cup; temperature: ice) and prompts the user to confirm, and the user can take over to adjust the displayed goods information according to the actual needs. After the user confirms, the electronic device will continue to automatically execute the process based on the goods information confirmed by the user. The automatic execution process shown in (d)-(i) of FIG. 11B is similar to the automatic execution process shown in (d)-(i) of FIG. 11A, which will not be described here.

[0532] Automatic execution mode 3 - multi-cup beverage

[0533] In some scenarios, the user input instruction contains two items of goods information. Exemplarily, the user input instruction is "help me order a cup of C store ice American coffee and H store hot pearl milk tea". Table 17 schematically shows the UI task operation steps in the automatic execution process of ordering multi-cup beverages. The UI task operation steps include the jump page information, the target control in each page, and the corresponding UI task of each page and whether the user takes over.

[0534] Table 17

[0535] FIG. 11C shows the interface schematic diagram of the UI task automatic execution method provided by the embodiments of the present application applied to the ordering of take-out scenarios. As shown in (a)-(l) of FIG. 11C, the user input instruction is "help me order a cup of C store ice American coffee and H store hot pearl milk tea", and the electronic device starts the automatic execution process corresponding to ordering take-out in response to the user input instruction.

[0536] The difference from FIG. 11A is that, as shown in (d)-(f) of FIG. 11C, the electronic device automatically searches for merchants, automatically searches for goods, automatically selects goods specifications, and automatically adds them to the shopping cart to complete the automatic ordering of C store ice American coffee. Then, as shown in (g)-(i) of FIG. 11C, the electronic device automatically searches for merchants, automatically searches for goods, automatically selects goods specifications, and automatically adds them to the shopping cart to complete the automatic ordering of H store hot pearl milk tea. Then, as shown in (j) of FIG. 11C, the electronic device displays the shopping cart page, and the shopping cart page displays two items of goods information. The automatic execution process shown in (k)-(l) of FIG. 11C is similar to the automatic execution process shown in (h)-(i) of FIG. 11A, which will not be described here.

[0537] Support for window switching and parallel execution in automatic execution process

[0538] FIG. 11D shows an interface schematic diagram of the UI task automatic execution method in the take-out scenario according to an embodiment of the present application. As shown in (a) to (k) of FIG. 11D, in the automatic execution process, when the user clicks the button for reducing to a small window, the desktop is returned, and the current automatic execution page is switched to a small window display, and the automatic execution process will not be interrupted. When the user clicks the address book icon in the desktop, the electronic device jumps from the desktop to the address book page, and the automatic execution process continues to be displayed in the small window, and the automatic execution process and the user click operation can be executed in parallel. When the user clicks the small window, the electronic device switches the small window display to full-screen display, at which time the address book page is returned to the background. When the user clicks the button for reducing to a small window, the electronic device switches the full-screen display to a small window display, at which time the address book page is displayed in the full-screen window, at which time the full-screen window and the small window are executed in parallel.

[0539] The interface schematic diagram of the UI task automatic execution method according to an embodiment of the present application is described in detail above, and the internal algorithm logic of the UI task automatic execution method is described in detail below.

[0540] In order to realize more intelligent and perfect UI task automatic execution technology, an overall algorithm technical framework for mobile phone UI task automatic execution is proposed in an embodiment of the present application. The framework can recognize user intent, automatically execute corresponding task operation sequences (or task operation steps) based on multi-modal screen fusion perception technology, and handle various abnormal situations and user interaction requirements during execution.

[0541] In order to better illustrate the implementation of each branch of the internal algorithm logic, the overall architecture is first described below in conjunction with multiple drawings.

[0542] As shown in FIG. 2A, the electronic device can receive and recognize user instructions to obtain user intent information, and determine UI task operation steps according to the user intent information. According to the UI task operation steps, the automatic execution process is started, and the pages of each step are displayed in turn; after all the steps are executed, the automatic execution process is ended, and the automatic execution result is displayed.

[0543] Among them, the task operation sequence ID can be determined according to the user intent information, and the robotic process automation (RPA) configuration information corresponding to the task operation sequence ID is obtained, and then the UI task operation steps can be determined based on the RPA configuration information, and the specific implementation process will be described below.

[0544] FIG. 12A is a more detailed flowchart based on FIG. 2A. Four execution scenarios that may occur in the automatic execution process are embodied in S203 in FIG. 12A.

[0545] Referring to the content shown in the dashed box of FIG. 12A, according to the UI task operation steps, the automatic execution of the UI task is started, and in the process of the automatic execution of the UI task, it is determined whether an interruption event occurs. Exemplarily, in the process of the automatic execution of the UI task, if it is detected that the user needs to take over, or an advertisement pop-up window occurs, or a target control is not recognized, etc., an interruption event is triggered.

[0546] Execution scenario 1: In the case where no interruption event occurs, the automatic execution of the UI task is continued, and it is determined whether an interruption event occurs.

[0547] Execution scenario 2: In the case where an interruption event occurs and the interruption event is triggered by the need for the user to take over, the user input is received, after the user interaction is completed, the page node is relocated and the optimal path is planned, and then the automatic execution of the UI task is continued according to the relocated page node and the planned optimal path, and it is determined whether an interruption event occurs. The case where the user needs to take over includes the appearance of an identity verification page, a page with multiple search results, an information confirmation page, or a password input page.

[0548] Execution scenario 3: In the case where an interruption event occurs and the interruption event is triggered by the appearance of an advertisement pop-up window, the advertisement pop-up window is automatically closed, and then the automatic execution of the UI task is continued, and it is determined whether an interruption event occurs.

[0549] Execution scenario 4: In the case where an interruption event occurs and the interruption event is triggered by the failure to find a target control, it is determined whether the page is loaded, after the page is loaded, the page is scrolled to display to find the target control, after the target control is found, the automatic execution of the UI task is continued, and it is determined whether an interruption event occurs.

[0550] Among them, the automatic execution of the UI task includes identifying a target control in a page and determining the position coordinates of the target position, simulating a click / slide / input operation on the target control, and jumping from the current page to the next page.

[0551] FIG. 12B shows a schematic diagram of the interaction process of the functional modules involved in the above-mentioned UI task automatic execution method.

[0552] Among them, the functional modules involved in the UI task automatic execution method can include a system assistant, a task scheduling module, a screen perception module, a task execution module, a user interaction detection module, and a relocation and path planning module.

[0553] Briefly described, the system assistant identifies the user intent based on the voice instruction of the user, determines the task operation sequence ID and extracts the slot information according to the user intent, and delivers the task operation sequence ID and the slot information to the task scheduling module. The task scheduling module arranges the task operation sequence according to the recognized task operation sequence ID and the slot information. If the task needs to be taken over by the user, the system will prompt the user; if the task does not need to be taken over by the user, the task operation sequence will be issued to the screen perception module. The screen perception module comprehensively analyzes according to the page structure information (such as XML information) and the visual information (such as screenshots), locates the target control, detects whether an advertisement pop-up window appears or the target control is not found, and detects the user interaction demand to determine whether the user needs to take over. If the user needs to take over, the user is prompted to operate, and during the interaction process, the user interaction detection module continuously monitors the operation state of the user to determine whether the user interaction has been completed. After detecting that the user interaction is completed, the repositioning and path planning module locates the current page node and plans the optimal execution path / step based on the current page node to continue executing the remaining task operation steps.

[0554] S1-S2, the system assistant receives a user instruction, obtains intent information and slot information according to the user instruction, and sends the intent information and the slot information to the task scheduling module.

[0555] For example, the user instruction is a voice input "order a large cup of iced American coffee", the electronic device can receive and recognize the voice input, obtain the user intent information "order takeout", and obtain the keywords "large cup; iced; American" from the voice input as slot information.

[0556] S3, the task scheduling module obtains RPA configuration information according to the intent information, and determines a UI task operation step according to the RPA configuration information and the slot information. The UI task operation step includes M steps with serial numbers and the page, target control and UI task of each step. Wherein, M is an integer greater than 1.

[0557] It should be noted that one user intent information is preset to correspond to one task operation sequence ID. For example, referring to Table 1, the user intent information "query automatic renewal" corresponds to the sequence ID "1"; the user intent information "order takeout" corresponds to the sequence ID "6".

[0558] It should also be noted that each task operation sequence ID corresponds to a preset RPA configuration information. The RPA configuration information corresponding to different sequence IDs is different. Wherein, a preset RPA configuration information includes one or more task items, and each task item includes multiple slots, and each slot value is empty and in a to-be-filled state.

[0559] That is, the task operation sequence ID can be acquired according to the user intention information, and the RPA configuration information corresponding to the task operation sequence ID can be acquired. Then, each slot in the RPA configuration information is filled according to the slot information, and the UI task operation step is obtained through task arrangement.

[0560] Exemplarily, the configuration information corresponding to one step includes one page, one target control, and one UI task. Alternatively, one step includes one page, multiple target controls, and multiple UI tasks. Alternatively, one step includes one page and one pop-up window, multiple target controls, and multiple UI tasks.

[0561] S4, the task scheduling module determines whether user takeover is needed according to the UI task operation step.

[0562] For example, one step in the UI task operation step is to display an identity verification page, and accordingly, user takeover is needed when the step is executed.

[0563] S5, in the case where user takeover is not needed, the task scheduling module instructs the screen perception module to execute the i-th step of the UI task.

[0564] In the embodiments of the present application, the task scheduling module is responsible for executing the following tasks, including: in the process of automatically executing the UI task, the task scheduling module receives the task operation sequence ID and the slot information, and then arranges the task operation sequence according to the task operation sequence ID and the slot information. When each step is executed according to the task operation sequence, it is first determined whether user takeover is needed. If user takeover is needed, the user is prompted to operate; otherwise, the task operation sequence is continued to be scheduled, and the configuration information of each operation is sent to the screen perception module, and the screen perception module executes the operation, so as to ensure that the task operation sequence can be executed smoothly. Moreover, the task scheduling module can dynamically adjust the scheduling and arrangement of the task sequence according to the feedback received from other modules in the process of task execution.

[0565] S6, the screen perception module displays the page of the i-th step in response to the instruction, identifies the target control of the i-th step in the page and determines the position coordinates of the target control. Alternatively, the target control can be identified and the position coordinates of the target control can be determined through the task execution module.

[0566] S7, the screen perception module determines whether a pop-up window that needs user takeover appears.

[0567] For example, when the user clicks the exit button or clicks on a blank area in the page, a pop-up window is triggered, and the content of the pop-up window can be "whether to exit the execution, as the current automatic execution is in progress". The screen perception module can determine that the pop-up window is a pop-up window that needs to be taken over by the user through a pop-up window detection algorithm, and needs to be operated by the user.

[0568] S8, the screen perception module determines whether an advertisement pop-up window appears.

[0569] S9, the screen perception module determines whether the target control is not found.

[0570] S10, in the case that no pop-up window that needs to be taken over by the user appears, no advertisement pop-up window appears, and the target control is recognized, the screen perception module sends the position coordinates of the target control to the task execution module, and instructs to execute the UI task of the i-th step.

[0571] S11, the task execution module responds to the instruction, and simulates clicking the target control according to the position coordinates of the target control.

[0572] In some examples, when simulating clicking the target control, a cursor can be displayed on the target control.

[0573] In the automatic execution process, the task execution module can perform simulated operations such as clicking, sliding, or inputting. It can be understood that the simulated clicking is exemplarily illustrated here.

[0574] S12, the task execution module notifies the task scheduling module that the UI task of the i-th step has been completed.

[0575] S13, the task scheduling module determines whether i is less than M.

[0576] In the case that i is less than M, the task scheduling module adds 1 to i, i.e. i=i+1, returns to S4, and instructs to automatically execute the UI task of the i-th step, i.e. after completing the UI task of the current step, the UI task of the next step is continued to be executed.

[0577] In the case that i is equal to M, the automatic execution task of all steps is completed.

[0578] S14-S16, in the case that the user needs to take over during the automatic execution of the UI task, the screen perception module notifies the user interaction detection module, the user interaction detection module obtains user interaction information, and sends the user interaction information to the repositioning and path planning module. The repositioning and path planning module repositions the page node and plans the optimal path according to the user interaction information, and notifies the task scheduling module to reschedule. Correspondingly, the task scheduling module will schedule according to the repositioned page node and the planned optimal path, and continue the automatic execution process.

[0579] S17, in the case of detecting the occurrence of an advertisement pop-up window in the process of automatically executing the UI task, the screen perception module notifies the task execution module to close the advertisement pop-up window, and the task execution module identifies the close button of the advertisement pop-up window and automatically closes the advertisement pop-up window. After the advertisement pop-up window is closed, the automatic execution process continues.

[0580] Alternatively, in the case of detecting the occurrence of an advertisement pop-up window in the process of automatically executing the UI task, the screen perception module identifies the close button of the advertisement pop-up window and notifies the task execution module to close the advertisement pop-up window, and the task execution module automatically closes the advertisement pop-up window. After the advertisement pop-up window is closed, the automatic execution process continues.

[0581] S18, in the case of not finding the target control in the process of automatically executing the UI task, the screen perception module notifies the task execution module to find the target control again, and the task execution module scrolls to display after the page is loaded, and finds the target control. Wherein, finding the target control can include identifying the target control and determining the position coordinates of the target control. Then, the task execution module simulates clicking the target control according to the position coordinates of the target control, and continues the automatic execution process.

[0582] Through the scheme of the present application, a series of preset task operations can be automatically executed according to the user's intention. Moreover, through the scheme of the present application, not only complex operation sequences can be processed, but also diversified abnormal scenarios such as the need for user intervention, the occurrence of an advertisement pop-up window, or the inability to find a target control can be processed. Moreover, the target control can be accurately identified based on a multi-scale control matching algorithm, and the user can be prompted in time when user interaction is needed, so that the automatic execution process can be successfully completed according to the user's needs, helping the user to save a lot of time and effort.

[0583] The overall algorithm framework of the automatic execution process will be described below in conjunction with FIG. 12C.

[0584] It should be noted that in some cases, the page content is not fully displayed, which may result in the inability to find the target control in the page. The present application can provide a preset number of retry opportunities, and the single-step execution number is judged when each step is executed. When the single-step execution number is less than a threshold value, the target control can be detected again. The specific algorithm implementation process is described below.

[0585] S20-S21, after the process is started, the task operation sequence configuration information of the current step (ID) is read, and the execution number indicator is cleared.

[0586] S22, judge whether the execution number of the current step (ID) is greater than a preset threshold value (such as a preset threshold value of 3).

[0587] In the case where the execution number is greater than the threshold value, the execution failure is reported and the process is ended.

[0588] S23, display the page according to the configuration information, obtain the screenshot and structured information of the page, in the case that the execution times are less than or equal to the threshold value.

[0589] S24A, respectively, visual detection and structured information detection are performed, and the target control is detected based on the page structure and visual features (and page layout). For example, the target control can be located by comprehensively utilizing the structured information and the visual detection result. When the structured information and the visual detection result cannot achieve the expected effect, a page layout extraction algorithm is called for auxiliary analysis.

[0590] S24B, a visual algorithm is used to detect a pop-up window based on page visual features. For example, the pop-up window includes two types as follows: an advertisement pop-up window that needs to be closed, and a pop-up window that needs user interaction (i.e., user takeover).

[0591] S24C, a visual algorithm is used to detect a page loading state based on page visual features. The page loading state is completed or not completed.

[0592] S24A, S24B and S24C can be executed in parallel, and the specific implementation of each algorithm will be described in detail below.

[0593] S25, it is determined whether there is a pop-up window that needs user takeover. If there is a pop-up window that needs user takeover, the user takeover stage is entered.

[0594] S26, in the case that it is determined that there is no pop-up window that needs user takeover, it is determined whether there is an advertisement pop-up window. If there is an advertisement pop-up window, the automatic closing of the advertisement pop-up window is triggered. After the closing operation is executed, the target control detection process is triggered again.

[0595] S27, in the case that it is determined that there is no advertisement pop-up window, it is determined whether the target control is not detected.

[0596] S28-S30, if the target control is detected, a simulated click operation is performed on the target control. It is determined whether the current step is the last step. If the current step is the last step, it is reported that the execution is successful, and the process ends. If the current step is not the last step, the step ID is incremented (i.e., ID = ID + 1) and the S20 is returned to be executed.

[0597] S31-S33, if the target control is not detected, it is further determined whether the page loading state. If the page has been loaded, a screen scrolling operation is performed to trigger the scrolling display of the page. Then, the single-step execution times are added by 1, and the S22 is returned to be executed, and the target control detection is performed again to try to find the target control.

[0598] The possible implementation manners of the above four execution scenarios are described in detail as follows.

[0599] Execution scenario 1: automatic execution process without interruption events throughout the process

[0600] In this execution scenario, no interruption events occur throughout the process of automatically executing the UI task, for example, no user takeover is required for each step of the automatic execution, no ad pop-up occurs, and no unrecognized target control occurs. The automatic execution process of the UI task in this execution scenario is described in detail below in terms of UI interface and algorithm implementation.

[0601] First, the difference between FIG. 13A and FIG. 12B needs to be explained. FIG. 12B shows four execution scenarios in the automatic execution process from a whole perspective; the execution module includes a system assistant, a task scheduling module, a screen perception module, a task execution module, a user interaction detection module, and a relocation and path planning module.

[0602] FIG. 13A mainly shows the automatic execution process without interruption events throughout the process (i.e., execution scenario 1). In FIG. 13A, the detailed interaction process between the system assistant and the UI agent module is included, the system assistant includes a user interaction unit, an intent recognition unit, and a dialogue management unit, the UI agent module includes a task scheduling module, a screen perception module, a task execution module, and a business result acquisition module, the task scheduling module includes a scheduling manager, a task manager, an RPA configuration manager, and an interaction manager.

[0603] The module interaction diagram of the automatic execution process without interruption events throughout the process is described in detail below in conjunction with FIG. 13A.

[0604] S40-S46, the user interaction unit receives the user instruction, the intent recognition unit identifies the user intent according to the user instruction to obtain the intent information. Then, the dialogue management unit matches the UI agent task (such as ordering takeout) according to the intent information and obtains the slot information of the UI agent task (such as a cup, C store, large cup, ice, latte). Then, the dialogue management unit sends the intent information and the slot information to the task scheduling module (such as the scheduling manager and the task manager) to apply for executing the UI agent task.

[0605] The user instruction can be a voice input, a text input, or a user input through other means.

[0606] In some embodiments, when the user instruction contains a lot of information, the slot information of the UI agent task can be directly obtained. For example, the user instruction is a voice input "order a large cup of ice latte in C store", and based on the user instruction, the slot information "a cup, C store, large cup, ice, latte" can be obtained.

[0607] In some embodiments, when the user instruction contains less information, the slot information of the UI agent task can be predicted according to the user instruction and the user behavior habit data, and the slot information can be determined after the user confirms or adjusts. For example, the user instruction is a voice input "order a cup of iced latte", and the user behavior habit data is "C store, large cup, ice, latte". Based on the user instruction and the user behavior habit data, the slot information "a cup, C store, large cup, ice, latte" is predicted, and the user is prompted to confirm or adjust. In the case of receiving a user confirmation operation, the dialog management unit determines that the slot information is "a cup, C store, large cup, ice, latte".

[0608] S47-S49, the task manager sends the intent information to the RPA configuration manager to apply for RPA configuration information. The RPA configuration manager finds the RPA ID corresponding to the intent information from the configuration library, and obtains the RPA configuration information corresponding to the RPA ID. The RPA configuration manager returns the RPA configuration information to the task manager.

[0609] Among them, the configuration library stores a preset corresponding relationship between the intent information and the RPA ID.

[0610] Exemplarily, the intent information "order takeout" corresponds to the RPA ID "6". The RPA configuration information corresponding to the RPA ID "6" is:

[0611] "order(app=, shop=, product=, no=, size=, temperature=, sweetness=, desLocation=)".

[0612] The RPA configuration information includes parameters such as application app, merchant shop, product product, quantity no, cup size, temperature temperature, sweetness sweetness, and location desLocation. Each parameter has a slot and the slot value is empty.

[0613] S50-S51, the task manager fills the slot in the RPA configuration information according to the slot information, generates the UI task operation steps through task scheduling, and sends the UI task operation steps to the scheduling manager.

[0614] Exemplarily, the filled RPA configuration information is: "order(app=None, shop=C store, product=latte, no=1, size=large cup, temperature=ice, sweetness=None, desLocation=None)".

[0615] Correspondingly, by task orchestration, the UI task operation steps generated are shown in Table 18.

[0616] Table 18

[0617] S52-S53, optionally, the scheduling manager requests the interaction manager for interaction display information. The interaction manager returns the interaction display information to the scheduling manager in response to the request. Exemplarily, the interaction display information includes full-screen window display (or small window display), and light effects displayed around the window, etc.

[0618] S54, the scheduling manager sends the UI task operation steps and the interaction display information to the screen perception module, and applies for execution of the UI agent task.

[0619] S55-S59, the screen perception module obtains an atomic operation sequence according to the UI task operation steps, i.e., each step is arranged in sequence, wherein each step is called an atomic operation. The screen perception module requests the task execution module to execute an atomic operation, and then the task execution module executes the atomic operation and returns the atomic operation result to the screen perception module. Thus, each step is executed in sequence in a loop. After each step is executed in sequence in a loop, the screen perception module obtains the task execution result. Among them, the atomic operation includes detecting the target control and determining the position coordinates of the target control, and performing a simulation operation (such as a simulation click operation or a simulation input operation) on the target control according to the position coordinates of the target control, to achieve the purpose of automatically executing the task.

[0620] Optionally, during the loop execution, the screen perception module updates the window display state according to the interaction display information, such as full-screen window display page and light effects displayed around the full-screen window, to prompt that the current is in the automatic execution state.

[0621] For example, taking ordering takeout as an example, the task execution result can be "selected and purchased, please settle the account".

[0622] For example, taking querying the renewal service as an example, the task execution result can include "querying the automatic renewal service", and can also include "whether to cancel the service".

[0623] S60-S62, in some cases, the screen perception module requests the service result acquisition module to acquire service information. The service result acquisition module acquires the service information according to the task execution result, and returns the service information to the screen perception module.

[0624] For example, taking querying the renewal service as an example, the service information can be "the first application has started the first renewal service".

[0625] S60-S61, the screen perception module sends the task execution result (and the business information) to the system assistant. The system assistant displays the task execution result (and the business information) through a dialogue management unit or the like.

[0626] The system assistant displays the task execution result (and the business information), and the interface schematic diagram provided in each of the above embodiments can be referred to.

[0627] The following describes a possible implementation of the detection of the target control and the determination of the position coordinates of the target control in S55-S59.

[0628] In the embodiments of the present application, the detection and positioning of the target control in each step can be completed based on the following target control detection algorithm (or screen perception algorithm).

[0629] Algorithm 1: Control detection algorithm based on structured information. The target control in the page can be identified and the position coordinates can be determined based on the structured information such as the xtensible markup language (XML) or the document object model (DOM) of the page. The XML structure information of the page can include custom tags and structured data.

[0630] Algorithm 2: Control detection algorithm based on vision. The target control in the page can be identified and the position coordinates can be determined based on the visual features of the page. For example, a page screenshot can be obtained, and then image features can be obtained based on the page screenshot to detect the target control in the page.

[0631] Algorithm 3: Page layout extraction algorithm based on vision. When the control detection algorithm based on structured information and / or the control detection algorithm based on vision cannot detect the target control, the page layout information can be extracted based on the visual features of the page, and the target control in the page can be identified and the position coordinates can be determined according to the page layout information.

[0632] Algorithm 4: Fusion perception algorithm. The target control in the page can be identified and the position coordinates can be determined by combining the control detection algorithm based on structured information, the control detection algorithm based on vision, and / or the page layout extraction algorithm based on vision.

[0633] According to any one of the above target control detection algorithms provided in the present application, the target control can be accurately identified through the scheme of the present application. Moreover, during the automatic execution of the UI task, even in the page without structured information such as XML or DOM, the target control in the page can be accurately detected.

[0634] In some embodiments, the target control is detected by the page XML structure information first, and in the case that the target control is not detected by the page XML structure information, the target control is detected by the page visual features.

[0635] In some other embodiments, the target control is detected by the page visual features first, and in the case that the target control is not detected by the page visual features, the target control is detected by the page XML structure information.

[0636] In some other embodiments, in the case that one or more qualified controls are detected by the page visual features, the target control is determined according to the detection result of the page visual features and the matching result of the page XML structure information. Or, in the case that the target control is not detected by the page visual features and the page XML structure information, the target control is detected by the visual-based page layout extraction algorithm.

[0637] It should be noted that the slot in the RPA configuration information can be preset with an initial value or be empty. In the embodiments of the present application, the slot value in the RPA configuration information can be replaced according to the intent information in actual implementation.

[0638] For example, taking a point coffee as an example, the RPA configuration information can include parameters such as application app, merchant shop, product product, quantity no, cup size, temperature temperature, sweetness sweetness, and delivery location desLocation. Each parameter corresponds to a slot. If the user instruction is "order a cup of hot latte", the slot value of the product in the RPA configuration information is replaced with "latte", the slot value of the quantity no is replaced with 1, and the slot value of the temperature temperature is replaced with "hot". After the slot value is replaced, the RPA configuration information includes: product = latte, no = 1, temperature = hot.

[0639] Among them, the slot value can be text, for example, it is a single text, or it is a text corresponding to a switch button. The slot information can also be a switch button or can be an icon control, or other possible forms.

[0640] It should be noted that the object corresponding to the slot value is the target control in the page. When automatically executing the UI task, the target control needs to be detected in the page and simulated operation on the target control to achieve the purpose of automatic execution.

[0641] The following illustrates three algorithm implementation modes for detecting the target control.

[0642] Scenario 1.1: According to the control and the associated text, the target control is determined.

[0643] In some embodiments, the slot value is text corresponding to the switch button (the text is referred to as associated text), the target control is the switch button, and the automatic execution task is to simulate a sliding operation on the switch button to trigger the switch button to change from the off state to the on state or from the on state to the off state. If multiple switch buttons of the same shape are displayed in the current page, the target control detection algorithm provided in the present application can accurately locate the target control based on the structural features and visual features of the control and the associated text.

[0644] For example, as shown in (a)-(c) of FIG. 13B, multiple setting options, such as flight mode, mobile data, and personal hotspot, are displayed in a mobile network setting page, where each setting option corresponds to a switch button, and each switch button has the same style. The automatic execution task is to turn off the flight mode, and the target control is the switch button corresponding to the flight mode. The flight mode is the associated text of the switch button. There are three switch buttons of the same style in the page, and the three switch buttons correspond to different associated texts. According to the solution provided in the present application, when the automatic execution is performed, all the switch buttons in the page can be recognized according to the visual detection result of the page, and the slot value (for example, the text “flight mode”) is compared with the XML structure information (for example, containing the text “flight mode”) of the current page to recognize the associated text (for example, the text “flight mode”) corresponding to the switch button in the page, so that the “switch button” of the “flight mode” can be accurately recognized.

[0645] In some embodiments, multiple target controls need to be detected in a page, and the multiple target controls are buttons (for example, switch buttons) of the same shape. Each icon corresponds to an associated text. In this case, according to the target control detection algorithm provided in the present application, the multiple target controls can be accurately distinguished based on the structural features and visual features.

[0646] Scenario 1.2: Page layout extraction algorithm based on vision, target control detection through OCR processing

[0647] In some embodiments, multiple words of the same name are displayed in a page, and one or more words of the target control need to be detected. Each icon corresponds to an associated text. In this case, according to the target control detection algorithm provided in the present application, the target control can be detected based on the structural features and visual features of the control and the associated text. If the target control cannot be detected based on the structural features and visual features, the page layout model is used to detect each target control through optical character recognition (OCR) processing.

[0648] For example, as shown in (a)-(c) of FIG. 13C, multiple specification options are displayed in the specification selection page, in which multiple same-named “regular” options exist. For example, the specification corresponds to a “regular” option, the sugar control corresponds to a “regular” option, the taste corresponds to a “regular” option, and the straw corresponds to a “regular” option. The target control is the “regular” option, and the associated text corresponding to the “regular” option includes the specification, the sugar control, the taste, and the straw.

[0649] Through the scheme of the present application, when automatically executed, the target control can be detected based on the page structure features and visual features corresponding to the control and the associated text. If the target control cannot be detected based on the page structure features and visual features, the page layout model is used to perform OCR recognition on the page, so that the target control can be more accurately detected.

[0650] In some embodiments, in a selection page such as a commodity specification selection page, if there is no structural information such as XML or DOM, the target control detection algorithm provided by the present application can be used to perform OCR recognition on the page by using the page layout model, so as to detect the target control in the page.

[0651] Through the scheme of the present application, when detecting the target control in the page, the target control detection algorithm provided by the present application can be used to distinguish buttons with the same style or the same text, so as to ensure accurate identification of the target control.

[0652] Scenario 1.3: Fusion algorithm

[0653] The embodiments of the present application provide three algorithms for detecting the target control by judging the associated text.

[0654] Algorithm 1: In the case where the associated text is a variable parameter, it is further judged whether the associated text matches.

[0655] Suppose that in the RPA configuration information, the associated text of the current step is a variable parameter. For example, when the slot value of the associated text is the text “flight mode”, the target control corresponding to the associated text is a switch button, in which “flight mode” is the associated text of the switch button, and the associated text “flight mode” belongs to a variable parameter.

[0656] In some embodiments, in the case where the associated text is a variable parameter, the XML structure information of the current page can be obtained, and the slot value (for example, the text “flight mode”) is matched with the XML structure information (for example, containing the text “flight mode”), so as to determine the position of the target control corresponding to the slot value in the current page.

[0657] In some embodiments, if the position of the target control cannot be determined according to the XML structure information, the current page can be further subjected to text recognition by using an OCR technique, and the slot value (e.g., the text "flight mode") is matched with the text recognition result (e.g., containing the text "flight mode"), so as to determine the position of the target control in the current page.

[0658] Algorithm 2: In the case where the associated text is determined not to be a variable parameter, XML structure information of the current page is obtained, or the current page is detected based on vision to obtain a visual detection result.

[0659] In some embodiments, in the detection of the target control, in the case where the associated text is determined not to be a variable parameter, the target control can be detected by the page XML structure information, and in the case where the target control is not detected by the page XML structure information, the target control can be detected by the visual detection result.

[0660] For example, assuming that the target control is an option under a certain specification, such as the "hot" option under the temperature specification, the associated text "temperature" of the option is not a variable parameter, and the "hot" option can be detected according to the matching of the XML structure information of the current page and the associated text "temperature", and in the case where the "hot" option is not detected by the XML structure information of the current page, the "hot" option can be detected by the visual detection result.

[0661] In some embodiments, in the detection of the target control, in the case where the associated text is determined not to be a variable parameter, the target control can be first detected by the visual detection result, and in the case where the target control is not detected by the visual detection result, the target control can be detected by the page XML structure information.

[0662] For example, assuming that the target control is a search box, the target text (recommended text content that can appear in the search box) of the search box is not a variable parameter, and the search box can be detected according to the visual features of the current page, and in the case where the search box is not detected by the visual features of the current page, the search box can be detected by the page XML structure information.

[0663] Algorithm 3: In the case where the associated text is determined not to be a variable parameter, the XML structure information of the current page and the visual detection result are obtained, the position coordinates of the target control are determined as coordinate 1 according to the XML structure information, and the position coordinates of the target control are determined as coordinate 2 according to the visual detection result. Then it is determined whether the coordinate 1 and the coordinate 2 overlap.

[0664] In the case where the coordinate 1 and the coordinate 2 overlap, the position coordinates of the target control are determined as the coordinate 1 (i.e., the coordinate 2).

[0665] ​​​​​​​​In the case that the coordinate 1 and the coordinate 2 do not overlap and are close to each other, a center point coordinate between the coordinate 1 and the coordinate 2 is calculated, and the calculated center point coordinate is determined as the position coordinate of the target control.

[0666] In the case that the coordinate 1 and the coordinate 2 do not overlap and are far away from each other, the position coordinate of the target control is re-determined.

[0667] In some embodiments, in a selection page such as a commodity specification page, if there are multiple specifications and options and they are not fully displayed in the current page, the target control detection algorithm provided in the present application can simulate the operation of sliding the screen to display more specifications, and ensure that the specifications and options (i.e. target controls) are found.

[0668] For example, referring to (a)-(c) in FIG. 13D, the content of the specification selection page is more, and multiple specification options have the same name “regular”. Through the scheme of the present application, when automatically executed, the multiple “regular” options can be distinguished, and the multiple specifications can be displayed by simulating the up and down sliding operation, so as to achieve no omission and no redundancy, and accurately select each specification and option.

[0669] In some embodiments, in a selection page such as a commodity specification page, if some options in the to-be-selected specification options are activated, and some options are not activated, the target control detection algorithm provided in the present application judges whether the specification button has been activated. If it has been activated, it will not be repeatedly selected, for example, the “simulated click button special effect” will not be displayed on the activated to-be-selected specification option.

[0670] Again referring to (a) in FIG. 13D, in the specification selection page, cup type, temperature, and sugar content specifications are displayed, and each specification corresponds to multiple options. Among them, the to-be-selected specification options are not activated (i.e. not selected). The automatic execution task is to identify the specifications and options (target controls) that need to be selected in the page and select the target controls. As shown in (b) in FIG. 13D, the cup type, temperature, and sugar content specifications are identified, and the target controls “cup type: large cup”, “temperature: hot”, and “sugar content: less sweet” and options are identified. The “simulated click button special effect” is displayed on each specification option that needs to be selected, for example, the “simulated click button special effect” is a circular cursor. Thus, each specification option is automatically selected.

[0671] For example, Figure 13E shows a schematic diagram of the interface in scenarios where the specification buttons are activated and deactivated. As shown in Figure 13E(a), all the candidate specification buttons on the page are activated; as shown in Figure 13E(b), during automatic execution, the circular click button effect is not displayed on the activated candidate specification options. As shown in Figure 13E(c), some of the candidate specification buttons on the page are activated, while others are deactivated (as shown in the dashed box); as shown in Figure 13E(d), during automatic execution, the circular click button effect is displayed on the deactivated candidate specification options, but not on the activated candidate specification options.

[0672] Execution Scenario 2: User intervention / interaction is required in the automated execution process.

[0673] This solution enables the automatic execution of a sequence of tasks on a user's mobile phone by recognizing the user's intent. Furthermore, it automatically pauses when user interaction is required during execution and resumes execution after the interaction is complete, preventing interruption or termination of the automatic execution process and improving the success rate and stability of task execution.

[0674] In this scenario, during the automatic execution of UI tasks, when encountering situations such as identity verification, requiring user confirmation of order information, or pop-up requests for permissions, the electronic device will enter the user takeover phase of the automatic UI task execution process. After the user completes the interaction, the electronic device can relocate page nodes and plan the optimal path based on the relocated page nodes, allowing the UI task to continue to execute automatically.

[0675] The following section provides a detailed explanation of the automatic execution process of UI tasks in this scenario, combining both the UI interface and algorithm implementation.

[0676] In this application embodiment, there are two scenarios that require user intervention during the automatic execution of UI tasks.

[0677] Scenario 1: The automatic execution sequence configuration includes pre-defined steps / pages that require user intervention.

[0678] During the automatic execution of UI tasks, the system will determine whether there are any steps or pages in the task configuration information that require user intervention.

[0679] In other embodiments, the method of pre-defining pages requiring user intervention includes: placing all Activities of pages requiring user intervention in the auto-execution sequence into an interaction list. When the current page activity is on the interaction list, it is determined that user intervention is required. That is, when a page activity switches to a page activity requiring intervention, it is determined that user intervention is required.

[0680] In some embodiments, the manner of presetting the step requiring user takeover includes marking all steps in the automatic execution sequence that require user takeover, and setting the interactive trigger node by default. When the step is executed, the system automatically identifies that user interaction is required.

[0681] For example, taking the query automatic renewal service as an example, the RPA configuration information can include a step page in the UI task operation step as an identity verification page. Accordingly, when the automatic execution reaches this step, the identity verification page is displayed, and user takeover is required, for example, the user can input fingerprint features or facial features.

[0682] FIG. 14A shows an interface schematic diagram of user takeover when the identity verification page is displayed. As shown in (a) of FIG. 14A, when the task is automatically executed, the identity verification page is displayed in full screen mode, and the user directly performs fingerprint or gesture operation on the identity verification page. As shown in (b) of FIG. 14A, when the task is automatically executed, the display page window can be switched between full screen mode and small window mode in response to user operation. When the identity verification page is displayed in small window mode, the small window mode can be automatically switched to full screen mode, and the user performs fingerprint or gesture operation on the identity verification page in full screen mode.

[0683] After the user inputs the fingerprint features, the electronic device performs identity verification. In the case of successful identity verification, the electronic device continues to automatically execute the UI task. In the case of failed identity verification, the electronic device prompts execution failure and exits the automatic execution of the UI task.

[0684] For another example, taking the opening of the renewal service as an example, the RPA configuration information can include a step page in the UI task operation step as a password input page. Accordingly, when the automatic execution reaches this step, the password input page is displayed, and user takeover is required. In the case that the user inputs the password and the password verification is successful, the electronic device continues to automatically execute the UI task. In the case that the password verification fails, the electronic device prompts execution failure and exits the automatic execution of the UI task.

[0685] Scenario 2: Displaying a dynamic pop-up window and requiring user interaction

[0686] In some embodiments, during the automatic execution of the UI task, a pop-up window appears in the page, requesting user authorization, at which time user interaction is required, and the electronic device suspends the execution of the task and enters the user takeover process.

[0687] For example, FIG. 14B shows an interface schematic diagram in which a pop-up window appears in the page and the user needs to take over / interact. As shown in FIG. 14B, a pop-up window appears in the page for scanning a code, requesting the user to allow the scanning code application to use the camera permission, at which time the user needs to interact. In the case where the user confirms the authorization, the electronic device continues to automatically perform the UI task. In the case where the user does not agree to the authorization, the electronic device prompts that the execution fails and exits the automatic execution of the UI task.

[0688] In some other embodiments, during the automatic execution of the UI task, a pop-up window appears in the page, requesting the user to confirm whether to agree to a certain agreement, at which time the user needs to interact, the electronic device suspends the execution of the task and enters the user takeover process. In the case where the user confirms to agree to the agreement, the electronic device continues to automatically perform the UI task. In the case where the user does not agree to the agreement, the electronic device prompts that the execution fails and exits the automatic execution of the UI task.

[0689] Suspension and resumption in the automatic execution process

[0690] In the implementation process, during the automatic execution process, some situations that require human intervention to ensure that the process can continue may be encountered, such as: human biometric recognition; single or multiple selection of specified content; scenarios involving money and important choices; pop-up windows that require user interaction, etc. For the above situations, the present application provides suspension and resumption functions to ensure stable and reliable automatic execution.

[0691] In the embodiments of the present application, during the process in which the electronic device responds to the user operation and executes the automatic query process, when the page automatically jumps, pages such as identity verification or requiring user confirmation appear, in which case the user needs to manually operate, at which time the page is suspended to receive user input, and after the electronic device receives the user input, the electronic device continues to execute the automatic jumping of the page. In the suspended state, the page enters an unlocked state.

[0692] The following takes the user interaction interface agent (UIAgent) scenario "WeChat sends a file" as an example for illustration.

[0693] Since most users are difficult to accurately say the name of the file they need to send at one time, and many times only have a vague impression of the file name, this leads to the fact that multiple matching files are likely to be encountered when searching for the file in the file management. At this time, the user may have several situations:

[0694] Different user operation habits are different, some users are used to selecting files and then clicking the "continue execution" button, and some users will actively click the share button after selecting the files and then click the "continue execution" button. The user free operation scene has multiple possibilities. At this time, the technical ability of pausing and resuming needs to know the step of the execution state from pausing to resuming to the step, whether there is a situation of deviating from the preset step, such as the user returning to the desktop, or answering a phone call, etc.

[0695] The embodiment of the application provides the following automatic execution capabilities:

[0696] (1) UIAgent RPA execution step page feature recognition capability.

[0697] Through this capability, the current page can be uniquely identified as the state of the first step in the execution sequence. The feature points of the page are recorded for identification, and minor changes on the page do not affect the overall judgment strategy.

[0698] (2) UIAgent RPA execution step redirection capability.

[0699] Through this capability, the first step operation that needs to be executed in the current display state can be accurately found. By combining the page feature recognition capability, the original execution intention can be continued to execute, the success rate is greatly increased, and the user experience is good.

[0700] (3) UIAgent RPA execution step offset detection capability.

[0701] Through this capability, it can be quickly found whether the current page state has deviated from the preset execution sequence. It can be observed and confirmed through engineering and algorithmic judgment method whether the execution of the current task exists deviation, and the success rate of task execution is greatly improved.

[0702] Referring to FIG. 7C, when the page automatically jumps, there are multiple search results or multiple names, the user needs to confirm, in this case, the user needs to manually operate, at this time, the page is paused to receive user input, after the electronic device receives the user input, the electronic device continues to execute the page automatic jump. The electronic device can adopt the features of the execution step page, and the execution step redirection, and the execution step offset detection, so that it can be flexibly automatically executed and not affected by user operation.

[0703] By the scheme, the current page can be uniquely identified as the state of the step in the execution sequence, the page feature points are recorded for identification, and the step to be executed in the current display state can be accurately found. By combining the page feature identification capability, the original execution intention can be quickly continued to execute, and it can be quickly found whether the current page state has deviated from the preset execution sequence. The judgment method of engineering and algorithm is used to observe and confirm whether the execution of the current task deviates in multiple dimensions, and the success rate of task execution is greatly improved.

[0704] Therefore, no matter whether the user selects the file and then clicks the continue execution button, or clicks the sharing button after selecting the file and then clicks the continue execution button, or jumps to other page nodes, the electronic device can continue to normally and automatically execute the process.

[0705] The processing mode and specific algorithm implementation process of the user takeover process in the automatic execution process are described below.

[0706] The UI task automatic execution method provided by the embodiment of the application is a user interaction supporting automatic execution method. When user interaction is required during task execution, the system can automatically pause, remind the user to interact, and after the user completes the interaction, reposition the current page node and continue to execute the remaining task operation sequence.

[0707] FIG. 14C shows a pause and resume process schematic diagram in the automatic execution process.

[0708] S1401, automatically execute the task operation sequence.

[0709] The electronic device can automatically execute the task operation sequence to complete a series of UI tasks by simulating user operations. The task operation sequence can be automatically predetermined by the system.

[0710] S1402, determine whether user interaction / takeover is required.

[0711] In each step of the task execution, the system detects in real time whether the current operation step requires user interaction. According to different detection methods (such as a preset interaction trigger node, Activity switching detection, and visual anomaly detection), it can be determined whether to pause the execution of the task. The specific detection method will be described below. In the case where it is determined that no user interaction / takeover is required, the task is automatically executed.

[0712] S1403, pause the execution of the task in the case where it is determined that user interaction / takeover is required.

[0713] S1404, prompt the user to perform the interaction operation.

[0714] If it is detected that user interaction is needed, the system will automatically pause the execution of the current task and remind the user to perform the necessary interaction operation through a prompt (such as a pop-up window, vibration, sound, etc.).

[0715] For example, the way to prompt the user to perform the interaction operation can include: prompting the user to perform the interaction operation by updating the prompt information to "please manually operate"; and can also guide the user to perform the interaction operation by displaying a hand operation animation.

[0716] S1405, determining whether the user interaction is completed.

[0717] The system continuously monitors the operation state of the user during the user interaction process to determine whether the user interaction has been completed. This module can monitor the completion state of the interaction in various ways (such as screen hot area monitoring, Activity switching detection, user active triggering, etc.), and the specific monitoring methods will be described below. In the case where it is determined that the user interaction is not completed, the user interaction state is continuously monitored until it is determined that the user interaction is completed.

[0718] S1406, in the case where it is determined that the user interaction is completed, positioning the current page node and planning an execution path.

[0719] When the user interaction is completed, the system will reposition the current page in the UI task operation sequence graph and plan the optimal execution path according to the graph to continue executing the remaining task operation sequence. The specific methods of positioning the page and planning the path will be described below.

[0720] S1407, continuing to automatically execute the task.

[0721] The system returns to the page node in the task operation sequence according to the positioned current page node and the planned optimal path, and continues to execute the remaining tasks until the entire task operation sequence is completed.

[0722] S1408, determining whether the task is completed.

[0723] In the case where it is determined that the task is completed, the process is ended. In the case where it is determined that the task is not completed, the automatic execution of the task operation sequence is continued.

[0724] The following describes the following three steps in the user takeover process in detail: 1) determining whether user takeover is needed, 2) determining whether the user interaction is completed, and 3) repositioning the current page and planning a path to continue executing the remaining task operation sequence.

[0725] 1) determining whether user takeover is needed

[0726] The embodiments of the present application provide the following three ways to determine whether user takeover is needed:

[0727] Method one: preset interaction trigger node. A step requiring user interaction is preset in the task operation sequence configuration, and the system automatically identifies that user interaction is required when the step is executed.

[0728] Method two: based on page Activity switching detection. Monitor whether the current page Activity belongs to the list of Activities requiring user interaction. The current page Activity is denoted as A c , and the list of Activities requiring user interaction is denoted as {A1, A2, …, A n}. When the condition A c ∈ {A1, A2, …, A n} is met, it is considered that user interaction is required.

[0729] Method three: based on visual anomaly detection. Based on visual detection, it is detected whether a pop-up window or an abnormal UI element requiring user interaction appears on the interface. If a pop-up window or an abnormal UI element requiring user interaction is detected, it is considered that user interaction is required.

[0730] It should be noted that at least one of the above three methods can be selected to determine whether user takeover is required, or any combination of the three methods can be used to determine whether user takeover is required.

[0731] FIG. 14D shows a flowchart for determining whether user takeover is required.

[0732] As shown in FIG. 14D, it is first determined whether the current step page is a preset interaction trigger node. In the case where the current step page is not a preset interaction trigger node, it is further determined whether the current page Activity is in the interaction list. In the case where it is determined that the current page Activity is not in the interaction list, it is further determined whether a pop-up window requiring user takeover is detected. In the case where no pop-up window requiring user takeover is detected, it is determined that no user interaction is required.

[0733] As shown in FIG. 14D, in one case, if it is determined that the current step page is a preset interaction trigger node, it is determined that user interaction is required. In another case, if it is determined that the current page Activity is in the interaction list, it is determined that user interaction is required. In another case, if a pop-up window requiring user takeover is detected, it is determined that user interaction is required.

[0734] In the automatic execution process, it is determined whether the current step v c requires user interaction. If user interaction is detected, the user is reminded to take over through the interface, and it is further determined whether the user interaction is completed.

[0735] 2) Determine whether the user interaction is completed

[0736] The embodiments of the present application provide the following three ways to determine whether the user interaction is completed:

[0737] The first way is screen hot area monitoring. The coordinates (x, y) of the user screen touch event are monitored to determine whether they fall within the set screen coordinate hot area Ω. If the condition is met, it is considered that the user interaction is completed.

[0738] The second way is based on page Activity switching detection. The current page Activity (denoted as A c ) is monitored to determine whether it switches back to the original task Activity (denoted as A orig ). If the condition is met, it is considered that the user interaction is completed.

[0739] The third way is user active triggering. The user clicks the "continue execution" button. If the system receives the "continue execution" signal (denoted as S c ), it is considered that the interaction is completed.

[0740] It should be noted that at least one of the above three ways can be selected to determine whether the user interaction is completed, or any combination of the three ways can be used to determine whether the user interaction is completed.

[0741] FIG. 14E shows a flowchart for determining whether the user interaction is completed.

[0742] As shown in FIG. 14E, it is first determined whether a touch event falls into a screen hot area. In the case where no touch event falls into the screen hot area, it is determined whether the current Activity switches to the original Activity. In the case where it is determined that the current Activity does not switch to the original Activity, it is determined whether the "continue button" is clicked. In the case where it is determined that the "continue button" is not clicked, the detection continues.

[0743] As shown in FIG. 14E, if it is detected that the touch event falls into the screen hot area, it is determined that the user interaction is completed. If it is detected that the current Activity switches to the original Activity, it is determined that the user interaction is completed. If it is detected whether the "continue button" is clicked, it is determined that the user interaction is completed.

[0744] During the user interaction, the electronic device continuously monitors the operation state of the user to determine whether the user interaction has been completed, so as to continue to execute the task sequence. If it is detected that the user interaction is completed, the page node relocation and task execution recovery module is notified to continue to execute the task.

[0745] FIG. 14F shows three interface schematic diagrams for determining whether the user interaction is completed through screen hot area monitoring. As shown in (a) of FIG. 14F, in the automatic execution process of sending a file, after the file and the contact are automatically selected, the user takes over the process, the user is prompted to manually share through the interface (prompt 1), the user is guided to operate through the hand animation (prompt 2), and the prompt information is updated to “automatic execution has ended, please manually share”, prompting the user to take over. The location of the “share” control is the screen hot area. After the user clicks the “share” control, the electronic device monitors that the touch event falls into the screen hot area, so it can be determined that the user interaction has been completed.

[0746] As shown in (b) of FIG. 14F, in the automatic execution process of sending a file, when multiple files are automatically searched, the user is prompted to select a file, and the user takes over the process. The area where the “continue execution” button is located is the screen hot area. After the user clicks the “continue execution” button, the electronic device monitors that the touch event falls into the screen hot area, so it can be determined that the user interaction has been completed.

[0747] As shown in (c) of FIG. 14F, in the automatic execution process of sending a photo, when the user needs to select a photo, the user is prompted to manually operate, and the user takes over the process. The area where the “continue execution” button is located is the screen hot area. After the user clicks the “continue execution” button, the electronic device monitors that the touch event falls into the screen hot area, so it can be determined that the user interaction has been completed.

[0748] In the automatic execution process, after detecting that the user interaction is needed, the user is reminded to take over through the interface, and it is further determined whether the user interaction is completed. After detecting that the user completes the operation, the current page is further repositioned and the path is planned, and the remaining steps are continued to be executed.

[0749] 3) Reposition the current page and plan the path

[0750] First, the graph structure of the UI task operation step is explained. FIG. 14G shows a structure schematic diagram of a UI task operation step graph, which is used to represent the page state and the bidirectional conversion relationship. In the graph, G=(V, E) is used to represent the graph, where V is a set of page nodes, representing the page state; E is a set of edges, representing the bidirectional conversion relationship between pages.

[0751] In the graph, a plurality of page nodes are included, and the state of each page node can be represented by a feature vector, for example, the feature vector of node v i is F i . The feature vector set of all page nodes can be represented as {F1, F2, …, F m}.

[0752] In the graph, a plurality of page nodes are included, and the state of each page node can be represented by a feature vector, for example, the feature vector of node v i∈V represents a page in the UI operation steps. (v) i v j )∈E means from page v i Go to page v j The transformation, edge (v) j v i )∈E means from page v j Go to page v i The conversion.

[0753] It should be noted that feature similarity comparison can be performed using a graph of UI task operation steps, and the optimal execution path can be planned based on the graph of UI task operation steps, which will be explained in detail below.

[0754] Figure 14H illustrates a flowchart of relocating the current page and planning the path. As shown in Figure 14H, embodiments of this application provide the following two methods for relocating the current page:

[0755] Method 1: Similarity matching based on page features.

[0756] First, extract the feature vector F of the current page. c And obtain the feature vector set {F1, F2, ..., F} of each page node in the graph. m Then, the features of the current page are compared with the feature set in the graph for similarity. The current page is then located to the page node with the highest similarity. This repositions the current page within the UI task operation step graph, identifies the optimal execution path, and continues the task sequence.

[0757] Optionally, the extraction methods include extracting page structural features and extracting page visual features. Extracting page structural features refers to extracting page node features based on the XML or DOM structural information of the current page. Extracting page visual features refers to extracting page visual features from a screenshot of the current page.

[0758] Method 2: Matching based on unique page elements.

[0759] First, define a unique set of page elements {E1, E2, ..., E...} for each page. m} and determine whether a unique page element belongs to the current page's set of interface elements E based on XML nodes or visual features. c If E i ∈E c If the i-th page node is located, then the system continues to execute the remaining task sequence according to the optimal path in the UI task operation step graph.

[0760] It should be noted that either of the above two ways can be selected to relocate the current page and plan the path, or both ways can be used to relocate the current page and plan the path.

[0761] The application scheme supports user interaction and task continuation. That is, the technical framework proposed in the application can detect the timing of user interaction in the automatic execution process, automatically pause task execution, and detect the timing of user completing the interaction to continue executing the task. In addition, the application proposes a method based on page feature similarity matching and unique page element matching to realize page node relocation after user interaction is completed. The system can relocate and continue to execute the remaining tasks according to the current page features, improving the efficiency and accuracy of task execution.

[0762] In some cases, after the user interaction operation is completed, the page may jump or change compared to before the user interaction operation, and the current page does not belong to the original operation path (the first operation sequence). In this case, the task graph can be used to relocate the page node and plan the optimal path based on the relocated page node to continue executing the task according to the re-planned path. The second operation sequence is different from the first operation sequence.

[0763] In some other cases, after the user interaction operation is completed, the page may jump or change compared to before the user interaction operation, and the current page belongs to the page node of the original operation path (the first operation sequence). In this case, the task graph can also be used to relocate the page node and plan the optimal path based on the relocated page node to continue executing the task according to the re-planned path, avoiding interruption or termination of the automatic execution process. The second operation sequence is different from the first operation sequence.

[0764] In some other cases, after the user interaction operation is completed, the page may not jump or change compared to before the user interaction operation, and the current page belongs to the original operation path (the first operation sequence). In this case, the task graph can be used to relocate the page node and plan the optimal path based on the relocated page node to obtain the second operation sequence, and the second operation sequence is the same as the first operation sequence.

[0765] In some other cases, after the user interaction operation is completed, if it is detected that the page has not jumped or changed compared to before the user interaction operation, no re-planning is performed, and the task is continued to be executed according to the original operation path (the first operation sequence).

[0766] The relocation of the current page and the planning of the path will be described below with reference to the accompanying drawings. FIG. 141 shows a schematic diagram of relocating the current page and planning the path based on the task graph. It is assumed that the task graph includes v1, v2, v 21 , v 22There are six page nodes for v3 and v4.

[0767] As shown in Figure 14I(a), the first operation sequence is v1, v2, v3, and v4. The automatic execution flow starts from the v1 page node.

[0768] As shown in Figure 14I(b), in the automatic execution process, the automatic execution process is paused at the v2 page node because it is detected that the user needs to take over, and the user interaction operation is detected.

[0769] As shown in Figure 14I(c), after the user interaction is completed, the page changes, from page node v2 to v 22 Page Nodes. This application's solution uses page node positioning to determine the current page node's position within the task graph. 22 .

[0770] As shown in (d) of Figure 14I, determine the node position v of the current page node in the task graph. 22 Then, based on the current page node's position v in the task graph... 22 The execution route was replanned, resulting in the second operation sequence: v 22 v2, v3 and v4.

[0771] The solution proposed in this application can accurately detect when user interaction is required during the automated execution of UI tasks, prompting the user to take over and seamlessly continuing task execution after the user completes the interaction, instead of requiring the task to be executed from the beginning. Furthermore, it can reposition the current page within the UI task operation step graph and find the optimal execution path to continue the task sequence.

[0772] The method provided in this application embodiment can respond to user instructions and automatically execute tasks according to a first operation sequence based on user commands (e.g., querying automatic renewal, sending a file to someone, disabling application permissions, optimizing overall device performance, disabling application notifications, ordering takeout). During the automatic task execution, if situations such as identity verification, requiring user confirmation of order information, or pop-ups are encountered, automatic execution can be paused, and a user-taken-over process can begin. After the user completes the interaction, the page node can be relocated, and an optimal path can be planned based on the relocated page node. The task can then continue execution along the replanned path, avoiding interruption or termination of the automatic execution process and improving the success rate and stability of task execution. This solution helps users automatically execute tasks without manual operation; users only need to input commands, improving user experience and operational efficiency.

[0773] Execution Scenario 3: A pop-up window appears during the automatic execution process.

[0774] During the automatic execution of the UI task, a pop-up window may appear in the current page. The pop-up window may be a normal type of user interaction pop-up window or a non-normal type of pop-up window, such as an advertisement pop-up window. When the target control is blocked by the advertisement pop-up window, the target control cannot be simulated to be clicked, which may cause the automatic execution process of the UI task to be temporarily interrupted.

[0775] In other words, when the electronic device executes the pre-set RPA automation process, various possible pop-up windows may be encountered, such as an advertisement pop-up window, a message pop-up window, a permission pop-up window, a privacy pop-up window, and the like. The abnormal situations may cause the target control of the current step to be unable to be found, the simulated interaction of the target control to be unable to be implemented, and the RPA automation process to be unable to be completed.

[0776] To this end, the solution provided in this application is to identify whether a window appearing in a page is an advertisement pop-up window. When it is identified that the pop-up window is an advertisement pop-up window, the closing control of the advertisement pop-up window can be accurately found, and the advertisement pop-up window can be automatically closed by simulating to click the closing control, so that the UI task can continue to be automatically executed without the need for user operation. When it is identified that the pop-up window is a pop-up window that needs to be taken over by the user, the user interaction can be prompted, and the user interaction operation can be detected. After the user interaction operation is completed, the process is continued. Through the solution, the user experience can be improved.

[0777] In the embodiments of this application, during the execution of the RPA automation process, the electronic device can use a pop-up window detection algorithm / model to detect. If an advertisement pop-up window is detected, the closing button of the advertisement pop-up window is identified by the algorithm, and the position information (such as coordinates) of the closing button is determined. Then, based on the position information of the closing button, an automatic simulation clicking operation is performed on the closing button to automatically close the advertisement pop-up window without the need for user operation, thereby improving the success rate of the execution of the RPA automation process.

[0778] In some other embodiments, during the execution of the RPA automation process, when the electronic device detects that a pop-up window that needs to be taken over by the user appears in the page, the electronic device can control the automatic execution mode to be adjusted to the user takeover mode, and the user can be prompted to select whether to close the pop-up window, so as to avoid the risk of mis-clicking. After detecting that the user interaction operation is completed, the electronic device can control the user takeover mode to be restored to the automatic execution mode to continue to automatically execute the process.

[0779] The embodiments of this application can determine whether there is an advertisement pop-up window that needs to be closed and a pop-up window that needs user interaction based on the visual features of the page. The automatic execution process of the UI task in this scenario is described in detail from the aspects of the UI interface and the algorithm implementation.

[0780] FIG. 15A shows an interface schematic diagram of identifying an advertisement pop-up window and automatically closing the advertisement pop-up window in the automatic execution process.

[0781] As shown in (a) of FIG. 15A, the electronic device detects that a pop-up window appears in the current page in the automatic execution process.

[0782] As shown in (b) of FIG. 15A, the electronic device identifies the pop-up window as an advertisement pop-up window through a pop-up window detection algorithm, and then locates to the closing button of the advertisement pop-up window, simulates a click operation on the closing button, and displays a simulated click cursor.

[0783] As shown in (c) of FIG. 15A, after closing the advertisement pop-up window, the electronic device continues to automatically execute the process, for example, detects the target control "design psychology" in the current page, simulates a click operation on the target control "design psychology", and displays a simulated click cursor.

[0784] Through the scheme, for a closable non-normal type of pop-up window (for example, an advertisement pop-up window), the electronic device can accurately identify the type of the pop-up window, identify the type of the closing button of the pop-up window, accurately determine the position coordinates of the closing button, automatically close the pop-up window based on the position coordinates of the closing button, and ensure that the UI task can continue to be automatically executed.

[0785] FIG. 15B shows a schematic diagram of the overall scheme framework for identifying a pop-up window in an automatic execution process.

[0786] As shown in FIG. 15B, in the embodiments of the present application, the overall scheme framework for automatically detecting and processing a pop-up window includes the following three stages: Activity filtering processing, processing through a visual detection model, and traversal detection based on a pop-up window type and a closing identification type.

[0787] Through Activity filtering processing, it can be identified whether the pop-up window is an advertisement pop-up window. Based on the visual detection model and the traversal detection, it can be determined whether to close the advertisement pop-up window, or to prompt the user to take over, or to click the target control. Through the pop-up window processing scheme, the pop-up window appearing in the automatic execution process can be accurately identified and processed, the automatic execution process is not interrupted, and the user experience is improved.

[0788] The possible implementation ways of the pop-up window processing method in the automatic execution task process are described below.

[0789] First stage: Activity filtering

[0790] Activity filtering includes: obtaining a pop-up window activity, and judging whether the pop-up window activity is in a system pop-up window activity whitelist.

[0791] If the pop-up window activity is in the system pop-up window activity whitelist, it can be determined that it is a system pop-up window, and the system pop-up window is a non-closable pop-up window.

[0792] Exemplarily, the system pop-up window includes a permission request pop-up window, a password input pop-up window, or an information confirmation pop-up window, etc.

[0793] If the pop-up window activity is not in the system pop-up window activity whitelist, it can be determined that it is not a system pop-up window, that is, the pop-up window is an advertisement pop-up window, and further the type of the pop-up window and the processing manner are determined through visual detection model and traversal detection.

[0794] Second stage: visual detection model

[0795] In the case of determining that it is an advertisement pop-up window, a screenshot of the pop-up window is obtained, and then the screenshot is input into the visual detection model. The embodiment of the application provides a visual detection model, and the input parameter is a screenshot, and the output parameter is information such as the type of the pop-up window and the closing identifier type.

[0796] Exemplarily, in the embodiment of the application, through the visual detection model, the opening screen page, the advertisement pop-up window, the multiple closable pop-up window, the closing button, the closing icon, and the non-closable pop-up window can be detected.

[0797] The model training and model inference of the visual detection model are described below.

[0798] First, the training process of the visual detection model provided by the embodiment of the application is described. The training process includes data collection, data labeling, data enhancement, and model training.

[0799] 1) Data collection: the user's commonly used application pages (for example, the pages of the top 100 applications) are obtained, and the user's commonly used application pages are combined with public data sets to obtain sample data.

[0800] 2) Data labeling: the types of the pop-up window and the closing button are summarized, and the different types of the pop-up window and the different types of the closing button in the sample data set are labeled.

[0801] In some embodiments, the type of the pop-up window includes an advertisement pop-up window, a closable pop-up window, and a non-closable pop-up window.

[0802] For example, the advertisement pop-up window is a central pop-up window advertisement or a small advertisement blocking.

[0803] For example, the closable pop-up window is a permission notification pop-up window or a function update pop-up window.

[0804] For example, the non-closable pop-up window is a privacy agreement pop-up window or a verification code pop-up window.

[0805] In some embodiments, the type of the closing button includes a closing icon button and a closing text button.

[0806] For example, the close icon button is an icon button in the form of a small cross "X". For example, the close text button can contain the text "Skip".

[0807] Table 19 shows various popup types and example scenarios for each popup type.

[0808] Table 19

[0809] The following Table 20 shows common close option types.

[0810] Table 20

[0811] 3) Data augmentation: based on real background cropping splicing, and based on data type paste method and random visual transformation, the sample data is enhanced.

[0812] Exemplarily, the data augmentation process includes cropping, close text paste method, popup paste method and close button (small cross) paste method.

[0813] In the cropping, the target region is cropped.

[0814] In the close text paste method, the aspect ratio is maintained, the edge is expanded, the text is pasted in the center, and random optical variable parameters are used. The aspect ratio is maintained to ensure that the aspect ratio of the text region is not changed during the pasting process to avoid distortion. The edge expansion means that the median of the color channel around the target region is calculated, and the edge of the target region is filled with the color to ensure a natural transition in vision. The center pasting means that the adjusted text region is pasted in the center of the target position to ensure aesthetics and consistency. The random optical variable parameters include color saturation, brightness, and channel randomness.

[0815] In the popup paste method, the aspect ratio is maintained, the edge is expanded, random optical variable parameters are used, and random visual transformation is performed. The aspect ratio can be maintained by deformation filling to ensure that the aspect ratio of the text region is not changed during the pasting process. The edge expansion means that the median color around the target region is calculated, and the edge of the target region is filled with the color to ensure a natural transition in vision. The random optical variable parameters include color saturation, brightness, and channel randomness. The random visual transformation can include rotation 180, or left and right and up and down flipping.

[0816] In the close button (small cross) paste method, random visual transformation can be added, such as adding a random amount of 90-degree rotation compared to the popup paste.

[0817] 4) Model training: YOLO can be used for stage training method. First, pre-training on the simulation sample data set, then focus on the accuracy and recall rate of different categories, supplement the corresponding real sample data (enhanced sample data) as the second stage sample data set, and then perform two-stage training on the second stage sample data set to obtain a visual detection model.

[0818] In the embodiments of the present application, by using the visual detection model trained above for model inference, the pop-up window type and the close button of the pop-up window can be accurately detected, and the accuracy of the automatic execution of the UI task is improved.

[0819] Among them, the pop-up window screenshot is input into the visual detection model and subjected to model inference.

[0820] First, the visual detection model can collect a preset number of prediction boxes based on the screenshot, and the prediction boxes contain the characteristics of the advertisement pop-up window and / or contain the closing mark. Each prediction box can partially overlap.

[0821] Then, the visual detection model can obtain the feature vectors of each prediction box, such as 6 feature vectors: the position coordinates of the prediction box (containing four coordinate values), the prediction box type, and the prediction box confidence.

[0822] Among them, the prediction box type can be an advertisement pop-up window type or a closing mark type.

[0823] Among them, the prediction box confidence can be a normalized value between 0 and 1. The larger the value, the higher the confidence.

[0824] Then, the visual detection model can output information such as pop-up window type and closing mark type. The information can be in the form of an array.

[0825] For example, the visual detection model outputs a 6*300 array, that is, the output result includes 300 prediction boxes, and the feature vector of each prediction box corresponds to 6 feature vectors.

[0826] It should be noted that the 6*300 array can represent information such as pop-up window type and closing mark type.

[0827] Third stage: traversal detection

[0828] In addition to system pop-up windows, there are various possible pop-up windows, such as opening screen pages, advertisement pop-up windows, and various pop-up windows may also have different situations such as closable pop-up windows and non-closable pop-up windows.

[0829] And the shapes, positions, and quantities of the closing marks of different pop-up windows are different.

[0830] For example, some closing marks are "X", some closing marks are the word "skip", and some have no closing marks.

[0831] For example, the closing mark of some pop-up windows is inside the pop-up window, and the closing mark of some pop-up windows is outside the pop-up window.

[0832] For another example, some pop-up windows have one closing mark, and some pop-up windows have two closing marks.

[0833] Therefore, after the pop-up window type and the closing mark type and other information are output by the pop-up window detection model, it is necessary to further process based on the pop-up window type and the closing mark type to determine the pop-up window processing mode.

[0834] In the embodiments of the present application, the traversal detection can include traversal detection based on the pop-up window type and the closing mark type, and then determining to close the advertisement pop-up window or prompting the user to take over or clicking the target control according to the traversal detection result.

[0835] The traversal detection rule in the embodiments of the present application is described below. FIG. 15C shows a flowchart of traversal processing.

[0836] 1) Input parameters and perform filtering. The input parameters are the prediction results output by the visual detection model. Through threshold filtering, the prediction boxes meeting the conditions are selected.

[0837] 2) Traversal detection and processing.

[0838] Firstly, it is determined whether there are two exits (for example, exit button ×) in the prediction result. If there are two exits, absolute filtering is performed to exclude the exit (for example, ×) in the high-frequency false point area. If there are no two exits (for example, ×), no filtering is needed, and n prediction boxes are obtained.

[0839] Then, it is determined whether there are closable pop-up windows or advertisement pop-up windows in the prediction boxes meeting the conditions. If there are closable pop-up windows or advertisement pop-up windows, traversal processing is performed from high to low according to the confidence. For example, the exit button × or other closing button inside or outside the closable pop-up window or the advertisement pop-up window with the highest recognition confidence is identified. The processing priority is arranged from high to low, for example: the exit button × inside the pop-up window, the exit button × outside the pop-up window, other closing buttons inside the pop-up window, and other closing buttons outside the pop-up window.

[0840] If there are no closable pop-up windows or advertisement pop-up windows, it is determined whether there are splash pages in the prediction boxes meeting the conditions.

[0841] If there are splash pages in the prediction boxes meeting the conditions, the “skip item” is traversed from high to low according to the confidence.

[0842] If there is no opening page in the prediction box meeting the conditions, it is determined whether there is a "non-closable popup" in the prediction box meeting the conditions. If there is a "non-closable popup", the user needs to take over, and if there is no "non-closable popup", no exception is detected, and the task is automatically executed.

[0843] Then, according to the processing result, the center point coordinates of the current exit (for example, X) or other close button are output. If no exception is detected, it is reported that no exception is detected.

[0844] 3) Screening detection. The RPA target control is matched with the center point coordinates to obtain an output suggestion.

[0845] In the case where the visual detection model outputs the coordinates of the close button and suggests closing the popup window, the electronic device determines whether the target control and the "close button" belong to a closable popup window or an advertisement popup window.

[0846] If the coordinates of the RPA target control match the center point coordinates of the close button, it indicates that the target control and the "close button" belong to a closable popup window or an advertisement popup window, the close suggestion is ignored, the target control is normally clicked, and the RPA process is executed.

[0847] If the RPA target control does not match the center point coordinates, it indicates that the target control and the "close button" do not belong to a closable popup window or an advertisement popup window, and the close button of the popup window is simulated.

[0848] Through automatic case testing, the popup window is detected and processed, and during the automatic case running process, the popup window can be automatically detected and processed, thereby improving the automation success rate.

[0849] Scenario 4: Target control not found in automatic execution process

[0850] In this scenario, during the automatic execution of the UI task, the electronic device needs to search and identify the target control in the current page, and then simulate clicking the target control to complete the automatic execution task in the current page. In actual implementation, different pages have different specifications, and different controls in the page have different display modes, so it is possible that the target control cannot be found during the search, which may cause the UI task automatic execution process to be temporarily interrupted.

[0851] To this end, the solution provided by the present application includes:

[0852] 1) In the case where the target control is not found, a specific algorithm process is used to search for the target control again, for example, scrolling down the page to search for the target control in the updated page;

[0853] 2) or, simulate a rightward swipe operation or a leftward swipe operation in a region including multiple controls, so that more controls are displayed;

[0854] 3) or, if the target control is not found, a "show all" button can also be simulated to trigger the display of a secondary menu page, and then the target control is searched in the secondary menu page.

[0855] In this way, the probability of searching for the target control can be improved, the UI task can continue to be automatically executed, and the user experience is improved.

[0856] The UI task automatic execution process of this scenario is described in detail below from the aspects of UI interface and algorithm implementation.

[0857] FIG. 16A shows an interface schematic diagram of searching for a target control again after page scrolling display. As shown in (a) of FIG. 16A, in the automatic execution process, the target control "browser" is searched for in the current page. As shown in (b) of FIG. 16A, after the current page is loaded, more content is displayed by scrolling down. As shown in (b) of FIG. 16A, the target control is searched again during the page scrolling display. In this way, the target control recognition success rate and stability are improved by retrying through page scrolling display, and the automatic execution process can be smoothly performed.

[0858] FIG. 16B shows an interface schematic diagram of simulating leftward and rightward swipe operations in a region including multiple icons to search for a target control. As shown in (a) to (d) of FIG. 16B, in the file sending scenario, the target control "App1 icon" needs to be automatically selected in the sharing page. Since the App icon selection region includes multiple App icons, some of which are visible to the user and some of which are invisible to the user. In the case where the target control is not searched for, a rightward swipe operation can be simulated in the region, so that more icons are displayed, and the target control is further searched for. If the target control is still not searched for, the "show all" control in the region is simulated to be clicked, so that all icons are displayed, and the target control is further searched for.

[0859] In some embodiments, during the automatic execution of the UI task, when the current step is executed, the target control is searched for in the page of the current step, and if the target control is not searched for, the counter is incremented by 1. Then, in the case where the counter value is less than a preset threshold, a swipe operation or page scrolling display is simulated in the page to search for the target control again. In the case where the counter value is greater than or equal to the preset threshold, the search for the target control fails, and an execution exception event is reported. The specific implementation process can be referred to FIG. 12C.

[0860] The preset threshold can be set according to actual use requirements, for example, the preset threshold can be set to 3.

[0861] The solution of this application can more accurately search for the target control by simulating a sliding / rolling operation or triggering a secondary menu page. On the interactive interface, it is manifested as sliding left / right in the application icon selection area, or scrolling up and down for display, which improves the success rate and stability of task execution.

[0862] The embodiment of this application provides a task agent model with task planning capabilities, which can convert user instructions into task orchestration results based on the DSL format. After the electronic device receives a user instruction, the electronic device can identify the user's intention according to the user instruction, obtain intention information, then obtain the task orchestration result according to the intention information, and then automatically execute the task according to the task orchestration result.

[0863] Exemplarily, as shown in FIG. 17A, the electronic device can input the user instruction into a pre-trained task agent model, perform model inference through the task agent model, output a task orchestration result (abbreviated as DSL data) based on the domain-specific language (DSL) format, and then parse the DSL data and execute the task. Among them, the DSL description language is a programming language designed specifically for a certain specific domain.

[0864] For example, taking the task of ordering takeout (ordering drinks) as an example, the user instruction is "Help me order a large iced latte from store C". After the electronic device receives the user instruction, it can input the user instruction into the task agent model, perform model inference through the task agent model, and output DSL data. Exemplarily, the DSL data can be:

[0865] [['1', 'order(app = None, shop = store C, product = latte, no = 1, size = large cup, temperature = iced, sweetness = None, desLocation = None)']].

[0866] Alternatively, the DSL data can also be:

[0867] [[1, order(shop = store C, product = latte, no = 1, size = large cup, temperature = iced)]].

[0868] After the electronic device obtains the DSL data, it can further parse the DSL data and execute the takeout ordering task. In this way, the user only needs to input an instruction, and the electronic device can help the user complete a series of operations for ordering takeout through the App, improving the user experience.

[0869] For example, taking the task of creating a reminder as an example, the user instruction is "book a meeting reminder at 14:00 today". After receiving the user instruction, the electronic device inputs the user instruction into the task agent model, performs model inference through the task agent model, and outputs DSL data corresponding to the create_reminder task. For example, the DSL data can be: [[1, create_reminder(content=meeting reminder, time=14:00)]].

[0870] After the electronic device obtains the DSL data corresponding to the create_reminder task, the DSL data can be parsed and the create_reminder task can be executed. In this way, the user only needs to input the instruction, and the electronic device can help the user complete a series of operations of creating a reminder, thereby improving the user experience.

[0871] For example, taking the task of taking a taxi as an example, the user instruction is "take a taxi to XX Mansion". After receiving the user instruction, the electronic device inputs the user instruction into the task agent model, performs model inference through the task agent model, and outputs DSL data corresponding to the take_taxi task. For example, the DSL data can be: [[1, take_taxi(deplocation=None, deslocation=XX Mansion, app=None)]].

[0872] After the electronic device obtains the DSL data corresponding to the take_taxi task, the DSL data can be parsed and the task can be executed. In this way, the user only needs to input the instruction, and the electronic device can help the user complete a series of operations of taking a taxi through the App, thereby improving the user experience.

[0873] For example, taking the task of navigation as an example, the user instruction is "navigate to XX Mansion by bike". After receiving the user instruction, the electronic device inputs the user instruction into the task agent model, performs model inference through the task agent model, and outputs DSL data corresponding to the navigate task. For example, the DSL data can be: [[1, navigate(deplocation=None, deslocation=XX Mansion, pathPoint=None, trafficType=bike, app=None)]].

[0874] After the electronic device obtains the DSL data corresponding to the navigate task, the DSL data can be parsed and the task can be executed. In this way, the user only needs to input the instruction, and the electronic device can help the user complete a series of operations of setting navigation through the App, thereby improving the user experience.

[0875] The reasoning process and the training process of the task agent model are described in detail below.

[0876] Reasoning process of task agent model

[0877] Embodiments of the present application provide a task agent model suitable for different tasks, which can perform task decomposition based on vertical domains, integrate lora data configured by different vertical domains with a general agent model, and then perform reasoning through the integrated task agent model. The general agent model is a large model trained based on multiple business data.

[0878] The overall reasoning process of the task agent model, reasoning using a large language model (LLM) combined with a prompt caching strategy, optimization of DSL design, and optimization of model output are described below.

[0879] Overall reasoning process

[0880] In embodiments of the present application, the task agent model needs to have an intent recognition function, and determines lora data corresponding to the vertical domain according to the identified intent information, and integrates the lora data with the general agent model. Different vertical domains provide lora models for different tasks, and the lora models are integrated with the general agent model to perform model reasoning through the integrated task agent model, which has better prediction performance.

[0881] Two different vertical domains, independent vertical domains and common card vertical domains, are provided in embodiments of the present application. The independent vertical domain has pre-set lora data corresponding to a single task, and the common card vertical domain has pre-set lora data corresponding to multiple tasks.

[0882] The independent vertical domain configures lora data, which means that the lora data of a certain business is configured on one GPU card.

[0883] The common card vertical domain configures multiple lora data, which means that the lora data corresponding to multiple businesses is collectively configured on one GPU card.

[0884] In embodiments of the present application, after intent recognition of user instructions, tasks such as ordering takeout, sending files, or querying for a renewal can be identified. For different tasks, lora data configured by different vertical domains can be integrated with a general agent model, and then reasoning can be performed through the integrated task agent model to obtain DSL data for different tasks.

[0885] For the service with less slot, the time delay pressure is less, and therefore the lora data related to the service with less slot can be configured in the common card vertical domain. For example, the service with less slot can be a service of querying for a renewal, sending a file, etc. The lora data related to sending a file (referred to as sending file lora) and the lora data related to querying for a renewal (referred to as querying renewal lora) can be configured in the common card vertical domain.

[0886] For the service with more slot, the time delay pressure is greater, and therefore the lora data related to the service with more slot can be separately configured in the independent vertical domain. For example, the service with more slot can be a service of ordering takeout, etc. The lora data related to ordering takeout (referred to as ordering takeout lora) can be configured in the common card vertical domain.

[0887] It should be noted that the service with less or more slot is illustratively described above, and which lora data is configured in the independent vertical domain or the common card vertical domain is illustratively described, and in actual implementation, which lora data is configured in the independent vertical domain or the common card vertical domain can be determined according to actual needs.

[0888] The embodiment of the present application constructs a task agent model framework based on vertical domain distribution, and designs a strategy of reasoning after integrating the lora based on the common card vertical domain and the general agent model, and a strategy of reasoning after integrating the lora based on the independent vertical domain and the general agent model.

[0889] FIGS. 17B and 17C respectively show internal implementation flow diagrams of a task agent model provided by the embodiment of the present application.

[0890] Referring to FIGS. 17B and 17C, after a user instruction is input into the task agent model, first, intent recognition is performed according to the user instruction to obtain intent information (for example, ordering takeout, sending a file, or querying for a renewal).

[0891] Then, the vertical domain is distributed according to the intent information. For example, if the intent information is ordering takeout, the independent vertical domain is distributed, and the independent vertical domain is preset with ordering takeout lora data. If the intent information is querying for a renewal, the common card vertical domain is distributed, and the common card vertical domain is preset with multiple lora data, such as sending file lora data and querying renewal lora data.

[0892] After the vertical domain is distributed, the lora data of the independent vertical domain or the common card vertical domain is integrated with the general agent model. The lora data of the independent vertical domain or the common card vertical domain uses the same task agent model base.

[0893] For example, if the intent information is ordering takeout and is distributed to the independent vertical domain, the ordering takeout lora data preset in the independent vertical domain is integrated with the general agent model.

[0894] For example, when the intent information is to query for a renewal, the query renewal lora data preset in the common card vertical domain is integrated with the general agent model. For example, when the intent information is to send a file, the send file lora data preset in the common card vertical domain is integrated with the general agent model.

[0895] In some cases, after being distributed to the common card vertical domain according to the intent information, if the lora data currently configured in the common card vertical domain is inconsistent with the intent information, the lora data currently configured in the common card vertical domain is switched to the lora data corresponding to the intent information.

[0896] For example, the intent information is to query for a renewal, and the lora data currently configured in the common card vertical domain is send file lora data. In this case, it can be determined that the lora data currently configured in the common card vertical domain is inconsistent with the intent information, and accordingly the lora data currently configured in the common card vertical domain can be switched to the query renewal lora corresponding to the intent information. After switching to the query renewal lora data, the query renewal lora data is integrated with the general agent model.

[0897] Then, model inference is performed through the integrated task agent model, and then the DSL data obtained through inference is output.

[0898] In the embodiments of the present application, the lora data of different tasks is integrated with the general agent model based on the characteristics of the vertical domain, and model inference is performed through the integrated task agent model, which can maximize the saving of time delay and resources.

[0899] The lora data is generated based on business understanding through a large model first, and then obtained through artificial annotation. The lora data of each business can be a list, and each element in the list is a sample (input: input; output: output).

[0900] The lora data of the point take-out business is exemplarily shown below.

[0901] [{"input":"<|im_start|>You are an agent with task planning capabilities, able to transform user input into a DSL for tool orchestration.\n1. The DSL requires a list of tasks. Example format: [['1','Tool Name 1 (Parameter 1 = Parameter Value 1, Parameter 2 = Parameter Value 2,...)'],['2','Tool Name 2 (Parameter 1 = Parameter Value 1,...)']]. The first item of each task is the task ID, sequentially encoded starting from 1; the second item is the tool name, tool parameters, and corresponding values; Parameter 1 and Parameter 2 are the parameters required by the tool. If a parameter does not have a specific value, that parameter and its value are not output; 2. Task orchestration must be based solely on the given candidate tools.\nNote: Only the DSL is output. Formatted results, no need to output analysis process or other content. Candidate Tool: 1. Ordering takeout. Tool Input Parameters: app (application name, e.g., Meituan, Mini Program, optional), shop (merchant brand name, e.g., Luckin Coffee, optional), product (product, e.g., Americano, Latte, optional), no (quantity, must be mapped to a number, e.g., one cup is 1, optional), size (portion, e.g., extra-large, optional), temperature (temperature, optional), sweetness (sweetness, optional), location (destination, optional). Output Parameters: Empty. User Input: Order me a Starbucks latte, extra-large with ice. DSL: <|im_end|>",

[0902] "output":"[['1','order(shop=Starbucks,product...

Claims

1. A UI task automatic execution method, characterized by, The method is applied to an electronic device, and the method comprises: When a first interface is displayed, a first prompt window is displayed, and a first user instruction is received; In response to the first user instruction, after M second interfaces are sequentially displayed, a third interface is displayed; the third interface comprises first information for prompting a user to take over; M is a positive integer; In response to a first operation input by the user on the third interface, after N fourth interfaces are sequentially displayed, a fifth interface is displayed; the fifth interface is an interface associated with the first user instruction, and N is a positive integer.

2. The method of claim 1, wherein, The method further comprises: When the M second interfaces are sequentially displayed, a cursor is displayed on each second interface; the cursor is displayed on a target control for triggering display of a next interface of a current interface.

3. The method according to claim 1 or 2, characterized in that, The sequentially displaying of the M second interfaces comprises: displaying one second interface in a full-screen window, and displaying a first control in the full-screen window; The method further comprises: in response to an operation of the user on the first control, switching the full-screen window to a floating window, displaying the floating window on the first interface, and continuing to sequentially display M-1 second interfaces in the floating window.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: When the M second interfaces are sequentially displayed through a first window, task execution progress information is displayed in the first window; and / or, a light effect layer is displayed on the first window; The first window comprises a full-screen window or a floating window.

5. The method according to claim 3 or 4, characterized in that, The method further comprises: In the full-screen window state, if an application of a preset type is started, the full-screen window is switched to the floating window; The application of the preset type comprises a video application or a game application.

6. The method of claim 4, wherein, The method further comprises: The light effect layer is drawn on a system customized dynamic effect layer level; When the light effect layer is displayed on the first window, if a touch operation of the user is received, the touch operation is transmitted to a next level of the dynamic effect layer level, and the touch operation is not responded.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: According to a task execution state change, the light effect layer displayed on the first window is updated; the task execution state comprises an executing state, a user takeover state, and an execution end state; According to the task execution state change, the light effect layer displayed on the first window is updated, which comprises: in the executing state, a first light effect layer is displayed on the first window; in the user takeover state, a second light effect layer is displayed on the first window; and in the execution end state, the light effect layer displayed on the first window is hidden.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: During the sequentially displaying of the M second interfaces, in response to an operation of the user on any second interface, second information is displayed; the second information comprises prompt information that does not support an operation when automatically executed, an exit control, and a continue execution control.

9. The method according to any one of claims 1 to 8, characterized in that, The third interface is an identity verification interface, a password input interface, an information confirmation interface, a permission request interface, a single selection or multiple selection interface, or a pop-up window that needs to be taken over by the user.

10. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: In response to the first user instruction, a first operation sequence is determined according to a first task graph, and the M second interfaces are sequentially displayed according to the first operation sequence; The first task graph includes a start node, a plurality of intermediate nodes, and an end node, the start node is a first interface of the M second interfaces, and the fifth interface is the end node.

11. The method of claim 10, wherein, After displaying the third interface, the method further includes: In response to a first operation input by a user on the third interface, a node position of a currently displayed interface in a first task graph is determined, and a second operation sequence is determined according to the node position and the first task graph; the second operation sequence includes the N fourth interfaces.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: In the case that an advertisement pop-up window appears on the second interface, the advertisement pop-up window is automatically closed.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: In the case that a target control is not displayed on the second interface, the second interface is triggered to be displayed by scrolling; After the second interface is displayed by scrolling, the target control in the second interface is displayed, and a cursor is displayed at the target control.

14. The method according to any one of claims 1 to 13, characterized in that, The method further includes that a continue execution control is displayed on the third interface. After displaying the third interface, the method further includes that, in the case that a user operation on the continue execution control is received, the N fourth interfaces are sequentially displayed.

15. The method according to any one of claims 1 to 14, characterized in that, The electronic device includes a system assistant and a UI agent module; the method further includes: The system assistant determines intent information and slot information according to the first user instruction, and sends the intent information and the slot information to the UI agent module; the intent information indicates a first business type; The UI agent module determines first robot process automation (RPA) configuration information according to the intent information, and a business type identifier of the first RPA configuration information corresponds to the first business type; The UI agent module generates a user interface (UI) task operation step according to the slot information and the first RPA configuration information, and the UI task operation step includes sequentially displaying the M second interfaces and displaying the fifth interface.

16. The method of claim 15, wherein, The first RPA configuration information includes a task identifier (ID), a tool name, and one or more tool parameters, and one tool parameter corresponds to one slot; The generation of the UI task operation step according to the slot information and the first RPA configuration information includes: The slot information is used to fill the slots in the first RPA configuration information, and the UI task operation step is generated, the UI task operation step includes the task identifier (ID), the tool name, and the one or more tool parameters, and each tool parameter has a slot value.

17. The method according to claim 15 or 16, characterized in that, The UI task operation step further includes a target control and a UI task, and the automatic execution process includes: The target control is identified in each interface, and a position coordinate of the target control is determined; The UI task is performed on the target control according to the position coordinate of the target control; After the UI task is performed, the current display interface jumps to a next interface.

18. The method of claim 17, wherein, The target control is at least one of the following: text, button, sliding switch, icon; The UI task is clicking or sliding the target control; The target control is a search box; 19. The method of claim 17 or 18, wherein, The UI task is inputting a keyword in the search box; The target control is an icon control and has associated text. The target control is detected based on page structure features; the page structure features include XML structure information and / or DOM.

20. The method of any one of claims 17-19, wherein, The target control is detected based on page visual features. The target control is detected based on page visual features and page structure features when the target control is an icon control and has associated text. The target control is detected based on page layout features when the target control is a first text and the page includes multiple first texts. The target control is detected based on the page visual features, the page structure features, and the page layout features when the target control is not detected based on the page visual features and / or the page structure features. The target control is detected based on the page visual features and the position coordinates of the associated text when the associated text is a variable parameter. The target control is detected based on the page visual features and the position coordinates of the associated text.

21. The method of claim 20, wherein, The user intent information is determined based on the first user instruction. The system assistant determines the user intent information based on the user instruction and user behavior habit data. The user intent information includes any of the following: querying automatic renewal, sending files, closing application permissions, optimizing overall performance, closing application notifications, ordering takeout.

22. The method of any one of claims 17-21, wherein, The method further includes: When the fifth interface is displayed, a second prompt window is displayed on the fifth interface, and the second prompt window displays business information corresponding to the first user instruction.

23. The method of any one of claims 17-22, wherein, The electronic device includes one or more processors and a memory; 24. The method of claim 18, wherein, The memory is coupled to the one or more processors, and the memory is configured to store computer program code including computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the method of any one of claims 1-24. ​ 25. An electronic device, comprising: ​ ​

Images