Task processing method and device and electronic equipment

Abstract

The invention discloses a task processing method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The task processing method comprises the steps of obtaining task execution information of a task under the condition that the task meets an automatic execution condition; wherein the task execution information comprises pre-recorded task step description information of at least one task step of executing the task; and executing the task according to the task execution information.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a task processing method, device and electronic equipment. Background Art

[0002] With the development of artificial intelligence technology, large-scale intelligent agents can be used to automatically control electronic devices, meeting user needs for automatic operation of electronic devices. For example, if a user inputs "Please order a takeout meal from Merchant A priced between 30 and 50 yuan to my office," the large-scale intelligent agent will launch the food delivery app, search for Merchant A within the app, then search for food items priced between 30 and 50 yuan on Merchant A's food page, add one of the searched items to the shopping cart, place an order, enter the user's company address as the delivery address on the order page, and then make payment.

[0003] However, the range of tasks that a large-model agent can reliably handle is limited by the scope of its training. If the task to be automated is one for which the large-model agent has not been trained or tested, the large-model agent will be unable to perform it, resulting in task failure. For example, if a large-model agent has not been trained for the automated taxi-hailing task, it will be unable to perform the task when asked to do so, resulting in a failure. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a task processing method, device and electronic device that can execute tasks that need to be executed automatically and avoid task execution failure.

[0005] In a first aspect, an embodiment of the present application provides a task processing method, including:

[0006] When the task meets the automatic execution condition, task execution information of the task is obtained; wherein the task execution information includes pre-recorded task step description information of at least one task step of executing the task;

[0007] Execute the task according to the task execution information.

[0008] In a second aspect, an embodiment of the present application provides a task processing device, including:

[0009] An acquisition module, configured to acquire task execution information of a task when the task satisfies an automatic execution condition; wherein the task execution information includes pre-recorded task step description information of at least one task step of executing the task;

[0010] The execution module is used to execute tasks according to the task execution information.

[0011] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the task processing method provided in the embodiment of the present application are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of task processing provided in the embodiment of the present application are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the task processing method provided in the embodiment of the present application.

[0014] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the task processing method provided in an embodiment of the present application.

[0015] In an embodiment of the present application, when a task satisfies an automatic execution condition, task execution information for the task is obtained; wherein the task execution information includes pre-recorded task step description information for at least one task step of executing the task; and the task is executed according to the task execution information. Because at least one task step of the task to be automatically executed is pre-recorded, the task execution information includes the pre-recorded task step description information for the at least one task step of the task. Therefore, even if the task is a task for which the large model intelligent agent has not been trained or tested, the task can be automatically executed according to the pre-recorded task step description information for the at least one task step of the task, thereby avoiding failure in the execution of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a task processing method provided by some embodiments of the present application;

[0017] Figure 2 is a schematic diagram of third prompt information provided in some embodiments of the present application;

[0018] Figure 3 is a schematic diagram of fifth prompt information provided in some embodiments of the present application;

[0019] Figure 4 is a schematic diagram of fourth prompt information provided in some embodiments of the present application;

[0020] Figure 5is a schematic diagram of second prompt information provided in some embodiments of the present application;

[0021] Figure 6 is a schematic diagram of an interface provided by some embodiments of the present application when no execution exception occurs;

[0022] Figure 7 is a schematic diagram of an interface provided by some embodiments of the present application when an execution exception occurs;

[0023] Figure 8 is a schematic diagram of first prompt information provided by some embodiments of the present application;

[0024] Figure 9 is a schematic diagram of the structure of a task processing device provided in some embodiments of the present application;

[0025] Figure 10 is a schematic structural diagram of an electronic device provided by some embodiments of the present application;

[0026] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0028] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The terms used in the embodiments of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is an explanation of the terms involved in the embodiments of this application.

[0030] A large model refers to a deep learning model with a huge number of parameters and strong reasoning capabilities. It can handle complex tasks and generate high-quality text, image or voice output. An intelligent agent refers to an entity or system that can perceive the environment, make autonomous decisions and perform actions to achieve specific goals. A large-model intelligent agent is an intelligent system that uses the reasoning capabilities of a large model to make autonomous decisions. It can perceive the environment, make inferences and perform complex tasks, and achieve human-like interaction and multi-scenario applications, such as automatic operation on a mobile phone. A control (also called a part, component, widget or control) is a graphical user interface element and the basic building block of the user interface, such as a window or text box, displayed in the program interface of any application. A control can be a button, text box, label, etc., which is used to control all data processed by each application and the interactive operations on this data.

[0031] The task processing method, device, and electronic device provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0032] It should be noted that the task processing methods provided in the embodiments of this application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and in-vehicle electronic devices. In some embodiments of this application, the task processing methods provided in the embodiments of this application are described by taking electronic devices as the execution entities to execute the task processing methods.

[0033] The task processing method provided in the embodiment of the present application can be applied to the scenario of automatic task execution; among them, one specific application scenario is the automatic taxi-hailing scenario, and another application scenario is the scenario of automatically reducing the screen brightness of an electronic device when the battery level is lower than a battery threshold.

[0034] It should be noted that the above application scenario is only an example, and actual application may also include other application scenarios, such as the scenario of automatically ordering takeout.

[0035] Figure 1 This is a flowchart of a task processing method provided in some embodiments of the present application. The task processing method may include:

[0036] Step 101: When a task satisfies an automatic execution condition, task execution information of the task is obtained; wherein the task execution information includes pre-recorded task step description information of at least one task step of executing the task;

[0037] In some embodiments of the present application, the task in the embodiment of the present application satisfies the self-execution condition including at least one of the following: the electronic device is within the range of triggering the automatic execution of the task, and the time of the electronic device reaches the time point of triggering the automatic execution of the task.

[0038] In some embodiments of the present application, the task step description information in the embodiments of the present application is information used to describe the task steps of the task; the task step description information in the embodiments of the present application includes at least one of the following: the execution sequence number of the task step, the step name of the task step, the gesture feature information of executing the task step, the input area of the text, the input text content, the interface screenshot during the execution of the task step, the interface description information of the interface screenshot, and the device system information when executing the task step.

[0039] Among them, the execution sequence number of the task step is used to indicate which step of the task the task step is; the gesture feature information for executing the task step includes but is not limited to the user gesture type and the user gesture area, among which the user gesture type includes but is not limited to sliding, clicking, double-clicking, etc., and the user gesture area is the area where the user gesture operates on the screen; the text input area is the area where the user inputs text; the interface screenshots during the execution of the task step include the interface screenshots before the task step is executed and the interface screenshots after the task step is executed; the interface description information of the interface screenshots is the information used to describe the interface corresponding to the interface screenshots, for example, the interface corresponding to the interface screenshots is the lock screen interface, and the interface corresponding to the interface screenshots is the desktop interface; the device system information when executing the task step includes but is not limited to: device time, device geographical location, device power, etc. The interface description information of the interface screenshots also includes: location information of some key elements in the interface corresponding to the interface screenshots, among which key elements include but are not limited to: application icons, search boxes, buttons, pictures, etc.

[0040] For example, in the aforementioned automatic taxi-hailing scenario, the task "Have a taxi to go to the office" satisfies the following automatic execution conditions: "Time: 8:00 AM on a weekday, Location: The electronic device's address is located within the user's residential area." If the time is 8:00 AM on a weekday and the electronic device's address is located within the user's residential area, the pre-recorded task step description for at least one step of the "Have a taxi to go to the office" task is obtained.

[0041] The "Take a taxi to the company" task includes the following steps:

[0042] Task step 1: Click the X Map application icon on the desktop of the electronic device to open the X Map application;

[0043] Task step 2: Click the search box on the main interface of the X Maps app;

[0044] Task step 3: Click the first record in the history and select the recommended location "Company";

[0045] Task step 4: Click the "Hit a Taxi" control in the interface;

[0046] Task step 5: Select the price range: 13-16 yuan;

[0047] Task step six: Click the "Get a taxi now" control in the interface.

[0048] Taking the task step description information of task step 1 "Click the X map application icon on the desktop of the electronic device to open the X map application" as an example, the task step description information obtained for task step 1 "Click the X map application icon on the desktop of the electronic device to open the X map application" is as follows:

[0049] Task: Take a taxi to the company; Task step execution sequence: 1; Task step name: Click the X Map application icon on the electronic device's desktop to open the X Map application; Gesture type for executing the task step: Click; Gesture area for executing the task step: (100, 300); Text input area: Empty; Input text content: Empty; Screenshot of the interface before executing the task step: before-move1.jpg; Screenshot of the interface after executing the task step: after-move1.jpg; Interface description of the screenshot before executing the task step: The interface corresponding to the electronic device's desktop; Interface description of the screenshot after executing the task step: The interface corresponding to the main interface of the X Map application; Device time when executing the task step: 8:00 AM, January 15, 2025; Device location when executing the task step: Community A.

[0050] For example, for the aforementioned "Lower Screen Brightness" task for automatically lowering the screen brightness when the electronic device's battery level falls below a battery threshold, the task satisfies the automatic execution condition of "the electronic device's battery level is below 40%." When the electronic device's battery level falls below 40%, task step description information for at least one pre-recorded task step for executing the "Lower Screen Brightness" task is obtained.

[0051] The "Reduce screen brightness" task includes the following steps:

[0052] Task step 1: Call up the "brightness adjustment" control through the status bar of the electronic device;

[0053] Task step 2: Use the "Brightness Adjustment" control to adjust the screen brightness of the electronic device to 40%.

[0054] Taking the task step description information of task step 2 "Adjust the screen brightness of the electronic device to 40% using the 'Brightness Adjustment' control" as an example, the task step description information obtained for task step 2 "Adjust the screen brightness of the electronic device to 40% using the 'Brightness Adjustment' control" is as follows:

[0055] Task: Lower the screen brightness; Task step execution sequence number: 2; Task step name: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control; Gesture type for executing the task step: Swipe down; Gesture area for executing the task step: Move from screen coordinates (200,100) to screen coordinates (200,160); Text input area: Empty; Input text content: Empty; Interface screenshot before the task step is executed: before-move2.jpg; Interface screenshot after the task step is executed: after-move2.jpg; Interface description information of the interface screenshot before the task step is executed: The interface corresponding to the interface screenshot is the control center interface; Interface description information of the interface screenshot after the task step is executed: The interface corresponding to the interface screenshot is the control center interface.

[0056] In some embodiments of the present application, before step 101, the task processing method provided by the embodiments of the present application may also include: receiving recording control input; recording task step description information of each task step in the task execution process in response to the recording control input; and generating task execution information based on the task step description information of each task step.

[0057] In some embodiments of the present application, the recording control input in the embodiments of the present application is used to trigger the task step of the recording task. The recording control input in the embodiments of the present application includes but is not limited to: the user's touch input to the control used to trigger the task step of the recording task through a touch object; wherein the touch object includes but is not limited to a touch device such as a finger or a stylus, and the touch input includes but is not limited to click input, sliding input, etc. The click input can be a single-click input, a double-click input, or any number of click inputs, and can also be a long press input or a short press input. The recording control input in the embodiments of the present application can be set and adaptively modified according to actual needs.

[0058] After the user touches the control for triggering the task step of the recording task through the touch object, the task step description information of each task step in the task execution process is recorded, and then the task execution information of the task is generated based on the recorded task step description information of each task step.

[0059] In some embodiments of the present application, task execution information of a task is generated based on the task step description information of each task step. Multiple task steps can be sorted according to the execution sequence number of each task step to obtain the task execution information of the task.

[0060] In some embodiments of the present application, before receiving the recording control input, the task processing method provided by the embodiments of the present application may also include: displaying a third prompt message when the execution frequency of the task is greater than the frequency threshold; wherein the third prompt message is used to prompt the user to confirm whether to record the task steps of the task.

[0061] In some embodiments of the present application, users often repeat a task at a certain time or place. At this time, the electronic device can actively identify tasks that are performed frequently and recommend the user to record the operation steps of the task. When the electronic device identifies tasks that are performed frequently, the electronic device can monitor the user's use record of the electronic device, including but not limited to: the user's operation process, operation time, the geographical location of the electronic device when operating the electronic device, the status of the electronic device, etc.; wherein the status of the electronic device includes but is not limited to the network connection information of the electronic device, the power of the electronic device, etc., and construct a scene feature vector for the continuous operation process. Calculate the similarity between each scene feature vector, obtain a scene feature vector that is similar to multiple scene feature vectors, take the operation process corresponding to the scene feature vector and multiple scene feature vectors similar to the scene feature vector as a task, determine the execution frequency of the task according to the time when the user performs the task, and determine the time when the task is triggered, the geographical location of the electronic device when the electronic device is used, the status of the electronic device, and other information based on the scene feature vector and multiple scene feature vectors similar to the scene feature vector.

[0062] For example, for a task requiring a user to take a taxi to the office at 8:00 AM every Monday through Friday, the task is performed five times per week. Assuming the frequency threshold is four times per week, if the task's execution frequency exceeds the frequency threshold, a prompt message will be displayed at 8:00 AM every Monday through Friday, prompting the user to confirm whether to record the task steps. If the user has already recorded multiple steps for the task requiring a taxi to the office at 8:00 AM, the prompt message will no longer be displayed.

[0063] For example, the third prompt information is explained by taking the above-mentioned "take a taxi to the company" task as an example. At 8:00 on January 15, 2025, a high-frequency task of taking a taxi to the company is detected, and the following prompt information can be displayed: Figure 2 The third prompt information shown is, Figure 2 This is a schematic diagram of the third prompt information provided by some embodiments of the present application. Figure 2 In the example, the third prompt message 201 is "A high-frequency taxi-to-company task is detected. Do you want to record the task steps of the taxi-to-company task to automatically execute the taxi-to-company task?" Figure 2The "Record" control 202 and the "Cancel" control 203 are also displayed. The "Record" control 202 is used by the user to confirm recording of the task steps, while the "Cancel" control 203 is used by the user to confirm not to record the task steps. When the user clicks the "Record" control 202, the task steps are recorded; when the user clicks the "Cancel" control 203, the task steps are not recorded.

[0064] In an embodiment of the present application, by displaying prompt information for prompting the user to confirm whether to record the task steps of the task, the user can be guided to record tasks that are performed frequently, thereby improving the recording efficiency of the task.

[0065] In some embodiments of the present application, after recording a certain frequently performed task, a fifth prompt message may be displayed to prompt the user to confirm whether to automatically record the task steps of the frequently performed task. For example, the fifth prompt message is described using the above-mentioned "take a taxi to the company" task as an example. After recording the task steps of the above-mentioned "take a taxi to the company", the following prompt message may be displayed: Figure 3 The fifth prompt information shown. Figure 3 This is a schematic diagram of the fifth prompt information provided by some embodiments of the present application. Figure 3 In the example, the fifth prompt message 301 is "The task steps of the task of taking a taxi to the company have been recorded. Do you want to automatically record the task steps of other frequently executed tasks?" Figure 3 Also displayed are a "Record" control 302 and a "Cancel" control 303; the "Record" control 302 is used by the user to confirm that the task steps of other frequently executed tasks should be automatically recorded, while the "Cancel" control 303 is used by the user to confirm that the task steps of other frequently executed tasks should not be automatically recorded. After the user clicks the "Record" control 302, if a frequently executed task is detected again, the task steps of the frequently executed task will be automatically recorded, and the third prompt message will no longer be displayed. After the user clicks the "Cancel" control 303, if a frequently executed task is detected again, the third prompt message will be displayed, allowing the user to confirm whether to record the task steps of the frequently executed task.

[0066] In some embodiments of the present application, before step 101, the task processing method provided by the embodiments of the present application may further include: displaying a fourth prompt message; wherein the fourth prompt message is used to prompt the user to confirm whether to automatically execute the task; and upon receiving input from the user confirming the automatic execution of the task, determining that the task meets the automatic execution conditions.

[0067] For example, the fourth prompt information is explained by taking the above-mentioned “take a taxi to the company” task as an example. On the second day after recording the task steps of the above-mentioned “take a taxi to the company” task, that is, January 16, 2025, when the electronic device meets the above-mentioned “take a taxi to the company” task, the following prompt information can be displayed: Figure 4 The fourth prompt information shown is, Figure 4 This is a schematic diagram of the fourth prompt information provided by some embodiments of the present application. Figure 4 In the example, the fourth prompt message 401 is "A recorded task of taking a taxi to the company has been detected. Do you want to automatically execute the task of taking a taxi to the company?" Figure 4 The electronic device also displays an "Execute" control 402 and a "Cancel" control 403; the "Execute" control 402 is used by the user to confirm that the task should be automatically executed, while the "Cancel" control 403 is used by the user to confirm that the task should not be automatically executed. When the user clicks the "Execute" control 402, the electronic device receives the input confirming the automatic execution of the task, determines that the task meets the automatic execution conditions, and then obtains the task execution information.

[0068] Step 102: Execute the task according to the task execution information.

[0069] After obtaining task execution information including pre-recorded task step description information for at least one task step of the task execution, the task is executed according to the task step description information of the at least one task step included in the task execution information. When executing a task step, the current screenshot is compared with the pre-recorded screenshot corresponding to the task step. If the current screenshot is identical to the pre-recorded screenshot corresponding to the task step, the task step is executed according to the task step description information. If the task step is successfully executed, the next task step is executed.

[0070] For example, the task execution steps are described by taking "Task step 1: click the X map application icon on the desktop of the electronic device to open the X map application" in the "take a taxi to the company" task of the above-mentioned automatic taxi-hailing scenario as an example.

[0071] First, obtain the current screenshot and the screenshot corresponding to "Task step 1: click the X map application icon on the desktop of the electronic device to open the X map application"; when the current screenshot and the screenshot corresponding to "Task step 1: click the X map application icon on the desktop of the electronic device to open the X map application" are the same, execute the task according to the task step description information of the above "Task step 1: click the X map application icon on the desktop of the electronic device to open the X map application", that is, execute the operation instruction corresponding to clicking the screen coordinates (100, 300) to complete the opening operation of the X map application.

[0072] For example, the task execution steps are explained by taking "Task Step 2: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" in the "Reduce Screen Brightness" task for the scenario of automatically reducing the screen brightness of the electronic device when the battery level is lower than the battery threshold as an example.

[0073] First, obtain the current screenshot and the screenshot corresponding to "Task step two: Adjust the screen brightness of the electronic device to 40% through the "Brightness adjustment" control"; when the current screenshot and the screenshot corresponding to "Task step two: Adjust the screen brightness of the electronic device to 40% through the "Brightness adjustment" control" are the same, execute the task according to the task step description information of the above "Task step two: Adjust the screen brightness of the electronic device to 40% through the "Brightness adjustment" control", that is, execute the operation instruction corresponding to sliding down from the screen coordinates (200,100) to the screen coordinates (200,160) to complete the adjustment of the screen brightness of the electronic device to 40%.

[0074] In an embodiment of the present application, when a task satisfies an automatic execution condition, task execution information for the task is obtained; wherein the task execution information includes pre-recorded task step description information for at least one task step of executing the task; and the task is executed according to the task execution information. Because at least one task step of the task to be automatically executed is pre-recorded, the task execution information includes the pre-recorded task step description information for the at least one task step of the task. Therefore, even if the task is a task for which the large model intelligent agent has not been trained or tested, the task can be automatically executed according to the pre-recorded task step description information for the at least one task step of the task, thereby avoiding failure in the execution of the task.

[0075] In some embodiments of the present application, after step 102, the task processing method provided by the embodiments of the present application may further include: displaying a second prompt message when the task execution is completed; wherein the second prompt message is used to prompt the user to confirm whether the task execution is successful; when the user confirms that the task execution fails through the second prompt message, displaying at least one task step image; wherein the task step image is a screenshot image of the content displayed on the display screen during the execution of each task step of the task; receiving a selection input for one of the at least one task step images; wherein the task step image selected by the selection input corresponds to the first task step; in response to the selection input, re-recording the second task step corresponding to the task step image selected by the selection input; replacing the pre-recorded first task step with the re-recorded second task step.

[0076] For example, the second prompt information is explained by taking the above-mentioned "take a taxi to the company" task as an example. After the above-mentioned "take a taxi to the company" task is completed, the following information may be displayed: Figure 5 The second prompt information shown is Figure 5 is a schematic diagram of the second prompt information provided by some embodiments of the present application. Figure 5 In the example, the second prompt message 501 is a text "Has the task of taking a taxi to the company been successfully completed?" Figure 5 Also displayed are a "Success" control 502 and a "Failure" control 503; wherein, the "Success" control 502 is used by the user to confirm that the recorded task has been successfully executed, and the "Failure" control 503 is used by the user to confirm that the task has failed. When the user clicks the "Failure" control 503, the user confirms that the task has failed, and at least one task step image is displayed. The user can use these task step images to confirm which task step has a problem that caused the task to fail. The user can click on the corresponding task step image. At this time, the electronic device receives the user's selection input for the task step image and then re-records the task step corresponding to the task step image. After the task step is recorded, the task step corresponding to the task step image in at least one pre-recorded task step of the task execution is replaced with the re-recorded task step, completing the update of at least one task step of the task.

[0077] For example, for the "take a taxi to the company" task in the above-mentioned automatic taxi-hailing scenario, when the "take a taxi to the company" task is completed, a second prompt message is displayed to prompt the user to confirm whether the task has been successfully executed; in the case where the user confirms through the second prompt message that the task has failed, a screenshot image of the display screen content during the execution of each task step of the "take a taxi to the company" task is displayed; the user can use these screenshot images to confirm which task steps have problems that caused the failure of "take a taxi to the company". Assuming that the user confirms through these screenshot images that the task step that caused the failure of "take a taxi to the company" is "Task Step Six: Click the "Have a Taxi Now" control in the interface", the user can click on the screenshot image corresponding to "Task Step Six: Click the "Have a Taxi Now" control in the interface" to re-record "Task Step Six: Click the "Have a Taxi Now" control in the interface". After the recording of "Task Step Six: Click the "Have a Taxi Now" control in the interface" is completed, replace the pre-recorded "Task Step Six: Click the "Have a Taxi Now" control in the interface" with the re-recorded "Task Step Six: Click the "Have a Taxi Now" control in the interface" to complete the update of "Task Step Six: Click the "Have a Taxi Now" control in the interface".

[0078] For example, for the above-mentioned "lower screen brightness" task in the scenario where the screen brightness of an electronic device is automatically lowered when the battery level is lower than a battery threshold, after the "lower screen brightness" task is completed, a second prompt message is displayed to prompt the user to confirm whether the task has been successfully executed; in the case where the user confirms through the second prompt message that the task has failed to execute, a screenshot image of the display screen content during the execution of each task step of the "lower screen brightness" task is displayed; the user can use these screenshot images to confirm which task steps have problems that caused the "lower screen brightness" to fail. Assuming that the user confirms through these screenshot images that the task step that caused the failure of "lower screen brightness" to execute is "Task step two: adjust the screen brightness of the electronic device to 40% through the "brightness adjustment" control", the user can click on the screenshot image corresponding to "Task step two: adjust the screen brightness of the electronic device to 40% through the "brightness adjustment" control" to re-record "Task step two: adjust the screen brightness of the electronic device to 40% through the "brightness adjustment" control". After the recording of "Task Step 2: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" is completed, replace the pre-recorded "Task Step 2: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" with the re-recorded "Task Step 2: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" to complete the update of "Task Step 2: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control".

[0079] In some embodiments of the present application, when re-recording, the user can also choose to re-record part of the task steps or all of the task steps; wherein the part of the task steps includes the task steps determined by the user to cause the task execution failure.

[0080] In the embodiment of the present application, by re-recording the task steps that lead to the failure of task execution, the occurrence of task execution failure during subsequent automatic execution of the task can be avoided, thereby increasing the probability of successful task execution.

[0081] In some embodiments of the present application, the task execution information includes task step description information of N task steps of the task; N is a positive integer; the task processing method provided by the embodiment of the present application may also include: in the event that an execution exception occurs during the execution of the i-th task step, generating execution exception description information corresponding to the i-th task step; wherein the i-th task step is any task step among the N task steps, i≤N, and i is a positive integer; performing a semantic comparison between the execution exception description information corresponding to the i-th task step and the registered reference exception description information; in the event that there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold, executing the i-th task step according to the exception handling method corresponding to the reference exception description information whose semantic similarity is greater than the similarity threshold.

[0082] In some embodiments of the present application, the execution exception includes at least one of the following:

[0083] Changes in interface content, appearance of an interface that was not displayed during the recording of the i-th task step, task execution response timeout, restricted task execution permissions, abnormal positioning of interface elements, obstruction of the interface, and abnormal application operation.

[0084] For example, when executing "Task Step 2: Click the search box in the main interface of the X map application" in the "Take a taxi to the company" task of the above-mentioned automatic taxi-hailing scenario, a pop-up window appears in the current interface, but no pop-up window exists in the interface corresponding to the pre-recorded "Task Step 2: Click the search box in the main interface of the X map application", then it is considered that "Task Step 2: Click the search box in the main interface of the X map application" in the "Take a taxi to the company" task has an abnormality.

[0085] For example, when executing the "Task Step Three: Click the first record in the history record and select the recommended location "Company"" of the above-mentioned "Take a taxi to the company" task, there is no content in the history record, but the pre-recorded "Task Step Three: Click the first record in the history record and select the recommended location "Company"" corresponds to the company's address in the history record of the interface, then it is considered that the "Task Step Three: Click the first record in the history record and select the recommended location "Company"" of the "Take a taxi to the company" task has an abnormality.

[0086] For example, when executing "Task Step Two: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" of the "Lower Screen Brightness" task in the scenario of automatically lowering the screen brightness of the electronic device when the battery level of the electronic device is lower than the battery level threshold, there is no "Brightness Adjustment" control in the control center interface displayed on the screen, while the control center interface corresponding to the pre-recorded "Task Step Two: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" has a "Brightness Adjustment" control, then it is considered that "Task Step Two: Adjust the screen brightness of the electronic device to 40% through the "Brightness Adjustment" control" of the "Lower Screen Brightness" task has an abnormality.

[0087] When an execution exception occurs during the execution of a task step, a description of the execution exception corresponding to the task step can be generated. This description includes, but is not limited to, the exception type, exception description, the task to which it belongs, and the corresponding action number. Exception types include, but are not limited to, changes in the aforementioned interface content, the appearance of interfaces not displayed during recording, task execution response timeouts, restricted task execution permissions, abnormal positioning of interface elements, blocked interfaces, and application operation anomalies.

[0088] For example, for the above-mentioned "Task Step 3: Click the first record in the history and select the recommended location "Company"" where an execution exception occurs, the execution exception description information generated for this task step is as follows:

[0089] Exception type: Part of the interface content changes;

[0090] Exception description: When selecting a destination, the history record is empty and cannot be selected;

[0091] Task: Take a taxi to the company;

[0092] Mission number: 5.

[0093] In some possible implementations of the present application, when comparing the semantic similarity of the execution exception description information of the task step where the execution exception occurs with the registered reference exception description information, the belonging task in the execution exception description information of the task step where the execution exception occurs can be compared with the belonging task in the registered reference exception description information. When the belonging tasks are the same, the task number in the execution exception description information of the task step where the execution exception occurs is compared with the task number in the registered reference exception description information. When the task numbers are the same, the exception type in the execution exception description information of the task step where the execution exception occurs is compared with the exception type in the registered reference exception description information. When the exception types are the same, the semantic similarity of the exception description in the execution exception description information of the task step where the execution exception occurs is compared with the exception description in the registered reference exception description information. When the semantic similarity between the exception descriptions is greater than the similarity threshold, the task step where the execution exception occurs is executed according to the exception handling method corresponding to the reference exception description information whose semantic similarity is greater than the similarity threshold.

[0094] For example, when comparing the semantic similarity of the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" with the registered reference exception description information, the belonging task in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" can be compared with the belonging task in the registered reference exception description information. When the belonging tasks are the same, the task number in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" can be compared with the task number in the registered reference exception description information. When the task numbers are the same, the task number in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" can be compared with the task number in the registered reference exception description information. Step three: Click the first record in the history record, select the exception type in the execution exception description information of the recommended location "Company" and the exception type in the registered reference exception description information. When the exception types are the same, compare the semantic similarity of the exception description in "Task Step Three: Click the first record in the history record, select the recommended location "Company"" and the exception description in the registered reference exception description information. When the semantic similarity between the exception descriptions is greater than the similarity threshold, execute "Task Step Three: Click the first record in the history record, select the recommended location "Company"" according to the exception handling method corresponding to the reference exception description information whose semantic similarity is greater than the similarity threshold.

[0095] For example, the reference exception description information corresponding to “Task Step 3: Click the first record in the history record and select the recommended location “Company”” whose semantic similarity is greater than the similarity threshold is as follows:

[0096] Exception type: Part of the interface content has changed;

[0097] Exception description: When selecting a destination, the history record is empty and cannot be selected;

[0098] Task: Take a taxi to the company;

[0099] Mission number: 3;

[0100] Processing method:

[0101] action:

[0102] Name: Enter the destination;

[0103] Enter text:

[0104] Region: A rectangular region with coordinates (50, 100) as the upper left corner and coordinates (70, 150) as the lower right corner;

[0105] Content: "Somewhere".

[0106] When executing "Task Step Three: Click the first record in the history record and select the recommended location "Company"" based on the reference exception description information whose semantic similarity with the execution exception description information corresponding to "Task Step Three: Click the first record in the history record and select the recommended location "Company"" is greater than the similarity threshold, enter "somewhere" in the rectangular area with coordinates (50, 100) as the upper left corner and coordinates (70, 150) as the lower right corner.

[0107] In some possible implementations of the present application, when comparing the semantic similarity of the execution exception description information of the task step where the execution exception occurs with the registered reference exception description information, the belonging task in the execution exception description information of the task step where the execution exception occurs can be compared with the belonging task in the registered reference exception description information; when the belonging tasks are the same, the task number in the execution exception description information of the task step where the execution exception occurs is compared with the task number in the registered reference exception description information; when the task numbers are the same, the exception type and exception description in the execution exception description information of the task step where the execution exception occurs are vectorized to obtain a vector corresponding to the exception type and exception description in the execution exception description information of the task step where the execution exception occurs; the vector is compared with the vector corresponding to the exception type and exception description in the registered reference exception description information to obtain vector similarity, and the vector similarity is used as semantic similarity; when the semantic similarity between the vectors is greater than the similarity threshold, the task step where the execution exception occurs is executed according to the exception handling method corresponding to the reference exception description information whose vector similarity is greater than the similarity threshold.

[0108] For example, when comparing the semantic similarity of the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" with the registered reference exception description information, the belonging task in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" can be compared with the belonging task in the registered reference exception description information; when the belonging tasks are the same, the task number in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" is compared with the task number in the registered reference exception description information; when the task numbers are the same, the exception type in the execution exception description information of "Task Step Three: Click the first record in the history record and select the recommended location "Company"" is compared with the task number in the registered reference exception description information. The exception type "partial content of the interface changes" and the exception description "when selecting a destination, the historical record is empty and cannot be selected" are vectorized to obtain a vector corresponding to the exception type "partial content of the interface changes" and the exception description "when selecting a destination, the historical record is empty and cannot be selected" in the execution exception description information of "task step three: click the first record in the historical record and select the recommended place "company""; compare the vector with the vectors corresponding to the exception type and exception description in the registered reference exception description information to obtain vector similarity, and use the vector similarity as semantic similarity; when the semantic similarity between the vectors is greater than the similarity threshold, execute "task step three: click the first record in the historical record and select the recommended place "company"" according to the exception handling method corresponding to the reference exception description information whose vector similarity is greater than the similarity threshold.

[0109] In some possible implementations of the present application, when there is no reference exception description information whose semantic similarity with the exception description in the execution exception description information of the task step where the execution exception occurs is greater than a similarity threshold, and there is no reference exception description information whose vector similarity with the exception type and exception description in the execution exception description information of the task step where the execution exception occurs is greater than a similarity threshold, the belonging task, task number, exception type and exception description in the execution exception description information of the task step where the execution exception occurs can also be vectorized to obtain a vector corresponding to the belonging task, task number, exception type and exception description in the execution exception description information of the task step where the execution exception occurs; compare the vector with the vector corresponding to the belonging task, task number, exception type and exception description in the registered reference exception description information to obtain vector similarity, and use the vector similarity as semantic similarity; when the semantic similarity between the vectors is greater than the similarity threshold, execute the task step where the execution exception occurs according to the exception handling method corresponding to the reference exception description information whose vector similarity is greater than the similarity threshold.

[0110] In an embodiment of the present application, by executing the task step in which the exception occurred according to the exception handling method corresponding to the reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information of the task step in which the exception occurred is greater than a similarity threshold, exceptions occurring during task execution can be automatically resolved, thereby improving exception handling capabilities. Furthermore, by comparing the associated tasks, task numbers, and exception types, and calculating the semantic similarity of the exception descriptions, the efficiency and accuracy of exception matching can be guaranteed.

[0111] In some embodiments of the present application, the task processing method provided by the embodiments of the present application may also include: displaying a first prompt message when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold; wherein the first prompt message is used to prompt the user to manually operate on the execution exception that occurs when executing the i-th task step; receiving the user's exception control input; and in response to the exception control input, executing the i-th task step according to the exception handling method corresponding to the exception control input.

[0112] In some embodiments of the present application, the first prompt message in the embodiments of the present application may be that the automatic processing of the exception has failed, please process it manually.

[0113] For example, the first prompt information is explained by taking the above-mentioned task of "taking a taxi to the company" as an example. Figure 6 is a schematic diagram of an interface provided by some embodiments of the present application when no execution exception occurs. Figure 7 This is a schematic diagram of an interface provided by some embodiments of the present application when an execution exception occurs. Figure 6 This is the interface that appears when you click the search box in the main interface of the X Maps app when recording the "Take a taxi to the company" task; Figure 7 This is the interface displayed after executing "Task Step 2: Click the search box in the main interface of the X Map application" when executing the "Take a taxi to the company" task according to the task processing method provided in the embodiment of the present application.

[0114] After executing "Task Step 2: Click the search box in the main interface of the X map application" of the above-mentioned "Take a taxi to the company" task according to the task processing method provided by the embodiment of the present application, a pop-up window 701 appears in the interface. However, no pop-up window exists in the interface before "Task Step 3: Click the first record in the history record and select the recommended location "Company"" after the pre-recorded "Task Step 2: Click the search box in the main interface of the X map application". Therefore, it is considered that "Task Step 3: Click the first record in the history record and select the recommended location "Company"" of the "Take a taxi to the company" task has an abnormality.

[0115] When there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to "Task Step 3: Click the first record in the history record and select the recommended location "Company"" is greater than the similarity threshold, the following can be displayed: Figure 8 The first prompt information shown, Figure 8 This is a schematic diagram of the first prompt information provided by some embodiments of the present application. Figure 8 In the example, the first prompt message 801 is "Automatic processing of pop-up window exception failed, please manually process the pop-up window", where: Figure 8 Also displayed are an “OK” control 802 and a “Cancel” control 803. After the user clicks the “OK” control 802, he clicks the “Close” control 702 of the pop-up window 701 to close the pop-up window 701.

[0116] In an embodiment of the present application, when automatic exception handling fails, successful exception handling can be ensured by prompting the user to manually handle the exception.

[0117] In some embodiments of the present application, when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information of the task step where the execution exception occurred is greater than a similarity threshold, the exception handling methods corresponding to the first N reference exception description information, sorted from largest to smallest by semantic similarity, can be used to attempt to execute the task step where the execution exception occurred. N can be set according to actual needs, for example, N is 3. If the execution of the task step where the execution exception occurred fails using the exception handling methods corresponding to the first N reference exception description information, the first prompt message described above is displayed.

[0118] In some embodiments of the present application, when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information of the task step where the execution exception occurs is greater than a similarity threshold, the exception handling methods corresponding to the first N reference exception description information sorted in descending order of semantic similarity can be used to attempt to execute the task step where the execution exception occurs. When the execution of the task step where the execution exception occurs fails using the exception handling methods corresponding to the first N reference exception description information, the task step where the execution exception occurs can be executed according to the planned processing method using the method for planning the processing of the task step where the execution exception occurs in the related art. If the execution of the task step where the execution exception occurs fails according to the planned processing method, the above-mentioned first prompt message is displayed.

[0119] In some embodiments of the present application, after receiving the user's abnormal control input, the task processing method provided by the embodiment of the present application also includes: registering the execution abnormality description information corresponding to the i-th task step and the abnormality handling method corresponding to the abnormal control input.

[0120] In some embodiments of the present application, when registering the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input, the semantic similarity between the execution exception description information corresponding to the i-th task step and the registered reference exception description information is first calculated. If there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold, the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input are registered; if there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold, the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input are not registered.

[0121] Exemplarily, for the above-mentioned execution exception of "Task Step Three: Click on the first record in the history record and select the recommended location "Company"", when the registered reference exception description information does not contain reference exception description information whose semantic similarity with the execution exception description information of "Task Step Three: Click on the first record in the history record and select the recommended location "Company"" is greater than the similarity threshold, the user executes "Task Step Three: Click on the first record in the history record and select the recommended location "Company"" by manually entering the address. Based on the execution exception description information of "Task Step Three: Click on the first record in the history record and select the recommended location "Company"" and the user's manual entry of the address, the reference exception description information corresponding to "Task Step Three: Click on the first record in the history record and select the recommended location "Company"" is generated. The reference exception description information corresponding to "Task Step Three: Click on the first record in the history record and select the recommended location "Company"" is as follows:

[0122] Exception type: Part of the interface content has changed;

[0123] Exception description: When selecting a destination, the history record is empty and cannot be selected;

[0124] Task: Take a taxi to the company;

[0125] Mission number: 3;

[0126] Processing method:

[0127] action:

[0128] Name: Enter the destination;

[0129] Enter text:

[0130] Region: A rectangular region with coordinates (50, 100) as the upper left corner and coordinates (70, 150) as the lower right corner;

[0131] Content: "somewhere".

[0132] The semantic similarity between the reference exception description information corresponding to “Task Step Three: Click the first record in the history record and select the recommended location “Company”” and the registered reference exception description information is compared through the above-mentioned comparison method; when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the reference exception description information corresponding to “Task Step Three: Click the first record in the history record and select the recommended location “Company”” is greater than the similarity threshold, the reference exception description information corresponding to “Task Step Three: Click the first record in the history record and select the recommended location “Company”” is registered; if there is a reference exception description information in the registered reference exception description information whose semantic similarity with the reference exception description information corresponding to “Task Step Three: Click the first record in the history record and select the recommended location “Company”” is greater than the similarity threshold, the reference exception description information corresponding to “Task Step Three: Click the first record in the history record and select the recommended location “Company”” is not registered.

[0133] It should be noted that the similarity threshold used when comparing the semantic similarity of the exception description information and the similarity threshold used when registering the reference exception description information can be different similarity thresholds, wherein the similarity threshold used when comparing the semantic similarity of the exception description information is smaller than the similarity threshold used when registering the reference exception description information. For example, the similarity threshold used when comparing the semantic similarity of the exception description information is 85%, and the similarity threshold used when registering the reference exception description information is 95%.

[0134] In some embodiments of the present application, for user-submitted operations or tasks that have failed user confirmation, the data corresponding to these operations or tasks can be desensitized and provided as negative samples to the automated task execution model to optimize and train the model. This can improve the success rate of the automated task execution model in executing tasks.

[0135] In some embodiments of the present application, the electronic device in the embodiments of the present application may include an operation recording module, a step execution module, an exception perception module, an exception registration module, an exception matching module, a step planning module, a result judgment module, and a task recommendation module; wherein the operation recording module is used to record each task step of the task, the step execution module is used to automatically execute each task step of the task, the exception perception module is used to perceive whether an execution exception occurs in the task step, the exception registration module is used to register the execution exception description information corresponding to the task step with the execution exception and the exception handling method corresponding to the exception control input; the exception matching module is used to calculate the semantic similarity between the execution exception description information corresponding to the task step with the execution exception and the registered execution exception description information; the step planning module is used to plan the handling method of the task step with the execution exception, and execute the task step with the execution exception according to the planned handling method; the result judgment module is used to determine whether the task step is executed successfully; and the task recommendation module is used to prompt the user to record each task step of the task.

[0136] The task processing method provided in the embodiment of the present application can be executed by a task processing device. In the embodiment of the present application, the task processing device provided in the embodiment of the present application is described by taking the task processing method executed by the task processing device as an example.

[0137] Figure 9 Schematic diagram of the structure of a task processing device provided in an embodiment of the present application. The task processing device 900 may include:

[0138] The acquisition module 901 is configured to acquire task execution information of the task when the task meets the automatic execution condition; wherein the task execution information includes pre-recorded task step description information of at least one task step of executing the task;

[0139] The execution module 902 is used to execute the task according to the task execution information.

[0140] In some embodiments of the present application, the task execution information includes task step description information of N task steps of the task; N is a positive integer; the task processing device provided in the embodiment of the present application further includes:

[0141] A first generating module is configured to generate, when an execution exception occurs during the execution of the i-th task step, execution exception description information corresponding to the i-th task step; wherein the i-th task step is any one of N task steps, i≤N, and i is a positive integer;

[0142] The comparison module is used to perform semantic comparison between the execution exception description information corresponding to the i-th task step and the registered reference exception description information;

[0143] Execution module 902 is also used to execute the i-th task step according to the exception handling method corresponding to the reference exception description information with a semantic similarity greater than the similarity threshold when there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold.

[0144] In some embodiments of the present application, the execution exception includes at least one of the following:

[0145] Changes in interface content, appearance of an interface that was not displayed during the recording of the i-th task step, task execution response timeout, restricted task execution permissions, abnormal positioning of interface elements, obstruction of the interface, and abnormal application operation.

[0146] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0147] A first display module is configured to display a first prompt message when there is no reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold in the registered reference exception description information; wherein the first prompt message is configured to prompt a user to manually operate on the execution exception that occurs when executing the i-th task step;

[0148] A first receiving module is used to receive abnormal control input from the user;

[0149] The execution module 902 is further configured to respond to an abnormal control input and execute the i-th task step according to an abnormal handling method corresponding to the abnormal control input.

[0150] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0151] The registration module is used to register the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input after receiving the user's exception control input.

[0152] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0153] A second display module is configured to display a second prompt message after the task is executed according to the task execution information and when the task is completed; wherein the second prompt message is used to prompt the user to confirm whether the task is successfully executed; and when the user confirms through the second prompt message that the task has failed, display at least one task step image; wherein the task step image is a screenshot of the display screen display content during the execution of each task step of the task;

[0154] A second receiving module is configured to receive a selection input of one of the at least one task step images; wherein the task step image selected by the selection input corresponds to the first task step;

[0155] a recording module for re-recording, in response to a selection input, a second task step corresponding to the task step image selected by the selection input;

[0156] The replacement module is used to replace the pre-recorded first task step with the re-recorded second task step.

[0157] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0158] A third receiving module is used to receive recording control input;

[0159] a recording module for recording task step description information of each task step in the task execution process in response to a recording control input;

[0160] The second generating module is used to generate task execution information based on the task step description information of each task step.

[0161] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0162] The third display module is used to display a third prompt message before receiving the recording control input when the execution frequency of the task is greater than the frequency threshold; wherein the third prompt message is used to prompt the user to confirm whether to record the task steps of the task.

[0163] In some embodiments of the present application, the task processing device provided by the embodiment of the present application further includes:

[0164] The fourth display module is used to display fourth prompt information before obtaining the task execution information of the task; wherein the fourth prompt information is used to prompt the user to confirm whether to automatically execute the task;

[0165] The determination module is configured to determine whether the task satisfies the automatic execution condition upon receiving an input from a user confirming the automatic execution of the task.

[0166] In some embodiments of the present application, the task step description information includes at least one of the following:

[0167] The execution sequence number of the task step, the step name of the task step, the gesture feature information for executing the task step, the text input area, the input text content, the interface screenshot during the execution of the task step, the interface description information of the interface screenshot, and the device system information when executing the task step.

[0168] The task processing device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0169] The task processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0170] The task processing device provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented in the embodiment of the task processing method are not described here.

[0171] Alternatively, as Figure 10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be run on the processor 1001, and when the program or instruction is executed by the processor 1001, the various steps of the task processing method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application.

[0173] The electronic device 1100 includes but is not limited to components such as a radio frequency unit 1101 , a network module 1102 , an audio output unit 1103 , an input unit 1104 , a sensor 1105 , a display unit 1106 , a user input unit 1107 , an interface unit 1108 , a memory 1109 , and a processor 1110 .

[0174] Those skilled in the art will understand that the electronic device 1100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0175] The processor 1110 is configured to: obtain task execution information of the task when the task meets the automatic execution conditions; the task execution information includes pre-recorded task step description information of at least one task step of executing the task; and execute the task according to the task execution information.

[0176] In some embodiments of the present application, the task execution information includes task step description information of N task steps of the task; N is a positive integer; the processor 1110 is further configured to:

[0177] When an execution exception occurs during the execution of the i-th task step, the execution exception description information corresponding to the i-th task step is generated; wherein the i-th task step is any task step among N task steps, i≤N, and i is a positive integer;

[0178] Perform semantic comparison between the execution exception description information corresponding to the i-th task step and the registered reference exception description information;

[0179] When there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold, the i-th task step is executed according to the exception handling method corresponding to the reference exception description information whose semantic similarity is greater than the similarity threshold.

[0180] In some embodiments of the present application, the execution exception includes at least one of the following:

[0181] Changes in interface content, appearance of an interface that was not displayed during the recording of the i-th task step, task execution response timeout, restricted task execution permissions, abnormal positioning of interface elements, obstruction of the interface, and abnormal application operation.

[0182] In some embodiments of the present application, the display unit 1106 is configured to: display a first prompt message when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold; wherein the first prompt message is used to prompt the user to manually operate on the execution exception that occurs when executing the i-th task step;

[0183] The user input unit 1107 is used to: receive abnormal control input from the user;

[0184] The processor 1110 is further configured to: in response to an abnormal control input, execute the i-th task step according to an abnormal handling method corresponding to the abnormal control input.

[0185] In some embodiments of the present application, the processor 1110 is further configured to:

[0186] After receiving the user's exception control input, the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input are registered.

[0187] In some embodiments of the present application, the display unit 1106 is further configured to:

[0188] After executing the task according to the task execution information, if the task execution is completed, a second prompt information is displayed; wherein the second prompt information is used to prompt the user to confirm whether the task execution is successful; if the user confirms that the task execution fails through the second prompt information, at least one task step image is displayed; wherein the task step image is a screenshot image of the display screen display content during the execution of each task step of the task;

[0189] The user input unit 1107 is further configured to: receive a selection input of one of the at least one task step images; wherein the task step image selected by the selection input corresponds to the first task step;

[0190] The processor 1110 is further configured to: in response to a selection input, re-record a second task step corresponding to the task step image selected by the selection input; and a replacement module configured to replace the pre-recorded first task step with the re-recorded second task step.

[0191] In some embodiments of the present application, the user input unit 1107 is further configured to: receive recording control input;

[0192] The processor 1110 is further configured to: record task step description information of each task step in the task execution process in response to a recording control input; and generate task execution information based on the task step description information of each task step.

[0193] In some embodiments of the present application, the display unit 1106 is also used to: before receiving the recording control input, when the execution frequency of the task is greater than the frequency threshold, display a third prompt message; wherein the third prompt message is used to prompt the user to confirm whether to record the task steps of the task.

[0194] In some embodiments of the present application, the display unit 1106 is further configured to: display fourth prompt information before obtaining task execution information of the task; wherein the fourth prompt information is used to prompt the user to confirm whether to automatically execute the task;

[0195] The processor 1110 is further configured to: upon receiving an input from a user confirming automatic execution of a task, determine whether the task satisfies an automatic execution condition.

[0196] In some embodiments of the present application, the task step description information includes at least one of the following:

[0197] The execution sequence number of the task step, the step name of the task step, the gesture feature information for executing the task step, the text input area, the input text content, the interface screenshot during the execution of the task step, the interface description information of the interface screenshot, and the device system information when executing the task step.

[0198] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0199] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data; wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0200] Processor 1110 may include one or more processing units. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0201] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the task processing method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0202] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, and examples of computer-readable storage media include non-transitory computer-readable media, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0203] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the task processing method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0204] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0205] An embodiment of the present application also provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the task processing method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0206] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0207] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0208] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A task processing method, characterized in that: The method comprises: When the task satisfies the automatic execution condition, task execution information of the task is obtained; wherein the task execution information includes pre-recorded task step description information of at least one task step for executing the task; The task is executed according to the task execution information.

2. The method according to claim 1, characterized in that The task execution information includes task step description information of N task steps of the task; N is a positive integer; the method further comprises: When an execution exception occurs during the execution of the i-th task step, generating execution exception description information corresponding to the i-th task step; wherein the i-th task step is any task step among the N task steps, i≤N, and i is a positive integer; Performing a semantic comparison between the execution exception description information corresponding to the i-th task step and the registered reference exception description information; When there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold, the i-th task step is executed according to the exception handling method corresponding to the reference exception description information whose semantic similarity is greater than the similarity threshold.

3. The method according to claim 2, characterized in that The execution exception includes at least one of the following: changes in interface content, appearance of an interface that was not displayed during the recording of the i-th task step, task execution response timeout, restricted task execution authority, abnormal interface element positioning, blocked interface, and abnormal application operation.

4. The method according to claim 2, characterized in that The method further comprises: If there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold, displaying a first prompt information; wherein the first prompt information is used to prompt the user to manually operate on the execution exception that occurs when executing the i-th task step; Receive abnormal control input from the user; In response to the abnormal control input, the i-th task step is executed according to the abnormal handling method corresponding to the abnormal control input.

5. The method according to claim 4, characterized in that After receiving the abnormal control input from the user, the method further includes: Register the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input.

6. The method according to any one of claims 1 to 5, characterized in that After executing the task according to the task execution information, the method further includes: When the task is completed, a second prompt message is displayed; wherein the second prompt message is used to prompt the user to confirm whether the task is successfully executed; When the user confirms through the second prompt information that the task execution has failed, displaying at least one task step image; wherein the task step image is a screenshot of the display screen display content during the execution of each task step of the task; receiving a selection input of one of the at least one task step images; wherein the task step image selected by the selection input corresponds to a first task step; In response to the selection input, re-recording a second task step corresponding to the task step image selected by the selection input; The pre-recorded first task step is replaced by the re-recorded second task step.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Receive recording control input; In response to the recording control input, recording task step description information of each task step in the process of executing the task; Based on the task step description information of each task step, task execution information is generated.

8. The method according to claim 7, characterized in that The method further comprises: Before receiving the recording control input, if the execution frequency of the task is greater than the frequency threshold, a third prompt message is displayed; wherein the third prompt message is used to prompt the user to confirm whether to record the task steps of the task.

9. The method according to any one of claims 1 to 5, characterized in that Before obtaining the task execution information of the task, the method further includes: Displaying a fourth prompt message; wherein the fourth prompt message is used to prompt the user to confirm whether to automatically execute the task; In a case where an input confirming automatic execution of the task is received from the user, it is determined that the task meets the automatic execution condition.

10. The method according to any one of claims 1 to 5, characterized in that The task step description information includes at least one of the following: the execution sequence number of the task step, the step name of the task step, the gesture feature information of executing the task step, the text input area, the input text content, the interface screenshot during the execution of the task step, the interface description information of the interface screenshot, and the device system information when executing the task step.

11. A task processing device, characterized in that: The device comprises: An acquisition module, configured to acquire task execution information of the task when the task satisfies an automatic execution condition; wherein the task execution information includes pre-recorded task step description information of at least one task step for executing the task; An execution module is used to execute the task according to the task execution information.

12. The device according to claim 11, characterized in that The task execution information includes task step description information of N task steps of the task, where N is a positive integer; the device further includes: A first generating module is configured to generate, when an execution exception occurs during the execution of an i-th task step, execution exception description information corresponding to the i-th task step; wherein the i-th task step is any one of the N task steps, i≤N, and i is a positive integer; A comparison module, configured to perform a semantic comparison between the execution exception description information corresponding to the i-th task step and the registered reference exception description information; The execution module is also used to execute the i-th task step according to the exception handling method corresponding to the reference exception description information with a semantic similarity greater than the similarity threshold when there is reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than the similarity threshold.

13. The device according to claim 12, characterized in that The execution exception includes at least one of the following: Changes in interface content, appearance of an interface that was not displayed during the recording of the i-th task step, task execution response timeout, restricted task execution permissions, abnormal positioning of interface elements, obstruction of the interface, and abnormal operation of the application.

14. The device according to claim 12, characterized in that The device further comprises: A first display module is configured to display a first prompt message when there is no reference exception description information in the registered reference exception description information whose semantic similarity with the execution exception description information corresponding to the i-th task step is greater than a similarity threshold; wherein the first prompt message is used to prompt a user to manually operate on the execution exception that occurs when executing the i-th task step; A first receiving module is used to receive abnormal control input from the user; The execution module is further configured to respond to the abnormal control input and execute the i-th task step according to the abnormal handling method corresponding to the abnormal control input.

15. The device according to claim 14, characterized in that The device further comprises: The registration module is used to register the execution exception description information corresponding to the i-th task step and the exception handling method corresponding to the exception control input after receiving the user's exception control input.

16. The device according to any one of claims 11 to 15, characterized in that The device further comprises: a second display module configured to, after executing the task in accordance with the task execution information, display a second prompt message when the task execution is completed; wherein the second prompt message is used to prompt the user to confirm whether the task execution is successful; and, if the user confirms through the second prompt message that the task execution fails, display at least one task step image; wherein the task step image is a screenshot image of the display screen display content during the execution of each task step of the task; a second receiving module configured to receive a selection input of one of the at least one task step image; wherein the task step image selected by the selection input corresponds to a first task step; a recording module, configured to, in response to the selection input, re-record a second task step corresponding to the task step image selected by the selection input; A replacement module is used to replace the pre-recorded first task steps with the re-recorded second task steps.

17. The device according to any one of claims 11 to 15, characterized in that The device further comprises: A third receiving module is used to receive recording control input; a recording module for recording task step description information of each task step in the process of executing the task in response to the recording control input; The second generating module is configured to generate task execution information based on the task step description information of each task step.

18. The device according to claim 17, characterized in that The device further comprises: The third display module is used to display a third prompt message before receiving the recording control input when the execution frequency of the task is greater than the frequency threshold; wherein the third prompt message is used to prompt the user to confirm whether to record the task steps of the task.

19. The device according to any one of claims 11 to 15, characterized in that The device further comprises: a fourth display module, configured to display fourth prompt information before acquiring the task execution information of the task; wherein the fourth prompt information is used to prompt the user to confirm whether to automatically execute the task; The determination module is configured to determine whether the task satisfies an automatic execution condition upon receiving an input from a user confirming automatic execution of the task.

20. The device according to any one of claims 11 to 15, characterized in that The task step description information includes at least one of the following: The execution sequence number of the task step, the step name of the task step, the gesture feature information for executing the task step, the text input area, the input text content, the interface screenshot during the execution of the task step, the interface description information of the interface screenshot, and the device system information when executing the task step.

21. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the task processing method according to any one of claims 1 to 10 are implemented.

Images