Task execution method and device

Abstract

The invention discloses a task execution method and device, and belongs to the technical field of electronics. The method comprises the following steps: receiving task information input by a user, wherein the task information input by the user is used for describing a first task; obtaining first action sequence information; executing a first task based on the task information input by the user and the first action sequence information; wherein the first action sequence information corresponds to a video selected by the user, the video is a screen recording video when the electronic equipment executes the second task, and the first action sequence information is used for describing an action sequence when the electronic equipment executes the second task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of electronics, and particularly relates to a task execution method and device. BACKGROUND

[0002] With the development of technology, smart electronic devices such as mobile phones provide more and more functions, such as seat booking, ticket purchase, meal ordering, etc. In order to improve the efficiency of life, users can control the electronic device to automatically execute the corresponding task through the automatic software in the electronic device, for example, automatically ordering a concert ticket through the ticket purchase software in the electronic device.

[0003] In the related art, when a user uses an electronic device to perform a certain task, the electronic device can listen to the user's operation through the automatic software, and identify and record the operation information such as the user's operation position, operation sequence and operation content. Subsequently, the electronic device can automatically execute the task based on the recorded operation information.

[0004] However, based on the above method, since the electronic device executes the task through the automatic software, it strictly executes the corresponding task according to the previously recorded operation information, so the electronic device can only automatically execute the previously executed task, resulting in that the electronic device is not flexible in automatically executing the task. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a task execution method and device, which can improve the flexibility of the electronic device in automatically executing the task.

[0006] In a first aspect, the embodiments of the present application provide a task execution method, which comprises: receiving task information input by a user, the task information being used to describe a first task; obtaining first action sequence information; and executing the first task based on the task information and the first action sequence information, wherein the first action sequence information corresponds to a video selected by the user, the video being a screen recording video when an electronic device executes a second task, and the first action sequence information being used to describe an action sequence of the electronic device executing the second task.

[0007] In a second aspect, the embodiments of the present application provide a task execution device, which comprises a receiving module, an obtaining module and an executing module. The receiving module is configured to receive task information input by a user, the task information being used to describe a first task. The obtaining module is configured to obtain first action sequence information. The executing module is configured to execute the first task based on the task information received by the receiving module and the first action sequence information obtained by the obtaining module. Wherein the first action sequence information corresponds to a video selected by the user, the video being a screen recording video when an electronic device executes a second task, and the first action sequence information being used to describe an action sequence of the electronic device executing the second task.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, the device can receive task information input by the user to describe a first task, obtain first action sequence information corresponding to the video selected by the user, and then execute the first task based on the task information and the first action sequence information. The video is a screen recording of the electronic device performing a second task, and the first action sequence information describes the action sequence of the electronic device performing the second task. In this solution, after receiving the task information input by the user to describe the first task, the electronic device combines this task information with the action sequence information corresponding to the screen recording video selected by the user. This allows the action sequence information to be adapted to the task requirements of the first task, enabling the electronic device to automatically execute the first task. Thus, the electronic device can automatically execute other tasks based on the action sequence information of a completed task, improving the flexibility of the electronic device in automatically executing tasks. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of the task execution method provided in the embodiments of this application;

[0014] Figure 2 This is the second flowchart of the task execution method provided in the embodiments of this application;

[0015] Figure 3 This is the third flowchart of the task execution method provided in the embodiments of this application;

[0016] Figure 4 This is the fourth flowchart of the task execution method provided in the embodiments of this application;

[0017] Figure 5 FIG. 5 is a flowchart of a task execution method according to an embodiment of the present application;

[0018] Figure 6 FIG. 6 is a flowchart of a task execution method according to an embodiment of the present application;

[0019] Figure 7 FIG. 7 is an example diagram of a first interface according to an embodiment of the present application;

[0020] Figure 8 FIG. 8 is an example diagram of a second interface according to an embodiment of the present application;

[0021] Figure 9 FIG. 9 is an example diagram of a fourth interface according to an embodiment of the present application;

[0022] Figure 10 FIG. 10 is an example diagram of a fifth interface according to an embodiment of the present application;

[0023] Figure 11 FIG. 11 is an example diagram of a display teaching square interface according to an embodiment of the present application;

[0024] Figure 12 FIG. 12 is an example diagram of a third interface according to an embodiment of the present application;

[0025] Figure 13 FIG. 13 is an example diagram of a third interface according to an embodiment of the present application;

[0026] Figure 14 FIG. 14 is a schematic diagram of a task execution device according to an embodiment of the present application;

[0027] Figure 15 FIG. 15 is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0028] Figure 16 FIG. 16 is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0031] The term "at least one (item)", "at least one of" and the like in the present application means any one, any two or a combination of two or more of the objects. For example, at least one (item) of a, b and c can mean "a", "b", "c", "a and b", "a and c", "b and c", and "a, b and c", where a, b and c can be single or multiple. Similarly, "at least two (items)" means two or more, and the meaning expressed is similar to "at least one (item)".

[0032] The task execution method and device provided by the embodiments of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios.

[0033] The embodiments of the present application can be applied to the scenario of automatic execution of tasks by electronic devices.

[0034] The task execution method provided by the embodiments of the present application will be described exemplarily below taking some specific scenarios of the embodiments of the present application as examples.

[0035] Scenario 1, online automatic ticket purchase scenario: the user can trigger the mobile phone to automatically complete the purchase of movie tickets.

[0036] Scenario 2, intelligent home automatic control scenario: the user goes out to work during the day, and can trigger the mobile phone to automatically complete the starting task of the intelligent home after coming home, such as opening the curtain, turning on the air conditioner, etc.

[0037] Scenario 3, online automatic appointment and registration scenario: the user has a chronic disease and needs to go to the hospital for registration and treatment for a long time, and can trigger the mobile phone to automatically complete the registration task, so as to facilitate the user to timely make an appointment with a doctor.

[0038] Scenario 4, online automatic shopping scenario: the user often purchases materials needed for work online, and can trigger the mobile phone to automatically complete the purchase task of the materials, including adding the goods to the shopping cart, filling in the address, etc.

[0039] It should be noted that the above scenarios 1 to 4 are only exemplary to list some scenarios in which the embodiments of the present application can be applied, and in actual implementation, the embodiments of the present application can also be applied to more scenarios of automatically performing tasks by electronic devices, which are not limited herein.

[0040] The embodiments of the present application provide a task performing method. After receiving task information input by a user to describe a first task, an electronic device combines the task information with action sequence information corresponding to a screen recording video selected by the user, so that the action sequence information can be adapted to the task requirement of the first task, thereby enabling the electronic device to automatically perform the first task. In this way, the electronic device can automatically perform other tasks based on the action sequence information of a completed task, thereby improving the flexibility of the electronic device in automatically performing tasks.

[0041] The execution subject of the task performing method provided by the embodiments of the present application can be a task performing device, which can be an electronic device, or a functional module or functional entity in the electronic device. The technical solutions provided by the embodiments of the present application are described below by taking an electronic device as an example.

[0042] Figure 1 A flowchart of a task performing method provided by the embodiments of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the task performing method provided by the embodiments of the present application can include the following steps 201 to 203.

[0043] Step 201: An electronic device receives task information input by a user.

[0044] In some embodiments of the present application, the task information input by the user is used to describe a first task.

[0045] It can be understood that the task information input by the user describes a specific task that the user wants to complete.

[0046] In some embodiments of the present application, the task information input by the user can include, but is not limited to, the following contents: specific content of the task, execution time of the task, parameters of the task, etc.

[0047] In some embodiments of the present application, the task information input by the user can include, but is not limited to, the following contents: text, image, audio, video, etc.

[0048] Exemplarily, the task information input by the user can include a description text of the first task. The description text is used to describe the first task. For example, in combination with scenario 1, the description text can be “buy two movie tickets for next Wednesday”.

[0049] In some embodiments of the present application, the first task is a task that the user needs the electronic device to automatically perform. For example, if the task information input by the user is "buy two movie tickets for next Wednesday", the first task is to buy two movie tickets for next Wednesday.

[0050] In some embodiments of the present application, the user can input the task information input by the user in the electronic device in various ways, including but not limited to text input, voice input, etc. For example, the user can input the text "buy two movie tickets for next Wednesday" or speak the same content through voice.

[0051] In some embodiments of the present application, if the user's input is text, the electronic device can directly obtain the text input by the user; if the user's input is voice, the electronic device can use voice recognition technology, such as a voice recognition model, to convert the voice into text.

[0052] Step 202, the electronic device obtains first action sequence information.

[0053] In some embodiments of the present application, the first action sequence information corresponds to a video selected by the user, the video being a screen recording video when the electronic device performs the second task, and the first action sequence information being used to describe the action sequence of the electronic device performing the second task.

[0054] In some embodiments of the present application, the first action sequence information refers to information corresponding to the video selected by the user and describing the action sequence of the electronic device performing the second task, which records detailed operation steps of the electronic device when performing the second task, such as the sequence of clicking, inputting, sliding, etc.

[0055] In some embodiments of the present application, the first action sequence information includes at least one first action information. Each first action information in the at least one first action information corresponds to a specific operation step of the electronic device performing the second task. In other words, the at least one first action information refers to information describing each specific operation step of the electronic device performing the second task.

[0056] For example, each first action information in the first action sequence information records operation information of a specific operation, which can include operation object, operation type, and operation parameter, etc.

[0057] For example, the form of the first action sequence information can be A orig = [a1, a2, …, a m ], where A orig represents the original action sequence, that is, the first action sequence information, and [a1, a2, …, a mis a sequence, which contains a series of action information, each action information, such as a1, a2, etc., represents a specific operation step of the user when performing the second task.

[0058] For example, if the second task is "buy a movie ticket for movie a on Saturday this week", then A orig may contain the following action information:

[0059] a1: open the ticket purchase application.

[0060] a2: select movie a.

[0061] a3: select Saturday this week as the date.

[0062] a4: select one as the number.

[0063] a5: click the "buy" button.

[0064] In some embodiments of the present application, the above-mentioned second task refers to the task actually performed by the electronic device in the screen recording video. It should be noted that the above-mentioned second task and the first task can be similar or related tasks, but the first task can be a task that the user wants to perform under different parameters or conditions, such as different dates, quantities, etc. For example, the second task and the first task are both ticket purchase tasks for movie a, but the first task is a ticket purchase task for movie a at 12:00 on August 7, and the second task is a ticket purchase task for movie a at 14:00 on August 6, and the only difference between the two is the time of the show.

[0065] In some embodiments of the present application, the above-mentioned screen recording video refers to a video that records the operation process of the user on the electronic device, which can completely show all operation steps of the user when performing a certain task on the electronic device, such as clicking, sliding, inputting, etc. Action and corresponding interface changes.

[0066] In some embodiments of the present application, the source of the above-mentioned screen recording video can be diverse, which can be an operation video recorded by the user through the screen recording function of the electronic device, an operation video recorded by the third-party screen recording software in the electronic device, or a recording video obtained by aiming the camera of one electronic device at the screen of another electronic device and recording the operation process of the user operating the other electronic device.

[0067] In some embodiments of the present application, the above-mentioned video selected by the user, also known as the demonstration video, is a screen recording video selected by the user, which records the operation process of the electronic device performing the second task. For example, the demonstration video shows the complete process of the user operating the phone to buy a movie ticket on Saturday this week.

[0068] In some embodiments of this application, the video selected by the user can be a video of the user operating an electronic device, recorded by the user themselves; or, the video selected by the user can be a video selected by the user in the teaching plaza interface. The teaching plaza interface is an interface that displays screen recording videos shared by other users, from which users can select screen recording videos relevant to their needs.

[0069] Optionally, in some embodiments of this application, the aforementioned first action sequence information includes at least one first action information. Combined with... Figure 1 ,like Figure 2 As shown, step 202 above can be achieved through steps 301 and 302 below.

[0070] Step 301: The electronic device acquires a keyframe sequence from the video.

[0071] In some embodiments of this application, the aforementioned keyframe sequence is a series of representative frame images extracted from a video, used to characterize key changes during the operation process. This sequence contains only dissimilar image frames, which can be considered as page changes occurring due to user actions.

[0072] In some embodiments of this application, the electronic device can perform frame sampling on the video, that is, extract video frame images from the video at a preset frequency (such as 5 frames per second) to obtain a pre-processed frame sequence.

[0073] In some embodiments of this application, the electronic device can perform similarity calculations on the pre-processed frame sequence to remove duplicates and obtain a non-repeating keyframe sequence.

[0074] In some embodiments of this application, the electronic device may include the following process when acquiring a keyframe sequence from a video:

[0075] First, electronic devices can use an object detection model and an OCR model (such as the Tesseract-OCR method) to detect objects in the pre-processed frame sequence. The object detection model can detect dynamic regions in the image, i.e., regions that may change, such as video playback areas, dynamic advertising areas, scrolling playback areas, etc. The OCR model can extract all text and its coordinates from the image.

[0076] Secondly, the electronic device can remove the dynamic region in each frame and calculate the pixel similarity in the non-dynamic region, for example, by the Structural Similarity Index Measure (SSIM) method, for any two adjacent frames in the continuous video frame images, and can obtain the text in the non-dynamic region in the two frames, and then calculate the text similarity, for example, using the Jaccard coefficient method.

[0077] Furthermore, the electronic device can weight the calculated pixel similarity and text similarity to obtain a weighted similarity, for example, the pixel similarity weight can be 0.6 and the text similarity weight can be 0.4. If the weighted similarity is higher than a preset threshold, for example, 0.85, it can be confirmed that the two frames are repeated, and the electronic device can delete one of them, for example, delete the latter one of the two adjacent frames, and then take the remaining one frame and the new adjacent frame as the new two adjacent frames for similarity calculation to delete the repeated frame.

[0078] Finally, the electronic device can repeat the above process to continuously delete the repeated frames, and so on, until the first frame to the last frame of the video is traversed to obtain the key frame sequence after deduplication.

[0079] The process of obtaining the key frame sequence will be described in detail below.

[0080] 1. Frame sampling:

[0081] The original frames are extracted at a frequency of 5fps to obtain a frame sequence F = {f1, f2, …, f n}, where F is the preliminary processed frame sequence, f1, f2, …, f is each frame image in the preliminary processed frame sequence.

[0082] 2. Dynamic region mask generation:

[0083] A target detection model, for example, YOLOv8, is used to identify the dynamic region to obtain M dynamic (f i ) = YOLOv8(f i ), where M dynamic (f i ) represents the mask generated after dynamic region detection of the i-th frame image f i In the mask, the position of the dynamic region is marked as a specific value, for example, 1 or 255, while the non-dynamic region is marked as other values (for example, 0). This mask is used in subsequent steps to distinguish between dynamic and non-dynamic regions for similarity calculation of the non-dynamic region. YOLOv8(f i ) represents the use of the YOLOv8 model to identify the dynamic region of the i-th frame image f iThe process of target detection.

[0084] Through the OCR model, for example, Tesseract-OCR, the text coordinates are extracted, and T(f i ) is obtained. k , x k , y k , w k , h k ), where T(f i ) represents the result of text detection and recognition on the i-th frame image f i , text k represents the text content recognized in the k-th text region, x k represents the starting horizontal coordinate of the k-th text region, y k represents the starting vertical coordinate of the k-th text region, w k represents the width of the k-th text region, and h k represents the height of the k-th text region.

[0085] 3. Similarity calculation

[0086] (1) Taking the SSIM method as an example, the pixel similarity of the non-dynamic region is calculated, and formula one is obtained:

[0087]

[0088] Where SSIM(f i , f j ) is the pixel similarity of two image frames f i and f j , μ i and μ j represent the average gray value of images f i and f j , i.e. the average brightness. and respectively represent the variance of images f i and f j , reflecting the contrast information of the image. σ ij represents the covariance of images f i and f j , reflecting the structural similarity information between the two images. C1 and C1 are two small constants, used to stabilize the division operation and avoid the case that the denominator or numerator is zero.

[0089] (2) Taking the Jaccard coefficient method as an example, the text similarity of the non-dynamic region is calculated, and formula two is obtained:

[0090]

[0091] where T i represents the set of text regions extracted from the image f i . T j represents the set of text regions extracted from the image f j . T i ∩T j represents the intersection of T i and T j , i.e., the text regions that appear in both T i and T j . T i ∪T j represents the union of T i and T j , i.e., all the text regions that appear in either T i or T j . |T i ∩T j | represents the size of the intersection, i.e., the number of text regions that appear in both sets. |T i ∪T j | represents the size of the union, i.e., the total number of text regions that appear in either set. Jaccard(T i , T j ) is the text similarity between the texts T i and T j extracted from the two image frames f i and f j respectively, with a value range between 0 and 1, where 1 means the two sets are identical and 0 means the two sets have no common elements.

[0092] (3) Calculate the weighted similarity, which is obtained by formula three:

[0093] S total = α·SSIM + β·Jaccard (Formula 3)

[0094] where S total is the weighted similarity, with a value range between 0 and 1, and the value closer to 1 means the two keyframes are more similar. α is the weight of the pixel similarity and β is the weight of the text similarity.

[0095] 4. Keyframe screening

[0096] Take deleting the latter one of two adjacent frames as an example. If S total > 0.85, then f j is deleted; otherwise, it is retained.

[0097] Step 302, the electronic device analyzes each two adjacent keyframes in the keyframe sequence to obtain at least one first action information.

[0098] In some embodiments of the present application, the electronic device can identify each two adjacent key frames in the key frame sequence to obtain the action description between the two key frames, i.e., the action performed by the user causing the change.

[0099] In some embodiments of the present application, the electronic device can horizontally splice the two adjacent key frames and input the multi-modal large model.

[0100] The prompt word is exemplarily illustrated below with specific examples:

[0101] prompt="

[0102] Task: You need to generate an operation description based on the continuous two screenshots of the action before and after the action.

[0103] Requirements:

[0104] 1. Description format: <in [interface name] interface, perform [action type], operation object is [control description]>

[0105] 2. Interface name: inferred from the top text / icon (such as application name, settings)

[0106] 3. Action type: click, input text, swipe, long press, etc.

[0107] 4. Control description: position + text (such as "right upper corner search bar")

[0108] 5. Exclude interface-independent changes (such as advertisement carousel)

[0109] Example input image: [horizontally spliced key frame f i and f i+1 ]

[0110] Example output description: [in the main interface of the ticket purchase application, click the search icon in the upper right corner]

[0111] "

[0112] In some embodiments of the present application, the electronic device can sequentially identify each two adjacent key frames in the key frame sequence to obtain at least one first action information. For example, the obtained at least one first action information can be: opening the ticket purchase application, selecting the amusement park ticket, selecting Friday of this week as the date, selecting 1 as the quantity, and clicking the purchase button.

[0113] In this way, the electronic device can accurately identify the key steps of user operation by extracting the key frame sequence from the video and analyzing the changes between adjacent key frames, generate detailed first action information, provide accurate action sequence for subsequent automated task execution, and ensure the accuracy of task execution.

[0114] It should be noted that the above step 202 can be performed after the above step 201, that is, the electronic device can first receive the task information input by the user, and then obtain the first action sequence information corresponding to the video selected by the user. Alternatively, the above step 202 can be performed before the above step 201, that is, the electronic device can first obtain the first action sequence information corresponding to the video selected by the user, and then receive the task information input by the user.

[0115] Step 203, the electronic device performs the first task based on the task information input by the user and the first action sequence information.

[0116] In some embodiments of the present application, the electronic device can generate the action sequence information of the first task based on the task information input by the user and the first action sequence information, that is, the second action sequence information described below, and then execute this new action sequence information to perform the first task.

[0117] In this way, by combining the action sequence information of the second task and the character information, the electronic device can generate the action sequence information of the first task, thereby realizing the automatic execution of the first task.

[0118] In some embodiments of the present application, the above second action sequence information includes at least one second action information. Each of the at least one second action information corresponds to a specific operation step of the electronic device when performing the first task. In other words, the at least one second action information refers to information describing each specific operation step of the electronic device when performing the first task.

[0119] For example, each second action information in the above second action sequence information records operation information of a specific operation, which can include operation object, operation type and operation parameter, etc.

[0120] Optionally, in some embodiments of the present application, in combination with Figure 1 As shown in the above step 202, the electronic device can first obtain the first action sequence information corresponding to the video selected by the user, and then receive the task information input by the user. Figure 3 The above step 203 can be implemented by the following steps 203a and 203b.

[0121] Step 203a, the electronic device generates second action sequence information based on the task information input by the user and the first action sequence information.

[0122] In some embodiments of the present application, the above second action sequence information is used to describe the action sequence of the electronic device performing the first task.

[0123] In some embodiments of the present application, the electronic device can update the first action sequence information based on the task information input by the user to obtain the second action sequence information.

[0124] In some embodiments of the present application, the electronic device can determine the first action information corresponding to the task information input by the user according to the first action sequence information. Figure 3 As shown in FIG. 3, the first action sequence information includes at least one first action information. Figure 4 The first action sequence information includes at least one first action information. The step 203a can be implemented by the following steps 203a1-203a3.

[0125] In step 203a1, the electronic device determines the first action information corresponding to the task information input by the user from the at least one first action information.

[0126] In some embodiments of the present application, the electronic device can match the task information input by the user with each first action information to determine the action information corresponding to the task information input by the user.

[0127] In some embodiments of the present application, the action information corresponding to the task information input by the user refers to the action information in the first action sequence information that matches the operation step or parameter in the task information input by the user and describes the specific operation step related to the task information input by the user.

[0128] In some embodiments of the present application, the action information corresponding to the task information input by the user can be one first action information. For example, the action information corresponding to the task information input by the user “buy tickets for July 19” can be “select the date as July 19”. Alternatively, the action information corresponding to the task information input by the user can be at least two first action information. For example, the action information corresponding to the task information input by the user “buy two movie tickets next Wednesday” can be “select the number of movie tickets as two” and “select the date as next Wednesday”.

[0129] In step 203a2, the electronic device generates second action information based on the task information input by the user and the first action information.

[0130] In some embodiments of the present application, the second action information is the action description generated according to the task information input by the user and the first action information.

[0131] In some embodiments of the present application, the electronic device can analyze the task information input by the user, extract key elements such as time, quantity, and content, and then combine these key elements with the first action information to generate new second action information.

[0132] For example, assuming that the task information input by the user is "buy two movie tickets for next Wednesday", and the first action information corresponding to the task information is "select a date of this Friday" and "select a number of movie tickets of one", the electronic device can extract the time "next Wednesday" and the number "two" in the task information, and then combine the first action information to obtain the second action information "select a date of next Wednesday" and "select a number of movie tickets of two".

[0133] In step 203a3, the electronic device updates the first action information in the first action sequence information as the second action information to obtain second action sequence information.

[0134] In the embodiments of the present application, the electronic device can insert the second action information into the position of the first action information in the first action sequence information, replace the original first action information, and thus obtain the second action sequence.

[0135] For example, assuming that the first action sequence information is [open a ticket purchase application, select a movie ticket, select a date of this Friday, select a number of movie tickets of one, click a purchase button], the first action information is "select a date of this Friday" and "select a number of movie tickets of one", and the second action information is "select a date of next Wednesday" and "select a number of movie tickets of two", the electronic device can replace the first action information in the corresponding position in the first action sequence information with each second action information to obtain the second action sequence information [open a ticket purchase application, select a movie ticket, select a date of next Wednesday, select a number of movie tickets of two, click a purchase button].

[0136] In some embodiments of the present application, the electronic device can update the action information associated with the task information input by the user through the playback analysis module to obtain action information matching the task information input by the user. For example, the original action information and the new task instruction can be input into the large language model to output new action information.

[0137] In some embodiments of the present application, the electronic device can construct a prompt word (Prompt) through the original action information and the new task instruction, and then input the prompt word into the large language model, so that the large language model can generate new action information.

[0138] The following will take specific examples to illustrate the prompt word input by the electronic device into the large language model:

[0139] For example, assuming that the original action sequence A orig =[open a ticket purchase application, select a movie ticket, select a date of this Friday, select a number of movie tickets of one, click a purchase button], and the user input task instruction Q new"Book 2 amusement park tickets for next Wednesday." The prompt word input by the electronic device to the large language model can include:

[0140] "System role: You are a mobile phone operation flow editor, and need to modify the original action sequence according to the new instruction.

[0141] Original action sequence: A orig = [Open ticket purchase application, select amusement park ticket, select date as this Friday, select quantity as 1, click purchase button]

[0142] User instruction: Q new "Book 2 amusement park tickets for next Wednesday."

[0143] Operation rules:

[0144] 1. Time reasoning: Convert date words (such as "next Wednesday") into specific dates (need to calculate the current date offset)

[0145] 2. Quantity replacement: Update quantity-related parameters (such as ticket quantity)

[0146] Output: Updated action sequence (format same as input)"

[0147] Then, the electronic device can input the prompt word to the large language model, so as to obtain the updated action sequence A new = [Open ticket purchase application, select amusement park ticket, select date as next Wednesday, select quantity as 2, click purchase button].

[0148] In some embodiments of the present application, the electronic device can directly input the prompt word to the built-in large language model to obtain the second action sequence information, or the electronic device can also send the prompt word to the cloud server to obtain the second action sequence information through the large language model of the cloud server.

[0149] In some embodiments of the present application, the cloud server, also known as the cloud, is a high-performance computing resource located in a remote data center, which can provide powerful computing and data processing capabilities. The electronic device directly running the large language model can quickly respond to user needs, suitable for tasks with high real-time requirements, and reduce cloud communication costs. While using the cloud to run the large language model, the electronic device can take advantage of the powerful computing power of the cloud to handle complex tasks, suitable for scenarios where the electronic device has low computing power resources or the task complexity is high. The specific way to run the large language model can be determined according to actual use requirements, and the embodiments of the present application are not limited.

[0150] In this way, the electronic device improves the flexibility and accuracy of the automatic task execution by accurately matching and updating the action information, so as to dynamically adjust the operation steps according to the task information input by the user, and enhances the adaptability of the electronic device to different task requirements.

[0151] In step 203b, the electronic device executes the first task based on the second action sequence information.

[0152] In some embodiments of the present application, after obtaining the second action sequence information, the electronic device can execute each action in the order of the action sequence to complete the first task.

[0153] In this way, the electronic device updates the action sequence information corresponding to the screen recording video selected by the user according to the task information input by the user, so that the updated action sequence information can adapt to the task requirements of the first task, thereby accurately executing the first task, not only realizing the automatic execution of the first task, but also improving the accuracy of task execution.

[0154] Optionally, in some embodiments of the present application, in combination with Figure 3 As shown in Figure 5 The second action sequence information includes n second action information, and n is a positive integer. The step 203b can be implemented by the following steps 203b1 to 203b4.

[0155] In step 203b1, the electronic device sends the second action sequence information to the cloud server.

[0156] In some embodiments of the present application, the cloud server can receive the second action sequence information and the screenshot image sent by the electronic device through the network, and generate action instructions based on the received second action sequence information and the screenshot image. For example, the cloud server can use an action instruction generation model to parse the action information and the corresponding screenshot image to generate specific operation instructions. Then, the cloud server can send the generated action instructions back to the electronic device through the network to guide the electronic device to complete the first task. Alternatively, the electronic device can also run the action instruction model locally to generate specific operation instructions, and then execute the first task based on the operation instructions. The specific operation instructions can be determined according to actual use requirements, and the embodiments of the present application are not limited.

[0157] Exemplarily, the action instruction generation model can be a multi-modal large model.

[0158] In some embodiments of the present application, the electronic device can send the second action sequence information to the cloud server.

[0159] In some embodiments of the present application, the electronic device can send the second action sequence information to the cloud server in its entirety, and send the screenshot image to the cloud server after each subsequent execution of the action instruction; or the electronic device can also send each action information in the second action sequence information to the cloud server along with the screenshot image after each execution of the action instruction.

[0160] Step 203b2, the electronic device receives the i-th action instruction from the cloud server.

[0161] In some embodiments of the present application, 1≤i≤n.

[0162] In some embodiments of the present application, the above-mentioned action instruction, also referred to as a standardized action instruction, refers to an instruction for guiding the electronic device to perform a specific operation, and the details of each action are specified through a standardized format. The action instruction is generated by the cloud server and sent to the electronic device to perform the corresponding operation.

[0163] Exemplarily, the format of the above-mentioned action instruction is as follows:

[0164]

[0165] Wherein, type is used to specify the operation type, "click" represents a click operation, "swipe" represents a swipe operation, and "input" represents an input text operation. pos is used to specify the starting position coordinates of the operation, and [x, y] is the coordinate value on the screen. text is used to specify the input text content, and when the operation type is input, this field contains the text to be input. end is used to specify the end position coordinates of the swipe operation, and when the operation type is swipe, this field contains the coordinates of the swipe end point.

[0166] Exemplarily, assume that the cloud server generates the following action instruction:

[0167]

[0168] This instruction will trigger the electronic device to perform a click operation at the (300, 500) position on the screen.

[0169] Exemplarily, assume that the cloud server generates the following action instruction:

[0170]

[0171]

[0172] This instruction will trigger the electronic device to perform a swipe operation from (100, 200) to (500, 600) on the screen.

[0173] In some embodiments of the present application, after the cloud server generates a specific operation instruction using the action instruction generation model, the cloud server can send the operation instruction to the electronic device, and the electronic device can receive the instruction.

[0174] In step 203b3, the electronic device executes the ith action instruction, and screens the interface after the execution of the ith action instruction to obtain an ith screenshot image.

[0175] In some embodiments of the present application, after receiving the action instruction, the electronic device can execute the action instruction, and after the execution of the action instruction is completed, the electronic device can screen the current screen interface to obtain a screenshot image, so as to capture the interface state after the execution of the action instruction is completed.

[0176] In step 203b4, the electronic device sends the ith screenshot image to the cloud server.

[0177] In some embodiments of the present application, after obtaining the screenshot image, the electronic device can send the screenshot image to the cloud server, so as to help the cloud server understand the current operation progress and interface change.

[0178] In some embodiments of the present application, when i is 1, the ith action instruction is an action instruction generated by the cloud server based on a first action information in the at least one second action information.

[0179] In some embodiments of the present application, after the cloud server receives the second action sequence information sent by the electronic device, the cloud server can generate a first action instruction according to a first action information in the second action sequence information.

[0180] For example, the first action information is “open the ticket purchase application”, and the first action instruction generated by the cloud server can be

[0181] Wherein, [200, 300] is the position of the ticket purchase application in the screen, and the execution of the action instruction can realize the operation of “opening the ticket purchase application”.

[0182] In some embodiments of the present application, when 1

[0183] In some embodiments of the present application, after the cloud server receives the ith-1 screenshot image sent by the electronic device, the cloud server can generate an ith action instruction according to the ith-1 screenshot image and an ith action information in the second action sequence information.

[0184] For example, assuming that the second action sequence information is [open the ticket purchase application, select the amusement park ticket, select the date as next Wednesday, select the quantity as 2, and click the purchase button], after receiving the third screenshot image sent by the electronic device, the cloud server can generate the fourth action instruction according to the fourth action information “select the quantity as 2” in the second action sequence information and the third screenshot image. The cloud server first analyzes the third screenshot image, identifies that the current interface is the date selection interface of the amusement park ticket, and the date has been successfully selected as next Wednesday. Then, the cloud server determines the operation position and manner of selecting the ticket quantity on the current interface according to the fourth action information “select the quantity as 2” by using the multi-modal large model, and generates the fourth action instruction:

[0185]

[0186] wherein [400, 500] is the position corresponding to 2 in the quantity selection control, and executing the action instruction can realize the operation of “selecting the quantity as 2”.

[0187] In some embodiments of the present application, the electronic device completes the automatic execution of the first task by executing the first to nth action instructions, and after executing the nth action instruction.

[0188] In this way, the electronic device can send the action sequence information to the cloud server for conversion into action instructions and execution, thereby utilizing the powerful computing capability of the cloud server and realizing the accuracy and dynamic adaptability of complex task execution.

[0189] Optionally, in some embodiments of the present application, in combination with Figure 3 For example, as shown in FIG. 2B, the second action sequence information includes n second action information, and n is a positive integer. Figure 6 The step 203b can be implemented by the following steps 203b5 to 203b7.

[0190] Step 203b5, the electronic device receives the ith action instruction from the cloud server.

[0191] In some embodiments of the present application, 1≤i≤n.

[0192] Step 203b6, the electronic device executes the ith action instruction, and screens the interface after executing the ith action instruction to obtain the ith screenshot image.

[0193] Step 203b7, the electronic device sends the ith screenshot image and the (i+1)th action information to the cloud server.

[0194] In some embodiments of the present application, when i is 1, the i-th action instruction is an action instruction generated by the cloud server based on a first action information in the at least one second action information.

[0195] In some embodiments of the present application, when i is 1, the electronic device can send a first action information in the at least one second action information to the cloud server, the cloud server can generate a first action instruction based on the first action information sent by the electronic device and send the action instruction to the electronic device, and the electronic device can send a first screenshot image and a second action information in the at least one second action information to the cloud server after executing the first action instruction.

[0196] In some embodiments of the present application, when 1

[0197] In some embodiments of the present application, when 1

[0198] In some embodiments of the present application, the electronic device executes the first to n-th action instructions, and after executing the n-th action instruction, the automatic execution of the first task is completed.

[0199] In this way, the electronic device can send the action sequence information to the cloud server for conversion into action instructions and execution, thereby taking advantage of the powerful computing capability of the cloud server to achieve accuracy and dynamic adaptability in complex task execution.

[0200] The embodiments of the present application provide a task execution method. After receiving task information input by a user to describe a first task, the electronic device combines the task information with action sequence information corresponding to a screen recording video selected by the user, so that the action sequence information can adapt to the task requirements of the first task, thereby enabling the electronic device to automatically execute the first task. In this way, the electronic device can automatically execute other tasks based on the action sequence information of a completed task, thereby improving the flexibility of the electronic device in automatically executing tasks.

[0201] Optionally, in some embodiments of the present application, the task execution method provided by the embodiments of the present application further includes the following steps 401 to 403.

[0202] Step 401: The electronic device displays a first interface.

[0203] In some embodiments of the present application, the first interface includes at least one identifier, and one identifier indicates one video, and each video is a screen recording video when the electronic device performs a task.

[0204] For example, taking a mobile phone as an example, as shown in Figure 7 The mobile phone displays a first interface 10 including an identifier 11 and an identifier 12, wherein the identifier 11 is used to select the video corresponding to the identifier 11 to perform a task, and the identifier 12 is used to select the video corresponding to the identifier 12 to perform a task.

[0205] In step 402, the electronic device receives a first input of a first identifier in the at least one identifier.

[0206] In some embodiments of the present application, the first identifier is used to indicate a video selected by the user.

[0207] In some embodiments of the present application, the first identifier is one of the at least one identifier.

[0208] In some embodiments of the present application, the first input is used to select a video.

[0209] In some embodiments of the present application, the first input includes but is not limited to a touch input of the first identifier by a user through a touch device such as a finger or a stylus, or a voice instruction input by the user, or a specific gesture input by the user, or other feasible inputs. The specific gesture can be determined according to actual use requirements, and the embodiments of the present application are not limited.

[0210] In some embodiments of the present application, the touch input can be a single-click input, a double-click input, or any number of click inputs, or can be a long-press input or a short-press input. The specific gesture can be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long-press gesture, an area change gesture, a double-press gesture, and a double-click gesture.

[0211] For example, the first input can be a click input of the first identifier by the user.

[0212] In step 403, the electronic device displays a second interface in response to the first input.

[0213] In some embodiments of the present application, the second interface includes information of the video selected by the user, and the second interface is used to input task information input by the user.

[0214] In some embodiments of the present application, the user can input a click input to the first identifier to trigger the electronic device to display the second interface.

[0215] For example, in combination with Figure 7If a user wants to perform a task using the video corresponding to identifier 11 above, they can click [here]. Figure 7 The identifier 11 in the middle triggers the phone to display... Figure 8 The second interface 13 shown includes a task information text box 14 and a parsing control 15. Subsequently, if the user wants to modify the task information, they can click on the task information text box 14 to trigger the phone to edit the task information in the text box 14. Alternatively, if the user wants to execute a task based on the selected video and the entered task information, they can click on the parsing control 15 to trigger the phone to display a fourth interface.

[0216] In some embodiments of this application, after the user inputs task information through the second interface, the electronic device can display a fourth interface, which is used by the user to select the execution time of the first task.

[0217] For example, combined Figure 8 ,like Figure 9 As shown, if a user wants to perform a task based on the selected video and the input task information, they can click the parsing control 15 to trigger the phone to display the fourth interface 16.

[0218] In some embodiments of this application, the first interface further includes an upload control for displaying a fifth interface. The fifth interface is used to upload screen-recorded videos and to input the name and information of the screen-recorded videos. Users can input information into the upload control, causing the electronic device to display the fifth interface.

[0219] For example, such as Figure 7 As shown, the first interface also includes an upload control 17. Combined with... Figure 7 ,like Figure 10 As shown, if a user wants to upload a screen recording video and use it to perform a task, the user can click on the upload control 17 to trigger the electronic device to display the fifth interface 18.

[0220] In some embodiments of this application, the first interface further includes a teaching plaza control, which is used to display the teaching plaza interface. The teaching plaza interface displays screen recording videos shared by other users, and users can select screen recording videos relevant to their needs from the teaching plaza interface.

[0221] For example, such as Figure 7 As shown, the first interface also includes a teaching plaza control 19. Combined with... Figure 7 ,like Figure 11 As shown, if a user wants to perform a task using a screen recording video shared by another user, the user can click on the teaching plaza control 19 to trigger the electronic device to display the teaching plaza interface 20, which includes at least one screen recording video shared by another user.

[0222] In some embodiments of the present application, the matching relationship between the action description sequence and the task instruction input by the user can be manually selected by the user (i.e., the user manually selects the screen recording video), or can be automatically searched from the task instruction library by the terminal or the cloud, that is, the electronic device or the cloud automatically selects the screen recording video corresponding to the task information of the user from the screen recording videos shared by other users in the demonstration square, that is, the electronic device or the cloud can search the action sequence information matching the task information input by the user from the action sequence information of the screen recording videos shared by other users according to the task information input by the user, and take the screen recording video corresponding to the action sequence as the screen recording video used by the user to perform the task.

[0223] In this way, by displaying the first interface for the user to select the screen recording video and displaying the second interface for inputting the task information input by the user after the user selects, intuitive operation guidance is provided for the user, and operation efficiency is improved.

[0224] It should be noted that the above step 401 can be executed after the above step 201, that is, the electronic device can first receive the task information input by the user, then display the first interface and accept the input of the user to select the video, and finally obtain the first action sequence information corresponding to the video selected by the user. Alternatively, the above step 401 can be executed before the above step 201, that is, the electronic device can first display the first interface and accept the input of the user to select the video, and then receive the task information input by the user.

[0225] Optionally, in some embodiments of the present application, the first interface further includes at least one analysis identifier, and one analysis identifier indicates the analysis state of one video. The task execution method provided by the embodiments of the present application further includes the following steps 501 and 502.

[0226] Step 501, the electronic device receives a second input on a first analysis identifier in the at least one analysis identifier.

[0227] In some embodiments of the present application, the first analysis identifier is used to indicate the analysis state of the video selected by the user.

[0228] In some embodiments of the present application, the second input is used to display the analysis state or the analysis result of the video selected by the user.

[0229] In some embodiments of the present application, the second input includes but is not limited to: touch input of the first analysis identifier by the user through a touch device such as a finger or a stylus, or a voice instruction input by the user, or a specific gesture input by the user, or other feasible inputs. The specific can be determined according to the actual use demand, and the embodiments of the present application are not limited.

[0230] Exemplarily, the first input can be a click input of the user on the first parsing identifier.

[0231] At step 502, the electronic device displays a third interface in response to the second input.

[0232] In some embodiments of the present application, the third interface is used to indicate a parsing state or a parsing result of the video indicated by the first parsing identifier, and the parsing result includes action sequence information corresponding to the video indicated by the first parsing identifier.

[0233] In some embodiments of the present application, the video indicated by the first parsing identifier is a screen recording video when the electronic device performs the second task, and the action sequence information corresponding to the video is information describing an action sequence of the electronic device performing the second task.

[0234] In some embodiments of the present application, after the user performs the second input, if the video indicated by the first parsing identifier is being parsed, the electronic device can display the parsing state of the video indicated by the first parsing identifier in the third interface.

[0235] Exemplarily, as shown in Figure 7 , the first interface further includes a first parsing identifier 21, a first parsing identifier 22, and a first parsing identifier 23, wherein the parsing states of the videos indicated by the first parsing identifier 21 and the first parsing identifier 22 are completed, and the parsing state of the video indicated by the first parsing identifier 23 is being parsed. If the user wants to view the parsing state of the video indicated by the first parsing identifier 23, the user can perform a click input on the first parsing identifier 23, as shown in Figure 12 , to trigger the electronic device to display a third interface 24, and the third interface 24 includes the parsing state of the video indicated by the first parsing identifier 23.

[0236] In some embodiments of the present application, after the user performs the second input, if the video indicated by the first parsing identifier is parsed, the electronic device can display a parsing result of the video indicated by the first parsing identifier in the third interface.

[0237] Exemplarily, in combination with Figure 7 , as shown in Figure 13 , if the user wants to view the parsing result of the video indicated by the first parsing identifier 21, the user can perform a click input on the first parsing identifier 21, as shown in Figure 12 , to trigger the electronic device to display a third interface 25, and the third interface 25 includes the parsing result of the video indicated by the first parsing identifier 21.

[0238] In some embodiments of the present application, the parsing result of the video includes an action description sequence of the video, and the user can directly modify the action description in the action description sequence, so that the automatic execution of the task can be more flexible and accurate.

[0239] In this way, the electronic device can show the user the parsing state or result of the video, so that the user can quickly understand whether the video has been parsed and view the specific parsing result, so as to facilitate the user to adjust the task according to the parsing state.

[0240] In some embodiments of the present application, the electronic device can first obtain a task instruction issued by a user and an action description sequence associated with the instruction, the action description sequence being generated in advance according to an operation screen recording provided by the user. The action description sequence, the operation screen recording, and the name are saved as a teaching record. Then, the action description sequence is updated based on the task instruction issued by the user, and the task instruction issued by the user is executed based on the updated action description sequence. If the task instruction issued by the user is consistent with the original action description sequence, the original action description sequence is directly executed subsequently without updating.

[0241] The embodiments of the present application provide a method for automatically executing a user instruction. An operation screen provided by a user is parsed to determine an action description of each operation step. The action description is updated based on a task instruction issued by the user. The task instruction is automatically executed based on the updated action description, thereby realizing the function of automatically executing a user instruction by a mobile phone.

[0242] Improvement point 1: The present scheme can parse the execution flow of the operation steps of the user from the screen recording. The user only needs to upload the operation teaching video in the form of a screen recording, and the learning cost of the user is low.

[0243] Improvement point 2: The present scheme identifies and parses the screen recording to generate an action description of the operation steps. The action description is a semantic flow step description, rather than a traditional form of coordinates, control identifiers (IDs), and the like, which are not easy to modify, and has stronger adaptability to changes in application interfaces.

[0244] Improvement point 3: The present scheme executes a newly issued task instruction of a user based on the action description generated from the previous screen recording. The action description can be updated based on the newly issued task instruction (in the form of natural language) of the user, thereby supporting the changing needs of the user.

[0245] In some embodiments of the present application, the electronic device can parse the execution flow of the operation steps of the user from the screen recording, thereby reducing the learning cost of the user. The present scheme automatically executes based on a semantic action description, rather than a traditional form of coordinates, control IDs, and the like, thereby having stronger adaptability to changes in application interfaces. The present scheme can update the original action description based on a newly issued task instruction (in the form of natural language) of the user, thereby supporting the changing needs of the user.

[0246] Each of the above method embodiments, or each of the various possible implementations of the method embodiments, can be executed alone or in combination with any two or more of the others, and the specific implementation can be determined according to actual use requirements, which is not limited in the embodiments of the present application.

[0247] The task execution method provided in the embodiments of the present application can be executed by a task execution device. The task execution device is taken as an example in the embodiments of the present application to illustrate the task execution device provided in the embodiments of the present application.

[0248] Figure 14 A possible structural schematic diagram of the task execution device involved in some embodiments of the present application is shown. As shown in the figure, the task execution device 70 can include a receiving module 71, an obtaining module 72 and an execution module 73, wherein: Figure 14

[0249] The receiving module 71 is configured to receive user input task information, and the user input task information is used to describe a first task.

[0250] The obtaining module 72 is configured to obtain first action sequence information.

[0251] The execution module 73 is configured to execute the first task based on the user input task information received by the receiving module 71 and the first action sequence information obtained by the obtaining module 72.

[0252] The first action sequence information corresponds to a video selected by the user, the video is a screen recording video when an electronic device executes a second task, and the first action sequence information is used to describe an action sequence of the electronic device executing the second task.

[0253] In a possible implementation, the execution module 73 is specifically configured to: based on the user input task information received by the receiving module 71 and the first action sequence information obtained by the obtaining module 72, generate second action sequence information, the second action sequence information is used to describe an action sequence of the electronic device executing the first task; and execute the first task based on the second action sequence information.

[0254] In a possible implementation, the first action sequence information includes at least one first action information. The execution module 73 is specifically configured to: determine, from the at least one first action information, first action information corresponding to the user input task information; generate second action information based on the user input task information and the first action information; and update the first action information in the first action sequence information to the second action information to obtain the second action sequence information.

[0255] ​In a possible implementation, the second action sequence information includes n second action information, where n is a positive integer. The execution module 73 is specifically configured to: send the second action sequence information to the cloud server; receive an i th action instruction from the cloud server, where 1≤i≤n; execute the i th action instruction, and obtain an i th screenshot image by taking a screenshot of the interface after executing the i th action instruction; and send the i th screenshot image to the cloud server, where i is 1, the i th action instruction is generated by the cloud server based on a first action information in the at least one second action information, and 1

[0256] In a possible implementation, the task execution apparatus provided in the embodiments of the present application further includes a display module. The display module is configured to display a first interface, where the first interface includes at least one identifier, and one identifier indicates one video, and each video is a screen recording video when an electronic device executes one task. The receiving module 71 is further configured to receive a first input on a first identifier in the at least one identifier displayed by the display module, where the first identifier is used to indicate a video selected by a user. The display module is further configured to display a second interface in response to the first input received by the receiving module 71, where the second interface includes information of the video selected by the user, and the second interface is used to input task information input by the user.

[0257] In the embodiments of the present application, a task execution apparatus is provided. After receiving task information input by a user and used to describe a first task, the task execution apparatus combines the task information with action sequence information corresponding to a screen recording video selected by the user, so that the action sequence information can be adapted to the task requirement of the first task, and the task execution apparatus can automatically execute the first task. In this way, the task execution apparatus can automatically execute other tasks based on the action sequence information of one completed task, and the flexibility of the task execution apparatus in automatically executing tasks is improved.

[0258] The task execution device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0259] The task execution device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0260] The task execution device provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0261] Optionally, such as Figure 15 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described task execution method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0262] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0263] Figure 16 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0264] The electronic device 100 includes, but is not limited to, a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110, etc.

[0265] Those skilled in the art can understand that the electronic device 100 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 110 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 16 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0266] The user input unit 107 is configured to receive user input task information, and the user input task information is used to describe a first task.

[0267] The processor 110 is configured to obtain first action sequence information.

[0268] The processor 110 is further configured to execute the first task based on the user input task information and the first action sequence information.

[0269] The first action sequence information corresponds to a video selected by the user, the video is a screen recording video when the electronic device executes a second task, and the first action sequence information is used to describe an action sequence of the electronic device executing the second task.

[0270] Optionally, the processor 110 is specifically configured to: based on the user input task information and the first action sequence information, generate second action sequence information, the second action sequence information is used to describe an action sequence of the electronic device executing the first task; and based on the second action sequence information, execute the first task.

[0271] Optionally, the first action sequence information includes at least one first action information. The processor 110 is specifically configured to: determine, from the at least one first action information, first action information corresponding to the user input task information; based on the user input task information and the first action information, generate second action information; update the first action information in the first action sequence information to the second action information to obtain the second action sequence information.

[0272] Optionally, the second action sequence information includes n second action information, n is a positive integer. The processor 110 is specifically configured to: send the second action sequence information to the cloud server; receive the i th action instruction from the cloud server, 1≤i≤n; execute the i th action instruction, and obtain the i th screenshot image by taking a screenshot of the interface after executing the i th action instruction; send the i th screenshot image to the cloud server; wherein, when i is 1, the i th action instruction is generated by the cloud server based on the first action information in the at least one second action information; when 1

[0273] Optionally, the display unit 106 is configured to display a first interface, the first interface including at least one identifier, one identifier indicating one video, and each video being a screen recording video when an electronic device performs one task. The user input unit 107 is further configured to receive a first input on a first identifier in the at least one identifier, the first identifier being used to indicate a video selected by a user. The display unit 106 is further configured to display, in response to the first input, a second interface including information of the video selected by the user, the second interface being used to input task information input by the user.

[0274] The embodiments of the present application provide an electronic device. After receiving task information input by a user and used to describe a first task, the electronic device can combine the task information with action sequence information corresponding to a screen recording video selected by the user, so that the action sequence information can adapt to the task requirement of the first task, thereby enabling the electronic device to automatically perform the first task. In this way, the electronic device can automatically perform other tasks based on the action sequence information of one completed task, thereby improving the flexibility of the electronic device in automatically performing tasks.

[0275] The electronic device provided by the embodiments of the present application can implement each process implemented by the method embodiments and achieve the same technical effects. To avoid repetition, details are not described here. The beneficial effects of various implementation manners in the present embodiment can refer to the beneficial effects of the corresponding implementation manners in the method embodiments. To avoid repetition, details are not described here.

[0276] It should be understood that in the embodiments of the present application, the input unit 104 can include a graphics processing unit (GPU) 1041 and a microphone 1042. The graphics processing unit 1041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 can include a display panel 1061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 can include two parts of a touch detection device and a touch controller. The other input devices 1072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, a joystick, and the like, which will not be described here.

[0277] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 109 can include a volatile memory or a non-volatile memory, or the memory 109 can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link DRAM (SLDRAM), and a direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 109 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0278] The processor 110 can include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes a wireless communication signal, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 110.

[0279] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores a program or instructions, the program or instructions are executed by a processor to realize each process of the task execution method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0280] The processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0281] The embodiment of the present application further provides a chip, and the chip includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run a program or instructions to realize each process of the task execution method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0282] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system, or a system on chip, etc.

[0283] The embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to realize each process of the task execution method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0284] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the method and apparatus of the present application can be carried out by more than one process, method, article, or apparatus either simultaneously, concurrently, or with intervening action that are carried out at the same time, either in a simultaneous fashion or in a fashion that is interleaved in time. For example, the described methods can be performed in a different order from that described, and / or various steps can be combined or omitted, and / or additional steps can be added, without departing from the scope of the present application. Also, features described with respect to certain examples can be combined in other examples.

[0285] From the above description of the embodiments, it is apparent that the above-described method of the embodiments can be realized by means of software and general-purpose hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.

[0286] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, rather than limiting, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims, which all belong to the protection of the present application.

Claims

1. A task execution method characterized by, The method comprises: receiving task information input by a user, the task information being used to describe a first task; obtaining first action sequence information; performing the first task based on the task information and the first action sequence information; wherein the first action sequence information corresponds to a video selected by the user, the video being a screen recording video of an electronic device performing a second task, and the first action sequence information being used to describe an action sequence of the electronic device performing the second task.

2. The method of claim 1, wherein, The performing of the first task based on the task information and the first action sequence information comprises: generating second action sequence information based on the task information and the first action sequence information, the second action sequence information being used to describe an action sequence of the electronic device performing the first task; performing the first task based on the second action sequence information.

3. The method of claim 2, wherein, The first action sequence information comprises at least one first action information. The generating of the second action sequence information based on the task information and the first action sequence information comprises: determining, from the at least one first action information, first action information corresponding to the task information; generating second action information based on the task information and the first action information; updating the first action information in the first action sequence information to the second action information to obtain the second action sequence information.

4. The method according to claim 2 or 3, characterized in that, The second action sequence information comprises n second action information, n being a positive integer. The performing of the first task based on the second action sequence information comprises: sending the second action sequence information to a cloud server; receiving an i th action instruction from the cloud server, 1≤i≤n; performing the i th action instruction and taking a screenshot of an interface after the i th action instruction is performed to obtain an i th screenshot image; sending the i th screenshot image to the cloud server; wherein, when i is 1, the i th action instruction is generated by the cloud server based on a first action information in the at least one second action information; when 1 5. The method of claim 1, wherein, The method further comprises: displaying a first interface, the first interface comprising at least one identifier, one identifier indicating one video, each video being a screen recording video of an electronic device performing a task; receiving a first input on a first identifier in the at least one identifier, the first identifier being used to indicate a video selected by the user; in response to the first input, displaying a second interface, the second interface comprising information of the video selected by the user, the second interface being used to input task information.

6. A task execution apparatus characterized by comprising: The method comprises: a receiving module, an obtaining module and an executing module; the receiving module is configured to receive task information input by a user, the task information being used to describe a first task; the obtaining module is configured to obtain first action sequence information; the executing module is configured to perform the first task based on the task information received by the receiving module and the first action sequence information obtained by the obtaining module. The first action sequence information corresponds to a video selected by a user, the video being a screen recording video when the electronic device executes a second task, and the first action sequence information is used to describe an action sequence of the electronic device executing the second task.

7. The apparatus of claim 6, wherein, The execution module is specifically configured to: generate second action sequence information based on the task information received by the receiving module and the first action sequence information obtained by the obtaining module, the second action sequence information being used to describe an action sequence of the electronic device executing the first task; and execute the first task based on the second action sequence information.

8. The apparatus of claim 7, wherein, The first action sequence information includes at least one first action information. The execution module is specifically configured to: determine first action information corresponding to the task information from the at least one first action information; and generate second action information based on the task information and the first action information; update the first action information in the first action sequence information to the second action information to obtain second action sequence information.

9. The apparatus of claim 7 or 8, wherein, The second action sequence information includes n second action information, n being a positive integer. The execution module is specifically configured to: send the second action sequence information to a cloud server; receive an i th action instruction from the cloud server, 1≤i≤n; execute the i th action instruction and take a screenshot of an interface after executing the i th action instruction to obtain an i th screenshot image; send the i th screenshot image to the cloud server; when i is 1, the i th action instruction is generated by the cloud server based on first action information in the at least one second action information; when 1 10. The apparatus of claim 6, wherein, The apparatus further includes a display module. The display module is configured to display a first interface, the first interface including at least one identifier, one identifier indicating one video, and each video being a screen recording video when an electronic device executes a task. The receiving module is further configured to receive a first input to a first identifier in the at least one identifier displayed by the display module, the first identifier being used to indicate a video selected by a user. The display module is further configured to display a second interface in response to the first input received by the receiving module, the second interface including information of the video selected by the user, and the second interface being used to input task information.

Images