Terminal function control method, apparatus, device, storage medium, and program product

Abstract

This disclosure provides a terminal function control method, apparatus, device, storage medium, and program product. By using a first application to process a first instruction input by a user, operation steps for simulating user operation are generated. By executing the operation steps within the terminal device, corresponding execution results are generated and displayed, thereby achieving automatic task execution. In this process, since the operation steps are matched with the heterogeneous operation units of the terminal device, it can be adapted to heterogeneous terminal devices that execute tasks through different heterogeneous operation units and in different ways, thereby achieving automated function operation on different types of terminal devices and improving operational efficiency and accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a terminal function control method, apparatus, device, storage medium, and program product. Background Technology

[0002] Currently, users need to operate the heterogeneous operating units provided by the terminal device to perform various functions in terminal devices (such as personal computers, wearable devices, etc.), such as a series of mouse and keyboard operations, to activate a specific function of the computer.

[0003] However, the aforementioned manual operation methods in related technologies suffer from problems such as low operation efficiency and poor operation accuracy, which affect the user experience. Summary of the Invention

[0004] This disclosure provides a terminal function control method, apparatus, device, storage medium, and program product to overcome the problems of low operating efficiency and poor operating accuracy.

[0005] In a first aspect, embodiments of this disclosure provide a terminal function control method, including:

[0006] The system receives a first instruction, which includes a task described in natural language, and the task is executed within a second application of the terminal device. The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation units of the terminal device.

[0007] Secondly, embodiments of this disclosure provide a terminal function control device, including:

[0008] An interaction module is configured to receive a first instruction, the first instruction comprising a task described in natural language, the task being executed within a second application of a terminal device;

[0009] The processing module is configured to process the first instruction through a first application, execute the operation steps corresponding to the task, and display the execution results corresponding to the operation steps, wherein the operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

[0010] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0011] The memory stores computer-executed instructions;

[0012] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the terminal function control method as described in the first aspect and various possible designs of the first aspect.

[0013] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the terminal function control method described in the first aspect and various possible designs of the first aspect.

[0014] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the terminal function control method described in the first aspect and various possible designs of the first aspect.

[0015] The terminal function control method, apparatus, device, storage medium, and program product provided in this embodiment receive a first instruction, which includes a task described in natural language, and the task is executed within a second application of the terminal device. The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operating units of the terminal device. By processing the first instruction input by the user using the first application, operation steps for simulating user operations are generated. By executing the operation steps within the terminal device and generating and displaying the corresponding execution results, the task is automatically executed. In this process, because the operation steps are matched with the heterogeneous operating units of the terminal device, it can adapt to heterogeneous terminal devices that execute tasks through different heterogeneous operating units and in different ways, thereby achieving automated function operation on different types of terminal devices and improving operational efficiency and accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application scenario diagram of the terminal function control method provided in the embodiments of this disclosure;

[0018] Figure 2 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 1 ;

[0019] Figure 3 This is a schematic diagram illustrating a process of performing a task, provided as an embodiment of the present disclosure.

[0020] Figure 4 This is a schematic diagram of an execution operation step chain provided by an embodiment of the present disclosure;

[0021] Figure 5 A flowchart illustrating the specific implementation of step S1022;

[0022] Figure 6 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 2 ;

[0023] Figure 7 This is a schematic diagram illustrating a process for displaying second information, provided as an embodiment of the present disclosure.

[0024] Figure 8 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 3 ;

[0025] Figure 9 This is a schematic diagram illustrating the execution process of an operation step provided in an embodiment of the present disclosure;

[0026] Figure 10 This is a structural block diagram of the terminal function control device provided in the embodiments of this disclosure;

[0027] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0028] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] The application scenarios of the embodiments of this disclosure are explained below:

[0036] The terminal function control method provided in this disclosure can be applied to applications (APPs) with intelligent agent functions, such as intelligent assistant applications and artificial intelligence (AI) tool applications. More specifically, it can be applied to automatic control scenarios for terminal devices such as computers and wearable devices equipped with heterogeneous operating units. The execution subject in this embodiment can be a terminal device running the aforementioned application with intelligent agent functions, a server deploying the server corresponding to the aforementioned application, or other electronic devices that perform similar functions. When the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the aforementioned application; when the execution subject is a server, the server of the aforementioned application with intelligent agent functions can run partially or entirely on the server, and the method provided in this embodiment is executed on the server side, while the terminal device runs the client of the application. Communication between the server and the terminal device is based on server-client communication, thereby enabling the terminal device to obtain the execution result of the method provided in this embodiment and display it as needed.

[0037] In some embodiments, the terminal device or server can implement the terminal function control method provided in this disclosure by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be program-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be local applications, i.e., programs that need to be installed in the operating system to run, or small programs embedded in any APP, i.e., programs that run in a browser environment. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin; the specific implementation can be configured as needed. Furthermore, in the process of implementing the terminal function control method provided in this disclosure, the terminal device can execute the method by running computer-executable instructions or computer programs set locally, or by calling computer-executable instructions or computer programs set in an external server. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminal devices.

[0038] Figure 1 This is an application scenario diagram of the terminal function control method provided in the embodiments of this disclosure, with reference to... Figure 1 As shown in the diagram, taking a personal computer as an example, the terminal device is equipped with operating devices such as a mouse and keyboard. Users can operate the terminal device using these external devices to complete corresponding tasks. For example, if a user needs to complete the task of "downloading this week's report B from website A," they first need to use the mouse to launch browser software from the operating system desktop, then enter the address of website A in the address bar of the browser software, and then search for report B on website A and download it. In the existing technology, completing the above task requires the user to manually perform a series of mouse and keyboard operations, resulting in low operating efficiency and poor operating accuracy, thus affecting the user experience.

[0039] In related technologies, schemes have been proposed to convert operation commands into corresponding program operations to achieve automatic operation functions. However, the above-mentioned schemes are only applicable to terminal devices with universal input units, such as smartphones. Since operations on smartphones are all completed using their own touchscreens (universal input units), any functional operation performed on a mobile phone can be converted into corresponding touch methods and touchscreen coordinates to achieve the execution of the operation commands. However, for terminal devices with heterogeneous operation units, such as computers and wearable devices, it is not possible to simply convert operation commands into operation steps based on universal input units, thus making it difficult to achieve the automatic execution of the aforementioned operation commands.

[0040] This disclosure provides a terminal function control method to solve the above problems.

[0041] refer to Figure 2 , Figure 2 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 1 The method of this embodiment can be applied to a terminal device or a server. In one possible implementation, for a terminal device executing the method provided in this embodiment, the terminal device can implement the terminal function control method provided in this embodiment by executing program code deployed locally and / or externally. In another possible implementation, a server can be used to deploy functional services implemented based on the terminal function control method provided in this embodiment, and the terminal device can implement the terminal function control method provided in this embodiment by accessing the server and calling the corresponding functional services. For example, the terminal function control method provided in this embodiment includes:

[0042] Step S101: Receive a first instruction, the first instruction including a task described in natural language, the task being executed within a second application of the terminal device.

[0043] Step S102: The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

[0044] refer to Figure 1 The illustrated application scenario diagram illustrates how a terminal device, as the executing entity, provides a terminal function control method. For example, the terminal device is a personal computer running a first application, which may be an intelligent assistant application based on artificial intelligence technology. Specifically, after running the first application, the terminal device displays its interactive interface on the operating system desktop. This interface includes an input box control for human-computer interaction. The user inputs a first command through this control. This first command, written in natural language, describes the task to be performed on the terminal device. For example, the first command might be "Download this week's report B from website A". The first application then, for example, invokes an agent to understand the first command and generates one or more consecutively executed operation steps, which are then executed within a second application. The second application can be an application within the terminal device's operating system, such as a browser or file manager, or a third-party application running on the operating system. Simultaneously, after each operation step is executed, the corresponding execution result is displayed until all operation steps are completed, thus fulfilling the task described by the first command. The above operation steps simulate user actions applied to the terminal device, which are implemented based on the heterogeneous operating units of the terminal device. In other words, the operation steps simulate the operation of the heterogeneous operating units so that the terminal device can receive the operation signals output by the heterogeneous operating units, thereby completing operations such as "clicking," "dragging," and "entering text," which can only be performed using heterogeneous operating units such as a mouse and keyboard.

[0045] In one possible implementation, a first instruction is input into an agent running within a first application to generate multiple operation objectives for achieving the task objective described by the first instruction. The task objective is, for example, "copy file 'a' from drive C to folder 'b' on drive D," and the operation objectives are decompositions of the task objective, such as "Step 1: Open drive C," "Step 2: Search for file 'b' on drive C," "Step 3: Copy file 'b' from the search results," and so on. Then, starting with the first operation objective, the following operations are performed sequentially:

[0046] Step S21: Obtain the current operation target.

[0047] Step S22: Identify the current interactive interface of the terminal device and obtain the identification result. The identification result represents the objects displayed in the current interactive interface and the positional relationship between the objects.

[0048] Step S23: Based on the identification result and the current operation target, call the matching heterogeneous operation unit to execute the corresponding operation steps to achieve the current operation target.

[0049] Step S24: If the current operation target is not the last operation target, return to step S21.

[0050] For example, for each operation target, the current interactive interface is identified sequentially to determine the content displayed on the interface, i.e., the identification result. This includes the objects displayed on the interface, such as icons, web pages, controls, etc., as well as the position and range of each object. Then, the agent is invoked to reason based on the identification result and the current operation target to determine the matching heterogeneous operation unit and the operation steps to be performed based on that heterogeneous operation unit. For example, if the matching heterogeneous operation unit is determined to be a "mouse," the "mouse" is invoked to perform movement, clicking, and other operation steps, ultimately achieving operation targets such as "opening the C drive" and "searching for file b on the C drive." This process is repeated until all operation targets are achieved, completing the operation steps corresponding to the task.

[0051] Figure 3 This is a schematic diagram of a task execution process provided by an embodiment of the present disclosure, which is described below in conjunction with... Figure 3 The above process will be described as follows: Figure 3 As shown, exemplarily, firstly, a dialog box control is displayed in the interactive interface of the first application. Then, the user inputs a first instruction through the dialog box control: "Download this week's report B from website A". After receiving the first instruction, the first application calls an intelligent agent to process the first instruction, generating one or more operation steps, and displays these operation steps sequentially in the interactive interface. For example, as shown in the figure, the interactive interface sequentially displays operation steps such as "Start browser", "Start browser - Complete", "Enter website A URL", "Enter website A URL - Complete", and the corresponding execution results. At the same time, the terminal device executes the above operation steps, opens a browser through the operating system, enters the URL of website A in the browser, accesses website A, and then downloads this week's report B from website A, ultimately completing the task of "Downloading this week's report B from website A".

[0052] Furthermore, in one possible implementation, step S102 is specifically implemented as follows:

[0053] Step S1021: Semantically understand the first instruction through the first application to obtain the task objective.

[0054] Step S1022: Based on the task objective and the device parameters of the terminal device, execute the operation step chain, wherein the device parameters represent the heterogeneous operation unit of the terminal device, and the operation step chain includes at least two operation steps, and the operation steps are matched with the heterogeneous operation unit.

[0055] For example, the first application first invokes an intelligent agent to perform semantic understanding of the first instruction. The intelligent agent is a program implemented based on a large language model, which can extract and recognize the semantic features of the first instruction to obtain the semantics described by the first instruction, i.e., the task objective, such as "download this week's report B from website A". Then, for this task objective, an operation path that can achieve the task objective is planned, i.e., an operation step chain. For example, the intelligent agent, through reasoning, determines that to achieve the task objective of "downloading this week's report B from website A", it is first necessary to launch a browser. The launch method needs to consider the heterogeneous operating units of the terminal device (such as mouse, keyboard, gamepad, etc.), i.e., device parameters. Therefore, for example, the intelligent agent, based on the heterogeneous operating unit "mouse" of a personal computer, sets the first operation step of the operation step chain as: moving the mouse to the browser icon, the second operation step as: clicking the left mouse button, until all operation steps of the operation step chain are completed.

[0056] In one possible implementation, the interactive interface of the first application is designated as the first interactive interface. Within this interface, first information is displayed, including at least one of the following: the name of the task, the corresponding operation step sequence, and the task's execution progress. For example, the first interactive interface might display the task name as "Report Download," the operation step sequence as step S1, step S2, step S3, etc., and the task's execution progress as, for example, 40% or 60%.

[0057] Furthermore, in one possible implementation, it further includes displaying a third and fourth window on the second interactive interface, where the second interactive interface is the interactive interface of the terminal device's operating system. The specific implementation of the operation steps corresponding to the task includes: inputting the object presented in the third window into the fourth window. Here, the third and fourth windows are two independent windows displayed within the second interactive interface; for example, the third window is a folder window and the fourth window is a document window. Inputting the object presented in the third window into the fourth window, for example, inserting an image file from the folder into the document, achieves multi-window interactive operation steps in the above manner.

[0058] Figure 4 This is a schematic diagram of an execution operation step chain provided by an embodiment of the present disclosure, which is described below in conjunction with... Figure 4 The specific process of the above-mentioned execution operation steps is described below, such as... Figure 4 As shown, in this embodiment, the first application is hidden after the task begins execution. The process of executing the operation steps in the background is as follows: First, for the task objective "download this week's report B from website A", the agent, based on device parameters, infers and determines the first operation step S1 as "move the mouse to position [x1, y1]", the second operation step S2 as "double-click the left mouse button", the third operation step S3 as "move the mouse to position [x2, y2]", the fourth operation step S4 as "select the text www.Axxx..." (website A's URL), the fifth operation step S5 as "drag the selected text to position [x3, y3]", and the sixth operation step S6 as "key". "Pin Enter..." and so on, continuously executing the above steps until each step of the operation step chain (S1-S2-S3-S4-...-Sn) is completed. Optionally, during the execution of the above operation step chain, before executing each step, the first application will also perform an additional screen capture step to obtain the content currently displayed on the terminal device's screen, as the basis for executing the next operation step; if, based on the content currently displayed on the screen, the agent determines that the previous operation step was unsuccessful, it will return to execute the previous step, thereby improving the success rate of the operation step chain execution.

[0059] Furthermore, in one possible implementation, such as Figure 5 As shown, the specific implementation of step S1022 includes:

[0060] Step S1022-1: Obtain the operation target corresponding to the current operation step.

[0061] Step S1022-2: Based on the operation target and equipment parameters, obtain the operating status of the candidate operation unit. The candidate operation unit is a heterogeneous operation unit used to achieve the operation target.

[0062] Step S1022-3: Determine the first operation unit based on the running status of the candidate operation units, and execute the operation steps matching the operation target based on the first operation unit.

[0063] For example, during the execution of the operation step chain, for at least one operation step, i.e., the current operation step, the operation target corresponding to the current operation step is first obtained, such as "moving the mouse" or "clicking control A". Then, based on the operation target and device parameters, heterogeneous operation units capable of achieving the above operation target are determined, i.e., candidate operation units, such as a built-in touch panel on the computer, mouse A, and mouse B. Next, the operating status of the candidate operation units is obtained, such as the mouse's occupancy status, power-saving status, and dots per inch (DPI). Based on the above operating status, a better one is selected as the first operation unit (i.e., the matching heterogeneous operation unit). Then, the operation step matching the operation target is executed based on the first operation unit. Through the above embodiment steps, when multiple candidate operation units exist on the terminal device, the operation step executed by the candidate operation unit whose operating status is more closely matched to the operation target can be selected, thereby improving operation accuracy (e.g., a matched DPI can improve mouse movement accuracy) and achieving energy saving.

[0064] In this embodiment, a first instruction is received, which includes a task described in natural language. The task is executed within a second application on the terminal device. The first application processes the first instruction, executes the corresponding operation steps, and displays the execution results. The operation steps simulate user operations applied to the terminal device, and these user operations are implemented using heterogeneous operating units on the terminal device. By processing the first instruction input by the user using the first application, operation steps for simulating user operations are generated. Executing these operation steps within the terminal device generates and displays the corresponding execution results, thus achieving automatic task execution. In this process, because the operation steps match the heterogeneous operating units of the terminal device, it can adapt to heterogeneous terminal devices that execute tasks using different heterogeneous operating units and methods. This enables automated function operation on different types of terminal devices, improving operational efficiency and accuracy.

[0065] refer to Figure 6 , Figure 6 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 2 This embodiment is in Figure 2 Based on the illustrated embodiment, step S102 is further refined, and the terminal function control method includes:

[0066] Step S201: Receive a first instruction, the first instruction including a task described in natural language, the task being executed within a second application of the terminal device.

[0067] Step S202: The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

[0068] Step S203: Display the first information in the first window of the first interactive interface.

[0069] Step S204: In response to the second operation, second information is displayed in the second window of the first interactive interface. The second operation includes an operation on the first operation step in the first information, and the second information is associated with the first operation step.

[0070] For example, the second information includes at least one of the following: the step name of the operation step, the execution progress of the operation step, and the execution details of the operation step. The step name of the operation step may be, for example, "moving the mouse" or "dragging and dropping a file icon," specific examples of which have been described in previous embodiments and will not be repeated here. The execution progress of the operation step can be a textual description, such as "executing" or "completed," or a percentage value. The execution details of the operation step may be, for example, the Chain of Thought (COT) information output by the agent. In one possible implementation, the second information can be directly displayed in the first interactive interface during task execution, i.e., the second information is displayed in the first interactive interface. In another possible implementation, the second information is displayed in a second window and triggered in response to the second operation.

[0071] Figure 7 This is a schematic diagram illustrating a process for displaying second information, as provided in an embodiment of this disclosure. Figure 7As illustrated, exemplarily, during the execution of the operation steps corresponding to a task by the intelligent agent, for each operation step, first information related to the task is displayed in the first window of the first interactive interface, including the task name, the corresponding operation step chain, and the task execution progress. Based on this, in response to a second user operation, such as clicking the currently executing first operation step (e.g., the "Launch Browser" step shown in the figure), second information is displayed in the second window of the first interactive interface. For example, as shown in the figure, the second window is a collapsible window; after responding to the user's second input, the second window expands and covers the first window. Simultaneously, the second window displays second information related to the first operation step, including the execution progress and execution details of the first operation step. For example, as shown in the figure, the content of the second information includes: "Locating browser location," "Move mouse to browser location," "Double-click left mouse button to launch browser," etc. Through the above embodiment steps, the collapsing and expanding of the second information is achieved, thereby enabling on-demand display of the second information for specific operation steps and improving the information display efficiency within the first interactive interface.

[0072] Optionally, this embodiment further includes:

[0073] Step S205: In response to the first operation, configure the task in the execution state to the paused state, and / or restore the task in the paused state to the execution state.

[0074] For example, during the operation steps corresponding to the above-mentioned task execution, the first application can further respond to the first operation applied by the user by pausing or resuming the currently executing task, that is, configuring the task in the executing state to a paused state, and resuming the task in the paused state to an executing state. In this way, the automatic execution process can be paused and adjusted at any time when the user discovers an execution abnormality, thereby reducing the probability of automatic execution failure and improving the flexibility of use.

[0075] refer to Figure 8 , Figure 8 Flowchart of the terminal function control method provided in the embodiments of this disclosure Figure 3 In this embodiment Figure 2 Based on the illustrated embodiment, the method further adds steps for pausing and resuming tasks, and the terminal function control method includes:

[0076] Step S301: Receive a first instruction, the first instruction including a task described in natural language, the task being executed within a second application of the terminal device.

[0077] Step S302: The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

[0078] Step S303: In response to the third operation, at the second operation step position in the operation step chain, suspend the execution of the task.

[0079] Step S304: In response to the fourth operation, configure the first content of the terminal device and generate the second content. The first content is associated with the third operation step, which is executed after the second operation step.

[0080] Step S305: In response to the fifth operation, continue to perform the task on the terminal device based on the second content.

[0081] For example, during the execution of an operation step chain, when the execution reaches a certain operation step, such as the second operation step, if the user applies a third operation, the execution of the task can be paused. The second operation step can be any operation step within the task, without restriction. The third operation, for example, is a click operation on a functional control within the interactive interface of the first application. After the terminal device receives this third operation, it controls the agent to pause the execution of subsequent steps, i.e., the task is configured to a paused state. Afterwards, the user can apply a fourth operation to configure the first content on the terminal device, generating second content. Specifically, the fourth operation is an operation to adjust the content stored on the terminal device, such as moving files, adjusting file sorting, or granting function authorization. Then, based on the adjusted second content, the task is resumed to continue executing subsequent operation steps. In this way, unexpected problems can be corrected through user intervention during task execution, thereby avoiding errors in the operation step chain execution, improving the success rate of the operation step chain execution, and reducing the error rate.

[0082] Furthermore, in one possible implementation, step S305 is specifically implemented as follows:

[0083] Step S3051: In response to the fifth operation, determine the first procedure corresponding to the third operation step;

[0084] Step S3052: Obtain the shortcut key for the third operation step in the first program through the first application, and execute the shortcut key. The shortcut key is associated with the second content.

[0085] For example, in one possible implementation, after modifying the first content to the second content in response to a fourth operation performed by the user, a subsequent third operation step is executed based on the second content. The second content includes an operation shortcut key; that is, the second content is generated based on the fourth operation, meaning at least one operation shortcut key is configured and generated based on the fourth operation. This operation shortcut key is for the first program, specifically the operation shortcut key for the current operation step within the first program, which is the program operated on in the third operation step. Subsequently, when the third operation step is executed, it is performed using the aforementioned operation shortcut key, thereby achieving quick execution of the steps.

[0086] Figure 9 This is a schematic diagram illustrating the execution process of an operation step provided in an embodiment of this disclosure. The following is in conjunction with... Figure 9 To further explain the above process, such as... Figure 9 As shown, exemplarily, the first application sequentially executes operation steps Pm-n, Pm-n+1, ​​Pm-n+2, ..., and operation step Pm (the second operation step) in the operation step chain. Operation steps Pm-n to Pm, for example, are used to set a specific chapter title in the document to a specific text format (e.g., size 2, bold). Operation step Pm-n is used, for example, to select a title; operation step Pm-n+1 is used to select a font size from the dropdown menu; operation step Pm-n+2 is used to select the corresponding font size from the dropdown menu... After n steps, until step Pm is completed, the process of setting a chapter title in the document to a specific text format is achieved. Subsequently, upon receiving the third operation from the user, the aforementioned task is paused. Then, in response to the fourth operation input by the user, the terminal device saves the aforementioned text format to the clipboard and inputs the request "Set chapter title using the format painter, format painter is configured" to the agent via a dialog box. In response to the fifth operation, the terminal device continues to execute the subsequent operation step Pm+1 (the third operation step). When executing this operation step Pm+1, it performs the second operation based on the content, such as setting the chapter title text format by inputting the shortcut key corresponding to the format painter (e.g., ctrl+shift+v) on the keyboard, thereby greatly improving the efficiency of text formatting.

[0087] In this embodiment, the first application can pause the task execution process in response to the user's operation during the task execution process, and readjust and configure the content of the terminal device by responding to the user's further input of a fourth operation and by engaging in further dialogue with the intelligent agent, so that the operation steps can be efficiently executed through operation shortcut keys in subsequent execution steps, thereby further improving the efficiency of automated execution and increasing the flexibility of use.

[0088] In this embodiment, the implementation of steps S301-S302 is the same as that in this disclosure. Figure 2 The implementation methods of steps S101-S102 in the illustrated embodiment are the same, and will not be described in detail here.

[0089] Corresponding to the terminal function control method in the above embodiments, Figure 10 This is a structural block diagram of a terminal function control device provided in an embodiment of this disclosure. The method described in the above embodiments can be executed by this terminal function control device, which can be implemented by software and / or hardware, and can be integrated into an electronic device with certain data processing capabilities. The electronic device may include, but is not limited to, mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0090] For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 10 The terminal function control device 4 includes:

[0091] Interaction module 41 is used to receive a first instruction, the first instruction including a task described in natural language, the task being executed within a second application of the terminal device;

[0092] The processing module 42 is used to process the first instruction through the first application, execute the operation steps corresponding to the task, and display the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

[0093] According to one or more embodiments of this disclosure, when the processing module 42 processes the first instruction through the first application and executes the operation steps corresponding to the task, it is specifically used to: perform semantic understanding of the first instruction through the first application to obtain the task target; and execute the operation step chain based on the task target and the device parameters of the terminal device, wherein the device parameters represent the heterogeneous operation unit of the terminal device, the operation step chain includes at least two operation steps, and the operation steps are matched with the heterogeneous operation unit.

[0094] According to one or more embodiments of this disclosure, the processing module 42 is specifically configured to: generate at least two operation targets based on a first instruction, wherein the at least two operation targets are used to implement the task; and sequentially perform the following operations for each operation target: identify the current interactive interface of the terminal device to obtain an identification result, wherein the identification result characterizes the objects displayed in the current interactive interface and the positional relationship between the objects; and, based on the identification result and the current operation target, call a matching heterogeneous operation unit to execute the corresponding operation steps to achieve the current operation target.

[0095] According to one or more embodiments of this disclosure, when the processing module 42 executes an operation step chain based on the task objective and the device parameters of the terminal device, it is specifically used to: obtain the operation objective corresponding to the current operation step; obtain the running status of candidate operation units according to the operation objective and the device parameters, wherein the candidate operation units are heterogeneous operation units used to achieve the operation objective; determine the first operation unit according to the running status of the candidate operation units, and execute the operation steps matching the operation objective based on the first operation unit.

[0096] According to one or more embodiments of this disclosure, the processing module 42 is further configured to: display first information on a first interactive interface, the first information including at least one of the following: the name of the task, the operation step chain corresponding to the task, and the execution progress of the task.

[0097] According to one or more embodiments of this disclosure, the processing module 42 is further configured to: in response to the first operation, configure the task in the execution state to a paused state, and / or, restore the task in the paused state to the execution state.

[0098] According to one or more embodiments of this disclosure, the processing module 42 is further configured to: display second information on the first interactive interface, the second information including at least one of the following: the step name of the operation step, the execution progress of the operation step, and the execution details of the operation step.

[0099] According to one or more embodiments of this disclosure, when the processing module 42 displays the first information on the first interactive interface, it is specifically used to: display the first information in the first window of the first interactive interface; when the processing module 42 displays the second information on the first interactive interface, it is specifically used to: in response to a second operation, display the second information in the second window of the first interactive interface, wherein the second operation includes an operation on the first operation step in the first information, and the second information is associated with the first operation step.

[0100] According to one or more embodiments of this disclosure, the processing module 42 is further configured to: display a third window and a fourth window on the second interactive interface; when the processing module 42 performs the operation steps corresponding to the task, it is specifically configured to: input the object presented in the third window into the fourth window.

[0101] According to one or more embodiments of this disclosure, the processing module 42 is further configured to: in response to a third operation, suspend the execution of the task at the second operation step position in the operation step chain; in response to a fourth operation, configure the first content of the terminal device and generate the second content, wherein the first content is associated with the third operation step and the third operation step is executed after the second operation step; and in response to a fifth operation, continue to execute the task on the terminal device based on the second content.

[0102] According to one or more embodiments of this disclosure, when the processing module 42 processes the first instruction through the first application and executes the operation steps corresponding to the task, it is specifically used to: determine the first program corresponding to the current operation step; obtain the operation shortcut key of the current operation step in the first program through the first application, and execute the operation shortcut key.

[0103] The interaction module 41 and the processing module 42 are connected. The terminal function control device 4 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0104] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 11 As shown, the electronic device 4 includes:

[0105] Processor 51, and memory 52 communicatively connected to processor 51;

[0106] Memory 52 stores instructions executed by the computer;

[0107] The processor 51 executes computer-executable instructions stored in the memory 52 to achieve, for example, Figures 2-9 The terminal function control method in the illustrated embodiment.

[0108] Optionally, the processor 51 and the memory 52 are connected via a bus 53.

[0109] For relevant instructions, please refer to the corresponding text. Figures 2-9 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0110] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-9 The terminal function control method provided in any of the corresponding embodiments.

[0111] This disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements this disclosure. Figures 2-9 The terminal function control method provided in any of the corresponding embodiments.

[0112] To implement the above embodiments, this disclosure also provides an electronic device.

[0113] refer to Figure 12The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0114] like Figure 12 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0115] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0116] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0117] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0118] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0119] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0120] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the specific unit itself.

[0123] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] In a first aspect, according to one or more embodiments of this disclosure, a terminal function control method is provided, comprising:

[0126] The system receives a first instruction, which includes a task described in natural language. The task is executed within a second application on the terminal device. The system processes the first instruction through the first application, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation units of the terminal device.

[0127] According to one or more embodiments of this disclosure, a first instruction is processed by a first application to execute operation steps corresponding to a task, including: semantically understanding the first instruction through the first application to obtain a task target; and executing an operation step chain based on the task target and device parameters of a terminal device, wherein the device parameters characterize heterogeneous operation units of the terminal device, and the operation step chain includes at least two operation steps, and the operation steps are matched with heterogeneous operation units.

[0128] According to one or more embodiments of this disclosure, an operation step chain is executed based on a task objective and device parameters of a terminal device, including: obtaining the operation objective corresponding to the current operation step; obtaining the running status of candidate operation units according to the operation objective and device parameters, wherein the candidate operation units are heterogeneous operation units used to achieve the operation objective; determining a first operation unit according to the running status of the candidate operation units, and executing operation steps matching the operation objective based on the first operation unit.

[0129] According to one or more embodiments of this disclosure, a first application processes a first instruction to execute operation steps corresponding to a task, including: generating at least two operation targets based on the first instruction, wherein the at least two operation targets are used to achieve the task; and performing the following operations sequentially for each operation target: identifying the current interactive interface of the terminal device to obtain an identification result, wherein the identification result characterizes the objects displayed in the current interactive interface and the positional relationships between the objects; and, based on the identification result and the current operation target, calling a matching heterogeneous operation unit to execute the corresponding operation steps to achieve the current operation target.

[0130] According to one or more embodiments of this disclosure, the method further includes: displaying first information on a first interactive interface, the first information including at least one of the following: the name of the task, the operation step chain corresponding to the task, and the execution progress of the task.

[0131] According to one or more embodiments of this disclosure, the method further includes: in response to a first operation, configuring a task in the execution state to a paused state, and / or resuming a task in the paused state to the execution state.

[0132] According to one or more embodiments of this disclosure, the method further includes: displaying second information on a first interactive interface, the second information including at least one of the following: the step name of the operation step, the execution progress of the operation step, and the execution details of the operation step.

[0133] According to one or more embodiments of this disclosure, displaying first information on a first interactive interface includes: displaying the first information in a first window of the first interactive interface; displaying second information on the first interactive interface includes: displaying the second information in a second window of the first interactive interface in response to a second operation, wherein the second operation includes an operation on a first operation step in the first information, and the second information is associated with the first operation step.

[0134] According to one or more embodiments of this disclosure, the method further includes: displaying a third window and a fourth window on a second interactive interface; performing operation steps corresponding to the task, including: inputting the object presented in the third window into the fourth window.

[0135] According to one or more embodiments of this disclosure, the method further includes: in response to a third operation, pausing task execution at a second operation step position in the operation step chain; in response to a fourth operation, configuring first content on the terminal device to generate second content, the first content being associated with the third operation step, the third operation step being executed after the second operation step; and in response to a fifth operation, continuing task execution on the terminal device based on the second content.

[0136] According to one or more embodiments of this disclosure, processing a first instruction through a first application and executing operation steps corresponding to a task includes: determining a first program corresponding to the current operation step; obtaining an operation shortcut key for the current operation step in the first program through the first application, and executing the operation shortcut key.

[0137] Secondly, according to one or more embodiments of this disclosure, a terminal function control device is provided, comprising:

[0138] An interaction module is used to receive a first instruction, which includes a task described in natural language, and the task is executed within a second application of the terminal device.

[0139] The processing module is used to process the first instruction through the first application, execute the operation steps corresponding to the task, and display the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation units of the terminal device.

[0140] According to one or more embodiments of this disclosure, when the processing module processes the first instruction through the first application and executes the operation steps corresponding to the task, it is specifically used to: perform semantic understanding of the first instruction through the first application to obtain the task target; and execute the operation step chain based on the task target and the device parameters of the terminal device, wherein the device parameters represent the heterogeneous operation unit of the terminal device, the operation step chain includes at least two operation steps, and the operation steps are matched with the heterogeneous operation unit.

[0141] According to one or more embodiments of this disclosure, when the processing module executes an operation step chain based on the task objective and the device parameters of the terminal device, it is specifically used to: obtain the operation objective corresponding to the current operation step; obtain the running status of candidate operation units according to the operation objective and the device parameters, wherein the candidate operation units are heterogeneous operation units used to achieve the operation objective; determine a first operation unit according to the running status of the candidate operation units, and execute operation steps matching the operation objective based on the first operation unit.

[0142] According to one or more embodiments of this disclosure, the processing module is specifically configured to: generate at least two operation targets based on a first instruction, wherein the at least two operation targets are used to implement the task; and sequentially perform the following operations for each operation target: identify the current interactive interface of the terminal device to obtain an identification result, wherein the identification result characterizes the objects displayed in the current interactive interface and the positional relationship between the objects; and, based on the identification result and the current operation target, call a matching heterogeneous operation unit to execute the corresponding operation steps to achieve the current operation target.

[0143] According to one or more embodiments of this disclosure, the processing module is further configured to: display first information on a first interactive interface, the first information including at least one of the following: the name of the task, the operation step chain corresponding to the task, and the execution progress of the task.

[0144] According to one or more embodiments of this disclosure, the processing module is further configured to: in response to a first operation, configure a task in the execution state to a paused state, and / or restore a task in the paused state to the execution state.

[0145] According to one or more embodiments of this disclosure, the processing module is further configured to: display second information on a first interactive interface, the second information including at least one of the following: the step name of the operation step, the execution progress of the operation step, and the execution details of the operation step.

[0146] According to one or more embodiments of this disclosure, when the processing module displays first information on the first interactive interface, it is specifically used to: display the first information in a first window of the first interactive interface; when the processing module displays second information on the first interactive interface, it is specifically used to: in response to a second operation, display the second information in a second window of the first interactive interface, wherein the second operation includes an operation on a first operation step in the first information, and the second information is associated with the first operation step.

[0147] According to one or more embodiments of this disclosure, the processing module is further configured to: display a third window and a fourth window on the second interactive interface; when the processing module performs the operation steps corresponding to the task, it is specifically configured to: input the object presented in the third window into the fourth window.

[0148] According to one or more embodiments of this disclosure, the processing module is further configured to: in response to a third operation, suspend the execution of the task at the second operation step position in the operation step chain; in response to a fourth operation, configure the first content of the terminal device and generate the second content, wherein the first content is associated with the third operation step and the third operation step is executed after the second operation step; and in response to a fifth operation, continue to execute the task on the terminal device based on the second content.

[0149] According to one or more embodiments of this disclosure, when the processing module processes the first instruction through the first application and executes the operation steps corresponding to the task, it is specifically used to: determine the first program corresponding to the current operation step; obtain the operation shortcut key of the current operation step in the first program through the first application, and execute the operation shortcut key.

[0150] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0151] The memory stores the instructions that the computer executes;

[0152] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the terminal function control method as described in the first aspect above and various possible designs of the first aspect.

[0153] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the terminal function control method described in the first aspect and various possible designs of the first aspect.

[0154] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the terminal function control method described in the first aspect above and various possible designs of the first aspect.

[0155] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0156] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0157] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A terminal function control method, comprising: Receive a first instruction, the first instruction comprising a task described in natural language, the task being executed within a second application of the terminal device; The first application processes the first instruction, executes the operation steps corresponding to the task, and displays the execution results corresponding to the operation steps. The operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

2. The method according to claim 1, wherein the step of processing the first instruction through the first application and executing the operation step corresponding to the task includes: The first application performs semantic understanding on the first instruction to obtain the task objective; Based on the task objective and the device parameters of the terminal device, an operation step chain is executed, wherein the device parameters characterize the heterogeneous operation unit of the terminal device, and the operation steps are matched with the heterogeneous operation unit.

3. The method according to claim 1, wherein the step of processing the first instruction through the first application and executing the operation step corresponding to the task includes: Obtain the operation target corresponding to the current operation step; Based on the operation objective and device parameters, the operating status of candidate operation units is obtained, wherein the candidate operation units are heterogeneous operation units used to achieve the operation objective. Based on the operating status of the candidate operation units, a first operation unit is determined, and operation steps matching the operation target are executed based on the first operation unit.

4. The method according to claim 1, wherein the step of processing the first instruction through the first application and executing the operation step corresponding to the task includes: Based on the first instruction, at least two operation targets are generated, and the at least two operation targets are used to implement the task; For each of the aforementioned operational objectives, perform the following operations in sequence: The current interactive interface of the terminal device is identified to obtain an identification result, which represents the objects displayed in the current interactive interface and the positional relationship between the objects; Based on the identification results and the current operation objective, the matching heterogeneous operation unit is invoked to execute the corresponding operation steps in order to achieve the current operation objective.

5. The method according to claim 1, further comprising: The first interactive interface displays first information, which includes at least one of the following: The name of the task, the corresponding operation steps of the task, and the execution progress of the task.

6. The method according to claim 5, further comprising: In response to the first operation, the task in the execution state is configured to a paused state, and / or the task in the paused state is restored to the execution state.

7. The method according to claim 5, further comprising: The second information is displayed on the first interactive interface, and the second information includes at least one of the following: The operation steps are named, the execution progress of each operation step is described, and the execution details of each operation step are described.

8. The method according to claim 7, wherein displaying the first information on the first interactive interface includes: The first information is displayed in the first window of the first interactive interface; The display of the second information on the first interactive interface includes: In response to the second operation, second information is displayed in the second window of the first interactive interface. The second operation includes an operation on the first operation step in the first information, and the second information is associated with the first operation step.

9. The method according to claim 1, further comprising: The third and fourth windows are displayed in the second interactive interface; The operation steps for performing the task include: The objects presented in the third window are entered into the fourth window.

10. The method according to claim 1, further comprising: In response to the third operation, the task is paused at the second operation step position; In response to the fourth operation, the first content of the terminal device is configured, and the second content is generated, the first content being associated with the third operation step, the third operation step being executed after the second operation step; In response to the fifth operation, the task continues to be performed on the terminal device based on the second content.

11. The method according to claim 1, wherein the step of processing the first instruction through a first application and executing the operation step corresponding to the task includes: Determine the first procedure corresponding to the current operation step; The shortcut key for the current operation step in the first program is obtained through the first application, and the shortcut key is executed.

12. A terminal function control device, comprising: An interaction module is configured to receive a first instruction, the first instruction comprising a task described in natural language, the task being executed within a second application of a terminal device; The processing module is configured to process the first instruction through a first application, execute the operation steps corresponding to the task, and display the execution results corresponding to the operation steps, wherein the operation steps are used to simulate user operations applied to the terminal device, and the user operations are implemented based on the heterogeneous operation unit of the terminal device.

13. An electronic device, comprising: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the terminal function control method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions, wherein when a processor executes the computer-executable instructions, the terminal function control method as described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program that, when executed by a processor, implements the terminal function control method as described in any one of claims 1 to 11.

Images