Operation execution method and device

Abstract

The invention discloses an operation execution method and device, and belongs to the field of automation. The method comprises the following steps: receiving first information input by a user, wherein the first information comprises at least one of voice information or text information; in response to the first information, performing semantic analysis on the first information to obtain an operation instruction sequence corresponding to the first information; based on the operation instruction sequence, acquiring DCDL data of an operation object indicated by each operation instruction in the operation instruction sequence; and sequentially executing the operation instructions in the operation instruction sequence based on the DCDL data.

Description

Technical Field

[0001] The present application belongs to the field of automation, and specifically relates to an operation execution method and device thereof. Background Art

[0002] With the development of science and technology, users can now use electronic devices by inputting commands to them, allowing them to understand the semantics of the commands, make decisions, and respond accordingly. For example, a mobile phone's voice assistant can complete intelligent interaction by receiving user voice commands.

[0003] In related technology, if a user requires an electronic device to perform a continuous operation, such as adding a Bluetooth headset to a shopping cart in shopping app A, the user triggers the electronic device to display the search interface of shopping app A, enters "Bluetooth headset" in the search box of the search interface, triggers the electronic device to display a product list interface for Bluetooth headsets, and then selects a Bluetooth headset and adds it to the shopping cart to complete the continuous operation. The entire process is too cumbersome and time-consuming. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an operation execution method and device thereof, which can automatically understand multiple continuous operation instructions input by a user and accurately execute the continuous operation instructions.

[0005] In the first aspect, an embodiment of the present application provides an operation execution method, which includes: receiving first information input by a user, the first information including at least one of voice information or text information; in response to the above-mentioned first information, performing semantic analysis on the above-mentioned first information to obtain an operation instruction sequence corresponding to the above-mentioned first information; based on the operation instruction sequence, obtaining DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence; based on the DCDL data, executing the operation instructions in the operation instruction sequence in sequence.

[0006] In the second aspect, an embodiment of the present application provides an operation execution device, which includes: a receiving module, a processing module and an acquisition module; the receiving module is used to receive first information input by a user, and the first information includes at least one of voice information or text information; the above-mentioned processing module is used to respond to the above-mentioned first information, perform semantic analysis on the above-mentioned first information, and obtain an operation instruction sequence corresponding to the above-mentioned first information; the above-mentioned acquisition module is used to obtain DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence based on the operation instruction sequence; the above-mentioned processing module is also used to execute the operation instructions in the operation instruction sequence in sequence based on the DCDL data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, an electronic device receives first information input by a user, and the first information includes at least one of voice information or text information; in response to the first information, the first information is semantically analyzed to obtain an operation instruction sequence corresponding to the first information; based on the operation instruction sequence, DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence is obtained; based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence. In this solution, the user triggers the electronic device to directly perform semantic analysis on the continuous operation instruction by inputting text information or voice information containing a continuous operation instruction, thereby obtaining DCDL data for describing the operation object indicated by each operation instruction in the continuous operation instruction, and then the electronic device can automatically execute the corresponding operation instruction according to the DCDL data of the operation object indicated by each operation instruction. In this way, the human-computer interaction step is saved by automatically executing the continuous operation instructions input by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an operation execution method provided in an embodiment of the present application;

[0013] Figure 2 This is a schematic diagram of an execution flow of an operation instruction sequence provided by an embodiment of the present application;

[0014] Figure 3 This is a schematic diagram of the structure of an operation execution device provided in an embodiment of the present application;

[0015] Figure 4This is a schematic diagram of the structure of an operation execution device provided in an embodiment of the present application;

[0016] Figure 5 This is a schematic diagram of the structure of an operation execution device provided in an embodiment of the present application;

[0017] Figure 6 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0018] Figure 7 This is the second hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0020] Some terms and nouns involved in the embodiments of this application are explained below.

[0021] 1) The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, the term "and / or" in this specification indicates at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0022] 2) The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the items included. For example, at least one of a, b, and c can be represented by: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one."

[0023] The logo in this application refers to text, symbols, images, etc. used to indicate information. Logos or other containers can be used as carriers for displaying information, including but not limited to text logos, image logos, symbol logos, etc.

[0024] It should be noted that the operation execution method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and in-vehicle electronic devices. In some embodiments of the present application, the operation execution method provided in the embodiments of the present application is described by taking an electronic device as the execution subject to execute the operation execution method.

[0025] The following describes in detail the operation execution method and device provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0026] It is understandable that the operation execution method provided in the embodiments of the present application can be applied to any of the following scenarios:

[0027] 1. Accessibility assistance: Provides automated operations based on voice-control mapping for visually impaired users;

[0028] 2. Automated testing: replaces manual application compatibility test script generation;

[0029] 3. Internet of Things (IoT) device control: Smart home appliance control applications through mobile phones.

[0030] In the context of barrier-free assistance, when a user requires an electronic device to automatically execute a single operation instruction, for example, the input text message or voice message is to open shopping application A. For example, the user can enter "open shopping application A" in the electronic device's smart assistant. The electronic device parses the text message input by the user and obtains the operation instruction to display the shopping application interface. At this point, the electronic device automatically runs shopping application A and displays the interface of shopping application A. It is understandable that the above-mentioned "input in the smart assistant" can be manual input or voice input. Generally, the above solution is applicable to voice input scenarios.

[0031] However, in the context of accessibility assistance, when a user requires an electronic device to automatically execute continuous operation instructions, for example, inputting a text message or voice message such as opening shopping app A, searching for a Bluetooth headset, and adding it to the shopping cart, the following two solutions can be used:

[0032] Solution 1: The user needs to manually operate the electronic device to trigger the electronic device to perform corresponding operations based on the input.

[0033] For example, the user clicks on the icon of shopping application A, triggering the electronic device to display the search interface of shopping application A. The user then enters "Bluetooth headset" in the search box in the search interface to trigger the electronic device to display the product list interface of Bluetooth headsets. Finally, the user clicks and enters to select one of the Bluetooth headsets, triggering the electronic device to add it to the shopping cart.

[0034] In this way, the user needs to input the operation instructions multiple times, and the user still needs to constantly perform manual operations, which makes it impossible to truly realize the automatic execution of the operation instructions, and the process is too cumbersome.

[0035] Solution 2: Electronic devices use accessibility services to pre-record operation scripts, that is, fixed operation videos of certain instructions.

[0036] For example, a user first records an action script for "opening shopping app A, searching for Bluetooth headphones, and adding them to the shopping cart." This script involves the user clicking on shopping app A, entering "Bluetooth headphones" in the search box, and adding the item to the shopping cart. The script also records the user's touch locations and touch methods on the electronic device screen. Then, when the electronic device receives the user's input for "opening shopping app A, searching for Bluetooth headphones, and adding them to the shopping cart," it automatically executes the input according to the location and method recorded in the script, achieving the effect of automatically executing the input command.

[0037] In this way, after recording the operation script, if the position of the control in the interface of shopping application A changes, there will be no control to be triggered at the original input position, so that the operation instruction cannot be executed, resulting in operation interruption.

[0038] Therefore, it is difficult for existing solutions to achieve compatibility between dynamic intent understanding and interface changes.

[0039] In the operation execution method provided in the embodiment of the present application, the electronic device receives first information input by the user, and the first information includes at least one of voice information or text information; in response to the first information, the first information is semantically analyzed to obtain an operation instruction sequence corresponding to the first information; based on the operation instruction sequence, the DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence is obtained; based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence. In this solution, the user triggers the electronic device to directly perform semantic analysis on the continuous operation instruction by inputting text information or voice information containing a continuous operation instruction, thereby obtaining DCDL data for describing the operation object indicated by each operation instruction in the continuous operation instruction, and then the electronic device can automatically execute the corresponding operation instruction according to the DCDL data of the operation object indicated by each operation instruction. In this way, the human-computer interaction step is saved by automatically executing the continuous operation instructions input by the user.

[0040] The execution subject of the operation execution method provided in the embodiments of the present application may be an operation execution device. For example, the operation execution device may be an electronic device, or a component within the electronic device, such as an integrated circuit or chip. The operation execution method provided in the embodiments of the present application will be described below using an electronic device as an example.

[0041] The embodiment of the present application provides an operation execution method, Figure 1 FIG1 shows a flow chart of an operation execution method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 1 As shown, the operation execution method provided in the embodiment of the present application may include the following steps 201 to 204.

[0042] Step 201: The electronic device receives first information input by a user.

[0043] In some embodiments of the present application, the first information includes at least one of voice information or text information.

[0044] In some embodiments of the present application, the electronic device can obtain voice information input by the user by receiving the user's voice input.

[0045] For example, the electronic device may obtain the voice information input by the user by recording the user's voice in the smartphone assistant.

[0046] In some embodiments of the present application, the electronic device can obtain text information input by the user by receiving the user's touch input.

[0047] For example, the electronic device may receive a user's touch input in a smartphone assistant and display the text information input by the user.

[0048] In some embodiments of the present application, the first information is used to represent the user's operation intention.

[0049] In some embodiments of the present application, the electronic device can obtain the user's operation intention by parsing the first information.

[0050] For example, the first information input by the user is "I want to view Bluetooth headsets in shopping application A", and the electronic device analyzes the user operation intention as "run shopping application A, search for Bluetooth headsets in the search page, and display the product list interface of Bluetooth headsets."

[0051] Step 202: The electronic device performs semantic analysis on the first information in response to the first information to obtain an operation instruction sequence corresponding to the first information.

[0052] In some embodiments of the present application, the above-mentioned operation instruction sequence is a parsed continuous operation instruction that is used to represent the execution of the electronic device required by the user.

[0053] In some embodiments of the present application, the above-mentioned operation instruction sequence includes at least one operation instruction.

[0054] In some embodiments of the present application, the above-mentioned operation instruction sequence can be obtained based on the text input by the user, or can be obtained based on the voice input by the user, or can be obtained in other forms.

[0055] In some embodiments of the present application, an electronic device can obtain a piece of text information by receiving text input or voice input from a user. The electronic device then inputs the text information into a large language model and extracts key text, such as keywords, through the model's prompts. The key text is then further decomposed into tasks using the model and a sequence of operation instructions is output. It should be noted that the above-mentioned sequence of operation instructions can be understood as a structured intent used to characterize the user's operation intention.

[0056] It can be understood that the above operation instruction sequence is a group of continuous operation instructions.

[0057] For example, the prompt word template may be:

[0058] "You are an application operation assistant, good at summarizing and refining keywords related to application operations, and removing redundant words such as modal words, adjectives, adverbs, etc. in the text that are irrelevant to application operations or interface descriptions. You can refer to the following keywords that should be extracted and redundant words that should be removed.

[0059] Keyword dictionary: [Keyword A, Keyword B, Keyword C...]

[0060] Redundant Word Dictionary: [Uh, ah, oh, trouble, hello...]

[0061] User input text: [text content].

[0062] Output command text: [command text].

[0063] For example, taking the example of an electronic device receiving user voice input "I want to search for Bluetooth headphones in shopping application A", the electronic device first converts the user's input voice into text information "I want to search for Bluetooth headphones in shopping application A", and then extracts the key text by inputting the text information into the prompt word template of the large language model.

[0064] Specifically, the specific content after the text information is entered into the prompt word template is as follows:

[0065] "You are an application operation assistant, good at summarizing and refining keywords related to application operations, and removing redundant words such as modal words, adjectives, adverbs, etc. in the text that are irrelevant to application operations or interface descriptions. You can refer to the following keywords that should be extracted and redundant words that should be removed.

[0066] Keyword dictionary: [Keyword A, Keyword B, Keyword C...]

[0067] Redundant Word Dictionary: [Uh, ah, oh, trouble, hello...]

[0068] The user enters text: [I want to search for Bluetooth headsets in shopping app A].

[0069] Output command text: [Run shopping app A to search for Bluetooth headsets].

[0070] Next, the instruction text obtained through the prompt word "Run shopping app A to search for Bluetooth headsets" is decomposed into tasks using a large language model, and the final operation instruction sequence is obtained: run shopping app A, locate the search box, enter Bluetooth headsets, and trigger the search.

[0071] In some embodiments of the present application, the electronic device performs semantic analysis on each operation instruction in the operation instruction sequence and obtains a semantic analysis result corresponding to each operation instruction.

[0072] In some embodiments of the present application, the above-mentioned semantic analysis results are used to characterize the execution behavior of the corresponding operation instruction, operation object and other information.

[0073] In some embodiments of the present application, the above semantic analysis results include but are not limited to: semantic keywords and text features.

[0074] In some embodiments of the present application, the electronic device may perform semantic analysis on each operation instruction through a model or semantic analysis technology to obtain a semantic analysis result in each operation instruction.

[0075] Illustratively, the above keywords may be semantic analysis results contained in the operation instruction, or may be semantic keywords associated with the operation instruction.

[0076] For example, when the operation instruction is "open shopping application A", the semantic analysis result can be shopping application A, product, search, etc.

[0077] Step 203: The electronic device obtains DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence based on the operation instruction sequence.

[0078] In some embodiments of the present application, the Dynamic Control Description Language (DCDL) is a standardized structured data format for defining controls in application interfaces. It is a domain-specific language for dynamically defining, configuring, and managing the behavior of user interface controls or system components. It is typically used in scenarios that require dynamically generated interfaces, flexible configuration of control properties, or real-time adjustment of interaction logic, such as low-code platforms, industrial automation control, game UI systems, or complex enterprise applications.

[0079] In some embodiments of the present application, the core features and uses of the DCDL include the following two:

[0080] 1. Dynamic control generation

[0081] Declarative syntax is used to describe the type of control, such as a button, input box, or chart, the layout position of the control, the style of the control, and the initial state of the control, so that the interface can be dynamically rendered at runtime without hard coding.

[0082] 2. Event-driven logic binding

[0083] It supports defining control events, such as clicks and input changes, and associating them with backend logic, such as interface calls and data processing, to achieve dynamic configuration of interaction logic.

[0084] In some embodiments of the present application, the above-mentioned DCDL data is used to represent relevant operation information of the operation object.

[0085] For example, when the operation object is an application, the operation instruction requires operating the application icon, such as clicking, long pressing, or dragging.

[0086] For example, when the operation instruction sequence is: open shopping application A, locate the search box, enter Bluetooth headset, and trigger search, the operation objects are: the application icon of shopping application A, the search box on the search interface, the text input area, and the search control.

[0087] In some embodiments of the present application, the relevant operation information of the above-mentioned operation object, that is, the above-mentioned DCDL data includes but is not limited to: semantic description information of the operation object, coordinates of the operation object, operation type of the operation object, name of the operation object, attributes of the operation object, etc.

[0088] Exemplarily, the above DCDL data can be understood as semantic description information of different operation objects through natural language.

[0089] In some embodiments of the present application, the above-mentioned DCDL data includes: DCDL data in the application identification layer, DCDL data in the interface description layer, and DCDL data in the control description layer.

[0090] It is understandable that the above DCDL data may be DCDL data of an application, DCDL data of an interface, or DCDL data of a control.

[0091] In some embodiments of the present application, the above DCDL data may be stored in a DCDL database.

[0092] In some embodiments of the present application, the DCDL database may include an application identification layer, an interface description layer, and a control description layer, wherein each data layer contains DCDL data for representing relevant operation information of the operation object.

[0093] In some embodiments of the present application, the DCDL data in the above-mentioned application identification layer includes but is not limited to: application package name (app), application description (app_description), and screen resolution (resolution).

[0094] In some embodiments of the present application, the above application package name is used to achieve unique identification of the target application.

[0095] Exemplarily, the application package name may adopt a standard package name, such as com.example.shopping, or directly adopt the application name, such as A application.

[0096] In some embodiments of the present application, the above application description provides semantic assistance for electronic devices to obtain DCDL data of applications.

[0097] For example, the above application description is to describe the application functional attributes through natural language. For example, if application A is a shopping application, then the application description of application A is a shopping application.

[0098] In some embodiments of the present application, the above screen resolution is used as a basis for coordinate conversion.

[0099] Exemplarily, the above screen resolution may be the screen resolution of the electronic device display screen when a certain application generates DCDL data, such as 1080x2400.

[0100] In this way, electronic devices use a dual positioning mechanism based on the application package name and application description to ensure the uniqueness of the target application indicated by the operation instruction. This mechanism also provides semantic auxiliary information for model decision-making, allowing the electronic device to accurately determine the operation object based on the operation instruction. Furthermore, the electronic device determines the coordinate position of the operation object by referring to the screen resolution in the application identification layer, thus resolving the problem of operation offset caused by different screen sizes.

[0101] In some embodiments of the present application, the DCDL data in the above-mentioned interface description layer includes but is not limited to: interface identifier (activity_id), interface name (activity), interface semantic description (description), and semantic keywords (dcdl_keywords).

[0102] In some embodiments of the present application, the above-mentioned interface identifier is a system-level unique identifier used to track interface version changes.

[0103] Exemplarily, the above interface identifier may be an interface version number, such as Activity 1.3.2.

[0104] In some embodiments of the present application, the electronic device obtains the interface version of the current interface by determining the interface identifier. If the interface identifier of the current interface is consistent with the identifier recorded in the DCDL data, it indicates that the current interface has not been updated. If the interface identifier of the current interface is inconsistent with the identifier recorded in the DCDL data, it indicates that the current interface has been updated. Since the layout and controls of the current interface may change, the electronic device needs to re-acquire the DCDL data of the current interface.

[0105] In some embodiments of the present application, the above-mentioned interface name can be the class name of the interface, such as home page, middle interface (MainActivity), search interface, etc.

[0106] In some embodiments of the present application, the above interface semantic description is a description of the interface function through natural language, such as the product search interface, payment interface, product details page, etc.

[0107] In some embodiments of the present application, the above-mentioned interface semantic description can be combined with the application package name of the application identification layer, the interface identifier, and the description of the interface function through natural language to construct a three-dimensional identification system, so that the electronic device can accurately obtain other DCDL data by detecting one of them.

[0108] In this way, the interface description layer constructs a three-dimensional identification system of the interface space through the association mapping of interface identifiers, interface names and interface semantic descriptions, and realizes the compatibility identification of cross-version interface changes.

[0109] In some embodiments of the present application, the control description layer is used to represent the location, function, and operation logic-related fields of the control.

[0110] In some embodiments of the present application, the DCDL data in the above-mentioned control description layer includes but is not limited to: control name (ID), control type (type), display text (text), control coordinate (bounds) information, operation type (action), verification condition (validation), and alternative positioning strategy (alternative_selectors).

[0111] In some embodiments of the present application, the above-mentioned control name is a unique identifier of the control, such as a search control (search_bar), which is used for logical reference.

[0112] In some embodiments of the present application, the above-mentioned control semantic description is a description of the control function through natural language, such as a product search input box.

[0113] In some embodiments of the present application, the above control types are used to characterize control functions.

[0114] Exemplarily, the above control types include but are not limited to: text input area (EditText), radio option group (RadioButton), button (Button), vertical scrolling list (ListView), image display (ImageView), progress indicator (ProgressBar).

[0115] In some embodiments of the present application, the above operation types are used to define the execution actions of the controls.

[0116] Exemplarily, the above operation types include but are not limited to: inputting text, checking operation, clicking to trigger immediate operation, double-clicking to trigger immediate operation, vertical sliding, long pressing operation, and inoperable operation.

[0117] It can be understood that one control type corresponds to at least one operation type.

[0118] For example, if the control type is a text input area, such as a search box, the operation type is input text, meaning the electronic device automatically enters the search text in the search box. If the control type is a radio option group, such as a gender checkbox group, the operation type is a check operation, meaning the electronic device performs a check operation on the checkbox for one of the genders. If the control type is a button, such as a search button, the operation type is click to trigger an immediate operation, meaning the electronic device triggers a search operation by clicking the search button.

[0119] In some embodiments of the present application, the above-mentioned display text is the text content currently displayed by the control, such as "Search for Products".

[0120] In some embodiments of the present application, the above-mentioned control coordinate information is used to indicate the position of the control in the interface.

[0121] Exemplarily, the above-mentioned control coordinate information can be the absolute coordinates converted from the pixel value array, such as [x1, y1], to cooperate with the dynamic coordinate conversion algorithm to achieve cross-device adaptation, that is, the reference coordinates obtained by the screen resolution are used to find the position of the control in the reference coordinates based on the absolute coordinates.

[0122] In some embodiments of the present application, the above-mentioned backup positioning strategy is a strategy for enabling a multi-path positioning solution when the control coordinate positioning fails.

[0123] In some embodiments of the present application, the above-mentioned alternative positioning strategies include but are not limited to: hierarchical path positioning and alternative text identification.

[0124] In some embodiments of the present application, the hierarchical path positioning is to obtain the location of the control through the storage path of the data of the control in the system.

[0125] For example, the above positioning path may be / / android.widget.EditText.

[0126] In some embodiments of the present application, the above-mentioned alternative text identifier is a text identifier of a control in the search interface to obtain the location of the control.

[0127] It is understandable that only when the control fails, the electronic device will use the backup text identification solution to locate the control position.

[0128] In some embodiments of the present application, the above verification conditions are used to define the expected results after the operation is executed.

[0129] Exemplarily, the expected result may be that the execution result of the operation instruction is correct, or that the execution result of the operation instruction is incorrect.

[0130] In some embodiments of the present application, the above verification conditions include two verification dimensions: interface jump mark (post_condition) and visual feature markers (visual_markers).

[0131] In some embodiments of the present application, the above-mentioned interface jump identifier is used to describe the interface features of the expected jump.

[0132] For example, if the interface jump indicator is "the shopping cart interface (CartActivity) appears", when the electronic device detects the shopping cart interface, it is considered that the execution result of the "add to shopping cart" operation instruction is correct. Alternatively, when the electronic device does not detect the shopping cart interface, it is considered that the execution result of the "add to shopping cart" operation instruction is incorrect.

[0133] In some embodiments of the present application, the above-mentioned visual feature marks are used to mark visual elements that need to be verified.

[0134] For example, the visual feature mark is a highlighted shopping cart icon mark. When the electronic device detects the presence of a shopping cart icon in the interface, the shopping cart icon is highlighted, so that after the electronic device recognizes the highlighted shopping cart icon, it determines that the execution result of the "add to shopping cart" operation instruction is correct. Alternatively, when the electronic device detects that there is no shopping cart icon in the interface, the electronic device does not recognize the highlighted shopping cart icon and determines that the execution result of the "add to shopping cart" operation instruction is incorrect.

[0135] In this way, electronic devices establish a semantic mapping between each control's unique identifier and its natural language description, enabling them to understand the control's functional intent, such as the input function of a "search box." Furthermore, by obtaining the operation type of each control, the electronic device can obtain the control's executable actions, such as click, input, and slide, and associate them with pre-set operation logic, such as automatically triggering the virtual keyboard to execute the corresponding operation instruction.

[0136] It can be understood that the above-mentioned DCDL data is a structured semantic description system proposed based on DCDL technology. Its idea is to abstract applications, application interfaces and controls into computable and inferable semantic data structures, and realize dynamic perception and execution of cross-interface operations through hierarchical definition. The interface data structure of cross-version applications is standardized and provided to electronic devices for performing automatic operations. In other words, the electronic device stores the DCDL data of each application, the interface of the application and the controls in the interface in the form of a natural language description in the DCDL data, so that the electronic device can match its semantic description information through different semantic analysis results to obtain the operation information of the corresponding operation object, such as the display position, characteristics, name, operation type and other data of the application, the interface of the application and the controls in the interface.

[0137] In some embodiments of the present application, the above-mentioned DCDL data may be DCDL data stored in a DCDL database, or may be DCDL data generated in real time.

[0138] Exemplarily, for a static interface, the electronic device stores the DCDL data of all elements in the static interface in a DCDL database in advance.

[0139] For example, for a dynamic interface, the electronic device will parse each element in real time based on the elements in the dynamically generated interface and generate DCDL data for each element.

[0140] It is understandable that the above elements are any elements in the interface, such as controls, display areas, logos, etc. The operation object can be any of the elements.

[0141] In some embodiments of the present application, the electronic device obtains DCDL data that matches the semantic analysis result of an operation instruction based on the semantic analysis result of the operation instruction.

[0142] In some embodiments of the present application, the electronic device may obtain DCDL data that matches the semantic analysis result of an operation instruction from the DCDL data stored in the DCDL database; or obtain DCDL data that matches the semantic analysis result from the DCDL data generated in real time.

[0143] In some embodiments of the present application, the electronic device can match the semantic description information in the DCDL data based on the semantic analysis result corresponding to an operation instruction, and determine the successfully matched DCDL data as the DCDL data of the operation object indicated by the operation instruction.

[0144] In some embodiments of the present application, the electronic device can match the semantic description information in the DCDL data stored in the DCDL database based on the semantic analysis result corresponding to an operation instruction. If there is DCDL data that matches the semantic analysis result, the DCDL data is directly used as the DCDL data of the operation object indicated by the operation instruction. If there is no DCDL data in the DCDL data stored in the DCDL database that matches the semantic analysis result, the electronic device parses all elements of the current interface, generates DCDL data for all elements, matches the semantic analysis result of the operation instruction with the generated DCDL data, and uses the successfully matched DCDL data as the DCDL data of the operation object indicated by the operation instruction.

[0145] Exemplarily, the electronic device may match semantic keywords in the semantic analysis result of the operation instruction with semantic description information in the DCDL data.

[0146] For example, when the operation instruction is "open shopping application A", its semantic analysis result includes shopping application A. At this time, the electronic device retrieves the semantic description information "shopping application A" in the DCDL data, and uses the corresponding DCDL data of "icon coordinates of shopping application A" and "operation type of shopping application A" as the DCDL data of the operation instruction, so that the electronic device can find the location of shopping application A and "click" to operate the shopping application A according to the operation type to trigger the electronic device to display the display interface of shopping application A.

[0147] It should be noted that when the semantic analysis record of the operation instruction input by the user includes the application name of a shopping application, the electronic device can specify the shopping application and the application name as the semantic keywords of the operation instruction, and match them in the DCDL data through the semantic keywords "shopping application" and "shopping application A". At this time, the electronic device can match the DCDL data of shopping application A in the DCDL database. At this time, the DCDL data obtained by the electronic device includes not only the DCDL data in the application identification layer of the application, but also information in other layers under the application, such as the interface information of the shopping application, or control information in the interface of the shopping application.

[0148] In some embodiments of the present application, when the electronic device obtains DCDL data of an operation object indicated by an operation instruction, it can package the obtained DCDL data into a data packet. That is, a data packet can contain DCDL information of no less than one data layer. For example, a data packet can contain data of the application identification layer of application A and data of the interface description layer of application A, such as interface layout, control location, etc.

[0149] In some embodiments of the present application, the electronic device encapsulates the DCDL data of the operation object indicated by an operation instruction into an independent object, and splices it after the operation instruction, so that when the electronic device encounters the same operation instruction again, it directly parses the DCDL data in the operation instruction and performs the corresponding operation based on the DCDL data obtained by parsing, so that the electronic device does not need to query the DCDL database a second time, thereby improving the execution efficiency of automated continuous operations.

[0150] Step 204: The electronic device sequentially executes the operation instructions in the operation instruction sequence based on the DCDL data.

[0151] In some embodiments of the present application, the electronic device executes a corresponding operation instruction based on DCDL data of an operation object indicated by an operation instruction.

[0152] In some embodiments of the present application, the electronic device sequentially executes corresponding operation instructions in the operation instruction sequence based on the DCDL data of the operation object indicated by each operation instruction.

[0153] In some embodiments of the present application, the electronic device executes each operation instruction in sequence according to the order of the operation instructions in the operation instruction sequence.

[0154] In some embodiments of the present application, the electronic device executes an operation instruction based on DCDL data representing the operation information of the operation object.

[0155] In some embodiments of the present application, the above-mentioned operation information includes but is not limited to: coordinates of the operation object, operation type of the operation object, resolution of the operation object, and screen resolution.

[0156] In some embodiments of the present application, the coordinates of the above-mentioned operation object are used to indicate the position where the electronic device locates the operation object.

[0157] In some embodiments of the present application, the operation type of the above-mentioned operation object is used to indicate the type of operation performed by the electronic device on the operation object, such as single-click, double-click, slide, long press, etc.

[0158] In some embodiments of the present application, the resolution of the operation object and the screen resolution are both used by the electronic device to obtain a more accurate operation position.

[0159] In some embodiments of the present application, when the electronic device receives an operation instruction, it obtains the screen resolution when the DCDL data is generated to obtain a reference coordinate system. When the coordinate system of the operation object corresponding to other DCDL data is different from the reference coordinate system, the coordinate position of the operation object can be adjusted proportionally, so that the electronic device can accurately know the coordinate position of the operation object and then execute the corresponding operation instruction.

[0160] For example, when executing an operation instruction, if the screen resolution of the electronic device is different from the original screen resolution recorded in the DCDL data, if the actual screen resolution and the original screen resolution are in a geometric scaling relationship, the coordinate scaling factor is calculated, and the coordinate (bounds) value is dynamically converted to the current device coordinate system, where each coordinate of the bounds value is * the scaling factor. For example, if the original screen resolution is 1080x2400, the bounds value is [100, 100, 200, 200], and the actual resolution is 540x1200, then the scaling factor is 0.5, and the converted bounds value is [100*0.5, 100*0.5, 200*0.5, 200*0.5]. If it is not a geometric scaling relationship, it is necessary to parse the DCCL structure of the current interface in real time and obtain the actual bounds value. In this way, by applying the screen resolution (resolution) in the identification layer and coordinating with the dynamic coordinate conversion algorithm, the cross-device adaptive positioning of the absolute coordinates of the control is realized, and the click offset problem caused by screen size differences is solved.

[0161] For example, when the operation instruction is to open shopping application A, the electronic device determines that the application icon with semantic description information "shopping application A" is the operation object in the DCDL database based on the semantic analysis result of the operation instruction "shopping application A", and further obtains the coordinates of shopping application A in the DCDL data as [x1, y1]. At this time, the electronic device can determine the position of the application icon of shopping application A based on this, and then, according to the obtained DCDL data, know that the operation type of the application icon of shopping application A is click. Therefore, the electronic device can automatically perform click input at the position [x1, y1] on the desktop to open shopping application A, thereby executing the operation instruction of "open shopping application A".

[0162] For example, the electronic device can obtain the first unfinished operation instruction in the operation instruction sequence, and then obtain the DCDL data of the target interface and controls in the operation instruction, and pass these DCDL data to the operating system to automatically perform the corresponding operation.

[0163] For example, for an interface, the operating system of an electronic device can automatically open the interface based on the operation type of the interface. For a control under the interface, the operating system of the electronic device needs to further locate the control and trigger the system's automatic execution, that is, further obtain the DCDL data of the control and then trigger the operating system to automatically execute the operation instruction based on the DCDL data.

[0164] For example, when the operation instructions are to locate the search box, enter Bluetooth headset, and trigger the search, the search box is first located according to the coordinate position information of the search box, then the keyword "Bluetooth headset" is entered, and finally the search button is automatically clicked to trigger the search.

[0165] In the operation execution method provided in the embodiment of the present application, the electronic device receives first information input by the user, and the first information includes at least one of voice information or text information; in response to the above-mentioned first information, the above-mentioned first information is semantically analyzed to obtain an operation instruction sequence corresponding to the above-mentioned first information; based on the operation instruction sequence, the DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence is obtained; based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence. In this solution, the user triggers the electronic device to directly perform semantic analysis on the continuous operation instruction by inputting text information or voice information containing a continuous operation instruction, thereby obtaining DCDL data for describing the operation object indicated by each operation instruction in the continuous operation instruction, and then the electronic device can automatically execute the corresponding operation instruction according to the DCDL data of the operation object indicated by each operation instruction. In this way, the human-computer interaction step is saved by automatically executing the continuous operation instructions input by the user.

[0166] Optionally, in some embodiments of the present application, the above step 203 can be implemented by the following step 203a or step 203b.

[0167] Step 203a: When there is first DCDL data matching the semantic analysis result of the first operation instruction in the DCDL database, the electronic device uses the first DCDL data as the DCDL data of the operation object indicated by the first operation instruction.

[0168] Step 203b: When there is no DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, the electronic device generates second DCDL data based on the semantic analysis result of the first operation instruction, and uses the second DCDL data as the DCDL data of the operation object indicated by the first operation instruction.

[0169] In some embodiments of the present application, the semantic analysis result of the first operation instruction is a semantic analysis result obtained after the electronic device performs semantic analysis on the first operation instruction. For a specific description, please refer to the description in step 202 above.

[0170] In some embodiments of the present application, the first operation instruction is an operation instruction in the operation instruction sequence.

[0171] In a possible embodiment, when the electronic device retrieves semantic description information that matches the semantic analysis result of the first operation instruction in the DCDL database, it determines the DCDL data indicated by the semantic description information, that is, the above-mentioned first DCDL data, as the DCDL data of the operation object indicated by the operation instruction.

[0172] For example, for static interfaces, such as settings and personal center interfaces, the electronic device can store interface information in the static interface, such as interface layout, interface name, and DCDL data of controls in the interface, in the DCDL database in advance.

[0173] For example, when the operation instruction is to open shopping application A, the electronic device needs to perform fuzzy matching based on the structured intent, that is, the semantic description keyword (dcdl_keyword) in the above operation instruction, and the semantic description information (description) field in the DCDL database to find the matching DCDL data of shopping application A, that is, the application package name (such as com.example.shopapp), screen resolution, etc.

[0174] In another possible embodiment, when the electronic device fails to retrieve semantic description information that matches the semantic analysis result of the first operation instruction in the DCDL database, the electronic device parses the current interface, obtains the operation object in the current interface that matches the semantic analysis result based on the semantic analysis result of the first operation instruction, and then generates DCDL data in real time, and determines the DCDL data as the DCDL data of the operation object indicated by the first operation instruction, that is, the above-mentioned second DCDL data.

[0175] For example, for a dynamic interface, such as a product list interface, the electronic device can generate DCDL data in real time through a DCDL parsing engine.

[0176] Exemplarily, the logic of the DCDL parsing engine is: parse the current interface through the system's accessibility service (AccessibilityService) or an application-specific built-in program, and then pass it to the operating system to obtain the current interface control tree in real time and generate DCDL data.

[0177] For example, when the operation instruction is to locate the search box, since the electronic device has completed the previous operation instruction and displayed the search interface of shopping application A on the electronic device, the electronic device can directly generate the DCDL data of the search interface based on the data information of the search interface, including the DCDL data of all controls contained in the search interface. At this time, the electronic device can use the semantic description keyword "search box" to match the control search box in the search interface according to the semantic description information contained in the DCDL data of the search interface, and then obtain the DCDL data of the search box.

[0178] It should be noted that, in one case, each operation instruction in the operation instruction sequence can find the corresponding DCDL data in the database, and in another case, a part of the operation instructions in the operation instruction sequence can find the corresponding DCDL data in the DCDL database, and the other part of the operation instructions need to obtain the corresponding DCDL data from the DCDL data generated in real time. For example, the electronic device has executed the operation instruction to open shopping application A, and packaged its corresponding DCDL data into a data packet, which includes the interface DCDL data of all interfaces in shopping application A, and the DCDL data of all controls. At this time, since the electronic device has opened the homepage of shopping application A and triggered an action, after entering the next interface, the electronic device can directly obtain the interface ID of the current interface from the data packet, and directly search the DCDL database for data based on the ID. If it exists, the corresponding DCDL data is directly obtained; if it does not exist, the DCDL data is generated in real time by the DCDL parsing engine.

[0179] It should be noted that the process of obtaining DCDL data for each operation instruction in the operation instruction sequence can refer to the description process of the above steps.

[0180] In this way, electronic devices build a two-tier architecture of "pre-stored DCDL database + real-time generation engine", which not only retains the pre-compiled data of high-frequency operation interfaces, but also supports the dynamic generation of temporary interfaces, such as the control description of the product list page.

[0181] Optionally, in some embodiments of the present application, after the above-mentioned step 204 "the electronic device sequentially executes the operation instructions in the operation instruction sequence based on the DCDL data", the operation execution method provided by the embodiments of the present application further includes step 301.

[0182] Step 301: After executing an operation instruction in an operation instruction sequence, the electronic device displays an interface corresponding to the operation instruction.

[0183] In some embodiments of the present application, the interface corresponding to the above-mentioned operation instruction is the interface on which the execution result after executing the operation instruction is displayed.

[0184] In some embodiments of the present application, the interfaces corresponding to each operation instruction may be the same or different.

[0185] In some embodiments of the present application, if the operation instruction is a cross-interface operation instruction, the displayed interface is different from the interface corresponding to the previous operation instruction; if the operation instruction is an operation instruction to perform an operation on the current interface, the displayed interface will not change, but the layout of the interface may change.

[0186] For example, when the operation instruction is "open shopping application A", the interface displayed as the result of the execution of the operation instruction is the main interface of shopping application A; or when the operation instruction is "locate the search box", the interface displayed as the result of the execution of the operation instruction is the search interface of shopping application A; or, when the operation instruction is "input Bluetooth headset", the interface displayed as the result of the execution of the operation instruction is still the search interface of shopping application A, but the content in the search box is displayed as "Bluetooth headset"; or when the operation instruction is "trigger search", the interface displayed as the product list interface of Bluetooth headsets.

[0187] In this way, the electronic device can clearly display the operation process and operation result of the operation instruction to the user each time an operation instruction is executed, so that the user can adjust the operation process in time according to the execution result.

[0188] Optionally, in some embodiments of the present application, after the above-mentioned step 204, the operation execution method provided by the embodiment of the present application further includes steps 401 to 403.

[0189] Step 401: After the electronic device executes the second operation instruction based on the DCDL data of the operation object indicated by the second operation instruction in the operation instruction sequence, the execution result of the second operation instruction is verified to obtain a verification result.

[0190] In some embodiments of the present application, the second operation instruction is one of the operation instructions in the operation instruction sequence.

[0191] It should be noted that the first operation instruction and the second operation instruction may be the same operation instruction or different operation instructions, and are essentially one operation instruction in an operation instruction sequence.

[0192] In some embodiments of the present application, the above verification result is used to indicate whether the execution result of the operation instruction is correct or incorrect.

[0193] In some embodiments of the present application, the electronic device performs verification according to the verification condition corresponding to the second operation instruction to obtain a verification result of the second operation instruction.

[0194] In some embodiments of the present application, after each operation instruction in the operation instruction sequence is executed, a screenshot of the interface corresponding to the displayed operation instruction is taken to obtain a screenshot of the interface corresponding to the operation instruction.

[0195] In some embodiments of the present application, after the electronic device executes the second operation instruction, a screenshot of the interface corresponding to the second operation instruction is taken and matched with the verification condition corresponding to the second operation instruction to verify the second operation instruction.

[0196] In some embodiments of the present application, the electronic device determines that the verification result of the second operation instruction is a correct execution result when the image feature information of the interface screenshot corresponding to the second operation instruction matches the verification condition corresponding to the second operation instruction; and determines that the verification result of the second operation instruction is an incorrect execution result when the image feature information of the interface displayed after the second operation result is executed does not match the verification condition corresponding to the second operation instruction.

[0197] In some embodiments of the present application, the electronic device obtains the above-mentioned interface screenshot by taking a screenshot of the interface corresponding to the second operation instruction, and further combines the image matching algorithm, such as OpenCV template matching or OCR technology, to extract the image feature information of the screenshot, and matches the image feature information with the verification conditions in the interface DCDL data for verification. If there is a match, it means that the execution meets the expectations, and the verification result of the second operation instruction is determined to be the correct execution result. If there is no match, it means that the execution does not meet the expectations, and the verification result of the second operation instruction is determined to be the correct execution result.

[0198] Exemplarily, if the above verification condition is a highlight mark on the operation object, it is detected whether there is a highlight mark in the screenshot image of the current interface. If so, the verification result of the second operation instruction is determined to be a correct execution result; if not, the verification result of the second operation instruction is determined to be an incorrect execution result.

[0199] Exemplarily, if the above-mentioned verification condition is an interface jump identifier, the screenshot image of the current interface is detected to see whether there is an interface jump identifier. If so, the verification result of the second operation instruction is determined to be a correct execution result. If not, the verification result of the second operation instruction is determined to be an incorrect execution result.

[0200] In some embodiments of the present application, the electronic device may further verify the execution result of the second operation instruction through a decision model. It is understood that the verification process of the decision model may refer to the description of the verification process above.

[0201] Step 402: If the verification result indicates that the execution result of the second operation instruction is correct, the electronic device changes the value of the completion status identifier carried by the second operation instruction to the first value.

[0202] In some embodiments of the present application, an operation instruction carries a completion status identifier.

[0203] Exemplarily, the completion status identifier may be a Boolean value, a binary value, or an identifier in any form.

[0204] In some embodiments of the present application, the completion status identifier is used to indicate the execution status of the corresponding operation instruction.

[0205] In some embodiments of the present application, the completion status identifier indicates different execution states of the corresponding operation instruction through different values.

[0206] Exemplarily, the above execution status includes: executed status and unexecuted status.

[0207] In some embodiments of the present application, if the verification result indicates that the execution result of the second operation instruction is correct, the electronic device changes the value of the completion status identifier carried by the second operation instruction from the second value to the first value.

[0208] In some embodiments of the present application, the electronic device indicates that the corresponding operation instruction is in an executed state through a first value, and indicates that the corresponding operation instruction is in a non-executed state through a second value.

[0209] It should be noted that the first value and the second value are two different values.

[0210] For example, when the completion status flag is 1, it indicates that the corresponding operation instruction is in the executed state. When the completion status flag is 0, it indicates that the corresponding operation instruction is in the unexecuted state.

[0211] It should be noted that when an operation instruction in the above operation instruction sequence is marked as executed, the electronic device can automatically transfer its associated DCDL data to the subsequent operation instruction, so that the subsequent operation instruction does not need to obtain certain repeated DCDL data a second time.

[0212] For example, if the completion status of the operation instruction "open the search interface" is identified as the first value indicating that the corresponding operation instruction is in the executed state, the DCDL data of the search interface of the operation instruction will be passed to the next operation instruction "locate the search box". Since the DCDL data of the search interface contains the DCDL data of all controls in the search interface, the electronic device can directly obtain the DCDL data of the search box from the DCDL data of the search interface without having to query in the DCDL database.

[0213] Optionally, in some embodiments of the present application, after the above step 402, the operation execution method provided by the embodiments of the present application further includes step 402a.

[0214] Step 402a: If the value of the completion status identifier carried by the third operation instruction is the second value, the electronic device executes the third operation instruction based on the DCDL data of the operation object indicated by the third operation instruction.

[0215] In some embodiments of the present application, the third operation instruction is the next operation instruction of the second operation instruction in the operation instruction sequence.

[0216] It should be noted that the first operation instruction, the second operation instruction and the third operation instruction may be the same operation instruction or different operation instructions, and are essentially an operation instruction in an operation instruction sequence.

[0217] In some embodiments of the present application, the second value indicates that the corresponding operation instruction is in an unexecuted state.

[0218] In some embodiments of the present application, after verifying the second operation instruction, the electronic device detects the first operation instruction in the operation instruction sequence that is in an unexecuted state, that is, the third operation instruction mentioned above, and continues to execute the operation.

[0219] Step 403: If the verification result indicates that the execution result of the second operation instruction is incorrect, the electronic device re-acquires the DCDL data of the operation object indicated by the second operation instruction.

[0220] In some embodiments of the present application, when the electronic device executes each operation instruction in the operation instruction sequence in sequence based on the DCDL data of each operation object, it will judge the execution result of each operation instruction. If the execution is correct, it will continue to execute the next operation instruction. If the execution is incorrect, it is necessary to re-acquire the DCDL data of the operation object corresponding to the operation instruction, and execute the operation instruction according to the re-acquired DCDL data until it is executed correctly.

[0221] It should be noted that the above description uses the second operation instruction as an example to illustrate the execution process of an operation instruction, wherein the execution process of each operation instruction in the operation instruction sequence can refer to the execution process of the second operation instruction.

[0222] In this way, the electronic device transmits the DCDL data in a chain by completing the state change, thereby saving the step of querying the DCDL data and improving the efficiency of automatically executing operation instructions.

[0223] Optionally, in some embodiments of the present application, the above step 401 can be specifically implemented through the following steps 401a and 401b, or steps 401a and 401c.

[0224] Step 401a: The electronic device matches the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction.

[0225] In some embodiments of the present application, the above-mentioned image feature information is used to represent the display content of the interface image.

[0226] In some embodiments of the present application, the electronic device may obtain image feature information in the screenshot image through a model or OCR technology.

[0227] In some embodiments of the present application, the above-mentioned image feature information may be identification information, layout information, or physical element information.

[0228] For example, when the screenshot image is a shopping cart interface image, the image feature information may be a shopping cart logo, shopping cart text, and the like.

[0229] Step 401b: When the image feature information matches the verification condition corresponding to the second operation instruction, the electronic device determines that the execution result of the second operation instruction is correct.

[0230] Step 401c: When the image feature information does not match the verification condition corresponding to the second operation instruction, the electronic device determines that the execution result of the second operation instruction is incorrect.

[0231] In some embodiments of the present application, the above verification conditions are verification conditions in the DCDL data of the first interface.

[0232] In some embodiments of the present application, the above verification conditions include but are not limited to interface jump identifiers (post_condition) and visual feature markers (visual_markers).

[0233] It should be noted that the relevant description of the above verification conditions can refer to the description of the above DCDL data, which will not be repeated here.

[0234] In some embodiments of the present application, the electronic device takes a screenshot of the result page and combines it with an image matching algorithm, such as OpenCV template matching or OCR technology, to match and verify the recognition results and the verification conditions in the interface DCDL data. If there is a match, it means that the execution has met expectations, and the operation instruction is judged to be executed correctly. If there is no match, it means that the execution has not met expectations, and the operation instruction is judged to be executed incorrectly.

[0235] In this way, by performing result verification according to the verification conditions in the DCDL data of the operation instruction, the electronic device can ensure that the execution result of the operation instruction meets expectations when automatically executing the operation instruction.

[0236] Optionally, in some embodiments of the present application, before the above-mentioned step 401, the operation execution method provided by the embodiment of the present application further includes steps 501 and 502.

[0237] Step 501: After executing the second operation instruction based on DCDL data of the operation object indicated by the second operation instruction, the electronic device adds a first identifier to the second operation instruction in the operation instruction sequence.

[0238] In some embodiments of the present application, the first identifier is used to indicate the most recent operation instruction executed by the electronic device.

[0239] In some embodiments of the present application, the first identifier may be a Boolean value, a binary value, or any other identifier.

[0240] In some embodiments of the present application, the first identifier is a different form of a completion status identifier.

[0241] In some embodiments of the present application, each time the electronic device executes an operation instruction, the electronic device marks the operation instruction with a first identifier.

[0242] Step 502: The electronic device inputs the interface screenshot corresponding to the second operation instruction and the operation instruction sequence marked with the first identifier into the decision model.

[0243] In some embodiments of the present application, the above-mentioned decision model is used to verify the operation instruction marked with the first identifier in the operation instruction sequence.

[0244] In some embodiments of the present application, the above-mentioned decision model can verify whether the execution result of the operation instruction is executed correctly by detecting whether the verification conditions of the current interface are met.

[0245] In some embodiments of the present application, the above-mentioned operation instruction sequence marked with the first identifier can be understood as marking one of the operation instructions in the operation instruction sequence with the first identifier.

[0246] For example, the sequence of operation instructions input into the decision model can be: Operation instruction 1 is completed, Operation instruction 2 is completed, Operation instruction 3 is not completed, and Operation instruction 4 is not completed. Among them, "completed" and "not completed" are both completion status indicators, and "true" is the first indicator mentioned above.

[0247] Optionally, in some embodiments of the present application, in combination with the above steps 501 and 502, the above step 401a can be implemented through the following steps 401a1 and 401a2.

[0248] Step 401a1: The electronic device matches the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction through a decision model.

[0249] In some embodiments of the present application, after each operation instruction in the operation instruction sequence is executed, a screenshot of the interface corresponding to the displayed operation instruction is taken to obtain a screenshot of the interface corresponding to the operation instruction.

[0250] In some embodiments of the present application, after the electronic device executes the second operation instruction, the interface screenshot corresponding to the second operation instruction and the operation instruction sequence marked with the first identifier are input into the decision model, and the image feature information of the interface screenshot is extracted through the decision model, and the extracted image feature information is matched with the verification conditions corresponding to the second operation instruction to verify the second operation instruction.

[0251] In some embodiments of the present application, when the electronic device determines through a decision model that the image feature information of the interface screenshot corresponding to the second operation instruction matches the verification condition corresponding to the second operation instruction, the electronic device determines that the verification result of the second operation instruction is a correct execution result; when the electronic device determines through a decision model that the image feature information of the interface displayed after the second operation result is executed does not match the verification condition corresponding to the second operation instruction, the electronic device determines that the verification result of the second operation instruction is an incorrect execution result.

[0252] It should be noted that the specific verification process is the same as that of the above step 401. Please refer to the relevant description of the above step 401 and will not be repeated here.

[0253] Step 401a2: After completing verification of the execution result of the second operation instruction, the electronic device clears the first identifier and outputs the operation instruction sequence.

[0254] In some embodiments of the present application, the above-mentioned “clearing the first identifier” is to prevent conflicts with other operation instructions marked with the first identifier when the operation instruction is verified by the decision model next time.

[0255] For example, the sequence of operation instructions input into the decision model of an electronic device is: Operation instruction 1 has been completed, Operation instruction 2 has been completed, Operation instruction 3 has not been completed (true), and Operation instruction 4 has not been completed. The electronic device identifies Operation instruction 3, which carries the first identifier "true" through the decision model. Then, the decision model further verifies whether the execution result of Operation instruction 3 is correct. If the execution result of Operation instruction 3 is correct, the first identifier "true" is cleared and the completion status identifier of Operation instruction 3 is changed to "executed", that is, the first value mentioned above. The final output operation instruction sequence is: Operation instruction 1 has been completed, Operation instruction 2 has been completed, Operation instruction 3 has been completed, and Operation instruction 4 has not been completed. At this point, the electronic device automatically executes Operation instruction 4.

[0256] In this way, the expected results after the operation is executed are defined through validation conditions (post_condition) and visual feature markers (visual_markers). Through image recognition and logical state matching, the operation results are automatically verified, ensuring that the operation flow proceeds as expected. Furthermore, the first marker is used to mark the most recently executed operation instructions, effectively ensuring the accuracy of the operation instructions judged by the decision model.

[0257] The following describes in detail the detailed process of the operation execution method provided in the embodiment of the present application using an embodiment.

[0258] Example 1:

[0259] This embodiment takes the example of a user inputting "I want to search for Bluetooth headsets in shopping application A" in the voice assistant of a mobile phone. Figure 2 As shown, the operation execution method provided by this application includes the following steps A1 to A5.

[0260] Step A1: Voice input analysis

[0261] Step A1-1: Voice signal reception: The user's voice command "I want to search for Bluetooth headsets in shopping application A" is collected through the mobile phone microphone.

[0262] Step A1-2: Speech feature extraction, using Mel-frequency cepstral coefficient algorithm for acoustic feature encoding.

[0263] Step A1-3: Automatic Speech Recognition (ASR) conversion, calling the cloud-based speech recognition application programming interface (API) to convert the speech into text information based on the speech characteristics, as well as the above-mentioned original text.

[0264] Steps A1-4: Semantic cleaning: Use the large model to extract key text information, remove redundant words such as modal expressions, adjectives, and adverbs that are irrelevant to app operations or interface descriptions, and output standardized instruction text. To further improve accuracy, you can also add a dictionary of redundant words, such as "er," "that," "hello," "of," and "trouble," as well as a keyword dictionary. For example, extract all description fields in the DCCL database as keywords to provide to the large model for reference.

[0265] For example, by inputting text information into the prompt word of the large model, the key text information "search for Bluetooth headsets in shopping application A" is obtained.

[0266] Step A2: Structuring Intentions

[0267] Exemplarily, the above-mentioned intention structuring can be understood as decomposing the instruction text into tasks and converting them into a sequence of operation instructions.

[0268] Exemplarily, the above instruction text is task-decomposed through a large language model to output structured intent, namely the above operation instruction sequence.

[0269] For example, each operation instruction in the above operation instruction sequence may include the following four core attributes:

[0270] 1) Semantic description of operation instructions: Use natural language to summarize operation instructions and provide explainable decision-making basis for large language models. For example, open shopping app A.

[0271] 2) Semantic Description Keywords (dcdl_keywords): Descriptive keywords extracted through semantic analysis are used for fuzzy matching with descriptions in the DCDL database, enabling dynamic retrieval of target interfaces and controls. This mechanism breaks through the limitations of traditional fixed package name matching and supports intent generalization across application scenarios, such as shopping app A, product search.

[0272] 3) DCDL associated data, that is, the DCDL data corresponding to the above-mentioned operation objects: including the DCDL data of the target interface or control, such as package name, ActivityID, control ID, etc. For complete and detailed fields, please refer to the description of the DCDL database in the technical solution to establish a direct mapping relationship between the operation instructions and the physical interface elements.

[0273] 4) Completion status flag (completed): The Boolean value is used to mark the execution status of the operation instruction, which is used to build the timing context of the operation flow.

[0274] For example, the complete output operation instruction sequence is as follows, which describes in detail the values ​​and action mechanisms of different core attributes in each operation instruction:

[0275] 1) Open shopping app A, which is the semantic description of the above steps

[0276] The dcdl_keywords field is retrieved from the operation instruction, using the semantic description keywords "JD.com, product, purchase, search" as described above. A fuzzy match is performed against the DCDL data for the entry interface of shopping app A in the DCDL database. The DCDL associated data field for the current operation instruction contains the complete DCCL structure of the entry interface, including metadata such as the application package name, activity_id, control name, and control coordinates. The completed field is initialized to an uncompleted state, indicating that the operation instruction has not yet been executed.

[0277] 2) Locate the search box, which is the semantic description of the above steps

[0278] The DCDL associated data field is the DCDL data associated with the search box control, including the control's coordinate positioning (bounds), operation type (action), and other execution parameters. The completed state is initialized to an unfinished state and is updated to a completed state after the system performs a click / focus operation.

[0279] 3) Enter the keyword: Bluetooth headset, which is the semantic description of the above steps

[0280] Enter the specific input value "Bluetooth Headset" in the located search box control DCDL, and the completed state is initialized to the unfinished state. After the input is completed, it is updated to the executed state.

[0281] 4) Trigger search, i.e. the semantic description of the above steps

[0282] The DCDL associated data fields above are the DCDL data bound to the search button, including execution parameters such as click coordinates (bounds), alternative positioning strategies (alternative_selectors), etc. The validation conditions (validation field) preset the characteristics of the jump target page, and the completed flag needs to be updated after a successful jump.

[0283] 5. Verify the result list, i.e. the semantic description of the above steps

[0284] The DCDL data fields above are the DCDL data associated with the search results page. Results are verified using activity_id change detection and visual_markers, such as the "Product List" title. The completed status of this operation instruction serves as the final completion indicator for the entire operation flow.

[0285] In this way, the entire operation instruction sequence implements operation parameter encapsulation through nested DCDL objects, dccl_keywords supports cross-version interface retrieval, and the completed field chain constitutes a state machine model, forming "interface positioning-control operation-result verification".

[0286] Step A3: Automatically execute the operation instruction sequence

[0287] For example, based on the operation instruction sequence output in step A2, the mobile phone can retrieve the first unfinished operation instruction in the operation instruction sequence. After obtaining the DCDL data of the target interface in the operation instruction, such as the search interface or the product list interface, and the control, the mobile phone can understand these target interfaces and controls as the operation objects, and pass the obtained DCDL data to the operating system for automatic operation.

[0288] For example, for an interface, the system automatically opens the interface. For controls within the interface, the system needs to further locate the control's position (bounds) and trigger the system's automatic execution. For example, first locate the search box, then enter the keyword "Bluetooth headset", then locate the search button, and finally automatically click the search button to trigger the search.

[0289] Step A4: Intelligent Cycle Decision

[0290] For example, after each operation instruction is executed, the operation instruction is marked in the operation instruction sequence, and the marked operation instruction sequence is input into the decision model. The decision model determines whether the operation instruction is executed correctly. If it is executed correctly, the completion status flag of the operation instruction is changed in the operation instruction sequence, and the operation instruction sequence is output, and then the next operation instruction is executed. If the execution is incorrect, the DCDL data associated with the operation instruction is changed in the operation instruction sequence, and the operation instruction sequence is output. The operation instruction is executed again based on the changed DCDL data until it is executed correctly.

[0291] Step A5: Verify operation results

[0292] For example, when the completion status indicator of the last operation instruction is changed to the completed status, a screenshot of the result page is taken and an image matching algorithm, such as OpenCV template matching or OCR technology, is used to match and verify the recognition result and the DCDL of the interface. If they match, it means that the execution has met expectations and the current operation instruction sequence is completed. If there is an obvious mismatch, you can go back to step A4 and make a new decision.

[0293] Thus, on the one hand, this embodiment has the advantage of cross-system compatibility, that is, DCDL data is independent of the application development language and can be compatible with different operating systems. On the other hand, this embodiment has the advantage of dynamic adaptability, that is, the continuous update mechanism of the DCDL database can meet the needs of automatically executing operation instructions of newly installed applications. On the other hand, this embodiment also has the advantage of dynamic decision-making ability, that is, through the closed loop of real-time interface DCDL data generation and model decision-making, it can handle complex operation processes without preset paths. This ensures the efficiency and expected results of the electronic device's automatic execution of multiple operation instructions.

[0294] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of this application do not limit this.

[0295] The operation execution method provided in the embodiment of the present application can be executed by an electronic device or an operation execution device. In the embodiment of the present application, the operation execution device provided in the embodiment of the present application is described by taking the operation execution method executed by the operation execution device as an example.

[0296] Figure 3 A possible structural diagram of the operation execution device involved in the embodiment of the present application is shown. Figure 3 As shown, the operation execution device 700 may include: a receiving module 701 , a processing module 702 and an acquisition module 703 .

[0297] The receiving module 701 is configured to receive first information input by a user, where the first information includes at least one of voice information and text information.

[0298] The processing module 702 is configured to perform semantic analysis on the first information in response to the first information received by the receiving module 701 to obtain an operation instruction sequence corresponding to the first information;

[0299] The acquisition module 703 is configured to acquire, based on the operation instruction sequence, DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence;

[0300] The processing module 702 is further configured to sequentially execute the operation instructions in the operation instruction sequence based on the DCDL data.

[0301] Optionally, in some embodiments of the present application, the acquisition module 703 is specifically configured to:

[0302] In the case that there is first DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, the first DCDL data is used as the DCDL data of the operation object indicated by the first operation instruction; or

[0303] If there is no DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, generating second DCDL data based on the semantic analysis result of the first operation instruction, and using the second DCDL data as the DCDL data of the operation object indicated by the first operation instruction;

[0304] The first operation instruction is an operation instruction in the operation instruction sequence.

[0305] Optionally, in some embodiments of the present application, combined with Figure 3 ,like Figure 4 As shown, the apparatus 700 further includes: a verification module 704; the verification module 704 is configured to verify the execution result of the second operation instruction after the processing module 702 sequentially executes the operation instructions in the operation instruction sequence based on the DCDL data, and after executing the second operation instruction based on the DCDL data of the operation object indicated by the second operation instruction in the operation instruction sequence, to obtain a verification result;

[0306] The processing module 702 is further configured to change the value of the completion status indicator carried by the second operation instruction to the first value if the verification result indicates that the execution result of the second operation instruction is correct;

[0307] The processing module 702 is further configured to reacquire DCDL data of the operation object indicated by the second operation instruction if the verification result indicates that the execution result of the second operation instruction is incorrect;

[0308] The second operation instruction is one of the operation instructions in the operation instruction sequence.

[0309] An operation instruction carries a completion status identifier, and the completion status identifier is used to indicate the execution status of the corresponding operation instruction. The first value indicates that the corresponding operation instruction is in the executed state.

[0310] Optionally, in some embodiments of the present application, the processing module 702 is further configured to execute the third operation instruction based on the DCDL data of the operation object indicated by the third operation instruction if the value of the completion status flag carried by the third operation instruction is the second value;

[0311] The third operation instruction is the next operation instruction of the second operation instruction in the operation instruction sequence, and the second value indicates that the corresponding operation instruction is in an unexecuted state.

[0312] Optionally, in some embodiments of the present application, combined with Figure 4 ,like Figure 5 As shown, the apparatus further includes: an interception module 705; the interception module 705 is configured to, before the verification module 704 verifies the execution result of the second operation instruction, take a screenshot of the displayed interface corresponding to each operation instruction in the operation instruction sequence after each operation instruction is executed, to obtain a screenshot of the interface corresponding to the operation instruction;

[0313] The verification module 704 is specifically used to:

[0314] Matching the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction;

[0315] If the image feature information matches the verification condition corresponding to the second operation instruction, determining that the execution result of the second operation instruction is correct; or

[0316] In a case where the image feature information does not match the verification condition corresponding to the second operation instruction, it is determined that the execution result of the second operation instruction is incorrect.

[0317] In the operation execution device provided in the embodiment of the present application, the operation execution device receives first information input by the user, and the first information includes at least one of voice information or text information; in response to the above-mentioned first information, the above-mentioned first information is semantically analyzed to obtain an operation instruction sequence corresponding to the above-mentioned first information; based on the operation instruction sequence, the DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence is obtained; based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence. In this solution, the user triggers the operation execution device to directly perform semantic analysis on the continuous operation instruction by inputting text information or voice information containing a continuous operation instruction, thereby obtaining DCDL data for describing the operation object indicated by each operation instruction in the continuous operation instruction, and then the operation execution device can automatically execute the corresponding operation instruction according to the DCDL data of the operation object indicated by each operation instruction. In this way, the human-computer interaction step is saved by automatically executing the continuous operation instructions input by the user.

[0318] The operation execution device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0319] The operation execution device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0320] The operation execution device provided in the embodiment of the present application can implement each process implemented in the embodiment of the operation execution method and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0321] Alternatively, as Figure 6 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be run on the processor 801. When the program or instruction is executed by the processor 801, the various steps of the above-mentioned operation execution method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0322] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0323] Figure 7 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0324] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101 , a network module 102 , an audio output unit 103 , an input unit 104 , a sensor 105 , a display unit 106 , a user input unit 107 , an interface unit 108 , a memory 109 , and a processor 110 .

[0325] Those skilled in the art will understand that the electronic device 100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0326] The user input unit 107 is configured to receive first information input by a user, wherein the first information includes at least one of voice information and text information;

[0327] The processor 110 is configured to perform semantic analysis on the first information in response to the first information received by the user input unit 107 to obtain an operation instruction sequence corresponding to the first information;

[0328] The processor 110 is configured to obtain, based on the operation instruction sequence, DCDL data of an operation object indicated by each operation instruction in the operation instruction sequence;

[0329] The processor 110 is further configured to sequentially execute the operation instructions in the operation instruction sequence based on the DCDL data.

[0330] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0331] In the case that there is first DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, the first DCDL data is used as the DCDL data of the operation object indicated by the first operation instruction; or

[0332] If there is no DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, generating second DCDL data based on the semantic analysis result of the first operation instruction, and using the second DCDL data as the DCDL data of the operation object indicated by the first operation instruction;

[0333] The first operation instruction is an operation instruction in the operation instruction sequence.

[0334] Optionally, in some embodiments of the present application, the processor 110 is further configured to, after sequentially executing the operation instructions in the operation instruction sequence based on the DCDL data, and after executing the second operation instruction based on the DCDL data of the operation object indicated by the second operation instruction in the operation instruction sequence, verify the execution result of the second operation instruction to obtain a verification result;

[0335] The processor 110 is further configured to change the value of the completion status indicator carried by the second operation instruction to the first value if the verification result indicates that the execution result of the second operation instruction is correct;

[0336] The processor 110 is further configured to reacquire DCDL data of the operation object indicated by the second operation instruction if the verification result indicates that the execution result of the second operation instruction is erroneous;

[0337] The second operation instruction is one of the operation instructions in the operation instruction sequence.

[0338] An operation instruction carries a completion status identifier, and the completion status identifier is used to indicate the execution status of the corresponding operation instruction. The first value indicates that the corresponding operation instruction is in the executed state.

[0339] Optionally, in some embodiments of the present application, the processor 110 is further configured to, if the value of the completion status flag carried by the third operation instruction is the second value, execute the third operation instruction based on the DCDL data of the operation object indicated by the third operation instruction;

[0340] The third operation instruction is the next operation instruction of the second operation instruction in the operation instruction sequence, and the second value indicates that the corresponding operation instruction is in an unexecuted state.

[0341] Optionally, in some embodiments of the present application, the processor 110 is further configured to, before the processor 110 verifies the execution result of the second operation instruction, after each execution of an operation instruction in the operation instruction sequence, take a screenshot of the displayed interface corresponding to the operation instruction, to obtain a screenshot of the interface corresponding to the operation instruction;

[0342] The processor 110 is specifically configured to:

[0343] Matching the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction;

[0344] If the image feature information matches the verification condition corresponding to the second operation instruction, determining that the execution result of the second operation instruction is correct; or

[0345] In a case where the image feature information does not match the verification condition corresponding to the second operation instruction, it is determined that the execution result of the second operation instruction is incorrect.

[0346] In the electronic device provided in the embodiment of the present application, the electronic device receives first information input by a user, and the first information includes at least one of voice information or text information; in response to the above-mentioned first information, the above-mentioned first information is semantically analyzed to obtain an operation instruction sequence corresponding to the above-mentioned first information; based on the operation instruction sequence, the DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence is obtained; based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence. In this solution, the user triggers the electronic device to directly perform semantic analysis on the continuous operation instruction by inputting text information or voice information containing a continuous operation instruction, thereby obtaining DCDL data for describing the operation object indicated by each operation instruction in the continuous operation instruction, and then the electronic device can automatically execute the corresponding operation instruction according to the DCDL data of the operation object indicated by each operation instruction. In this way, the steps of human-computer interaction are saved by automatically executing the continuous operation instructions input by the user.

[0347] It should be understood that in an embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0348] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0349] Processor 110 may include one or more processing units. Optionally, processor 110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 110.

[0350] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned operation execution method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0351] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0352] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned operation execution method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0353] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0354] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned operation execution method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0355] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0356] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0357] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An operation execution method, characterized in that: The method comprises: receiving first information input by a user, where the first information includes at least one of a voice message or a text message; In response to the first information, performing semantic analysis on the first information to obtain an operation instruction sequence corresponding to the first information; Based on the operation instruction sequence, obtaining dynamic control description language DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence; Based on the DCDL data, the operation instructions in the operation instruction sequence are executed in sequence.

2. The method according to claim 1, characterized in that The acquiring, based on the operation instruction sequence, DCDL data of the operation object indicated by each operation instruction in the operation instruction sequence comprises: In a case where there is first DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, using the first DCDL data as the DCDL data of the operation object indicated by the first operation instruction; If there is no DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, generating second DCDL data based on the semantic analysis result of the first operation instruction, and using the second DCDL data as the DCDL data of the operation object indicated by the first operation instruction; The first operation instruction is an operation instruction in the operation instruction sequence.

3. The method according to claim 1, characterized in that After sequentially executing the operation instructions in the operation instruction sequence based on the DCDL data, the method further includes: After executing the second operation instruction based on the DCDL data of the operation object indicated by the second operation instruction in the operation instruction sequence, verifying the execution result of the second operation instruction to obtain a verification result; If the verification result indicates that the execution result of the second operation instruction is correct, changing the value of the completion status identifier carried by the second operation instruction to the first value; If the verification result indicates that the execution result of the second operation instruction is wrong, re-acquiring DCDL data of the operation object indicated by the second operation instruction; Wherein, the second operation instruction is one of the operation instructions in the operation instruction sequence; An operation instruction carries a completion status identifier, and the completion status identifier is used to indicate the execution status of the corresponding operation instruction. The first value indicates that the corresponding operation instruction is in the executed state.

4. The method according to claim 3, characterized in that After changing the value of the completion status indicator carried by the second operation instruction to the first value, the method further includes: If the value of the completion status flag carried by the third operation instruction is the second value, executing the third operation instruction based on the DCDL data of the operation object indicated by the third operation instruction; The third operation instruction is the next operation instruction of the second operation instruction in the operation instruction sequence, and the second value indicates that the corresponding operation instruction is in an unexecuted state.

5. The method according to claim 4, characterized in that Before verifying the execution result of the second operation instruction, the method further includes: After each execution of an operation instruction in the operation instruction sequence, taking a screenshot of the displayed interface corresponding to the operation instruction to obtain a screenshot of the interface corresponding to the operation instruction; The verifying the execution result of the second operation instruction includes: matching the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction; If the image feature information matches the verification condition corresponding to the second operation instruction, determining that the execution result of the second operation instruction is correct; or In a case where the image feature information does not match the verification condition corresponding to the second operation instruction, it is determined that the execution result of the second operation instruction is incorrect.

6. An operation execution device, characterized in that: The operation execution device includes: a receiving module, a processing module and an acquisition module; The receiving module is configured to receive first information input by a user, where the first information includes at least one of voice information or text information; The processing module is configured to perform semantic analysis on the first information in response to the first information received by the receiving module, so as to obtain an operation instruction sequence corresponding to the first information; The acquisition module is configured to acquire, based on the operation instruction sequence, DCDL data of an operation object indicated by each operation instruction in the operation instruction sequence; The processing module is further configured to sequentially execute the operation instructions in the operation instruction sequence based on the DCDL data.

7. The device according to claim 6, characterized in that The acquisition module is specifically used to: In a case where there is first DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, using the first DCDL data as the DCDL data of the operation object indicated by the first operation instruction; or, If there is no DCDL data in the DCDL database that matches the semantic analysis result of the first operation instruction, generating second DCDL data based on the semantic analysis result of the first operation instruction, and using the second DCDL data as the DCDL data of the operation object indicated by the first operation instruction; The first operation instruction is an operation instruction in the operation instruction sequence.

8. The device according to claim 6, characterized in that The device further comprises: a verification module; The verification module is configured to verify the execution result of the second operation instruction in the operation instruction sequence after the processing module sequentially executes the operation instructions in the operation instruction sequence based on the DCDL data, and after executing the second operation instruction based on the DCDL data of the operation object indicated by the second operation instruction in the operation instruction sequence to obtain a verification result; The processing module is further configured to change the value of the completion status identifier carried by the second operation instruction to the first value if the verification result indicates that the execution result of the second operation instruction is correct; The processing module is further configured to reacquire DCDL data of the operation object indicated by the second operation instruction if the verification result indicates that the execution result of the second operation instruction is incorrect; Wherein, the second operation instruction is one of the operation instructions in the operation instruction sequence; An operation instruction carries a completion status identifier, and the completion status identifier is used to indicate the execution status of the corresponding operation instruction. The first value indicates that the corresponding operation instruction is in the executed state.

9. The device according to claim 8, characterized in that The processing module is further configured to execute the third operation instruction based on the DCDL data of the operation object indicated by the third operation instruction if the value of the completion status flag carried by the third operation instruction is the second value; The third operation instruction is the next operation instruction of the second operation instruction in the operation instruction sequence, and the second value indicates that the corresponding operation instruction is in an unexecuted state.

10. The device according to claim 9, characterized in that The device further comprises: an interception module; The interception module is configured to, before the verification module verifies the execution result of the second operation instruction, take a screenshot of the displayed interface corresponding to an operation instruction after each execution of an operation instruction in the operation instruction sequence, to obtain a screenshot of the interface corresponding to the operation instruction; The verification module is specifically used to: matching the image feature information of the interface screenshot corresponding to the second operation instruction with the verification condition corresponding to the second operation instruction; If the image feature information matches the verification condition corresponding to the second operation instruction, determining that the execution result of the second operation instruction is correct; or In a case where the image feature information does not match the verification condition corresponding to the second operation instruction, it is determined that the execution result of the second operation instruction is incorrect.

Images