Instruction execution method, computer equipment and computer program product

Abstract

The embodiment of the invention discloses an instruction execution method, computer equipment and a computer program product, and belongs to the technical field of artificial intelligence. The method comprises the steps that in the recording process of a target instruction, target path information of a target operation path for achieving the target instruction is generated based on an operation track model, and the operation track model comprises node information of a node corresponding to an interface in an application and side information of a behavior side corresponding to an operation behavior in the interface; the target operation path is composed of nodes and behavior edges between the nodes; and in response to an instruction execution operation on the target instruction, simulating an operation behavior based on a target operation path indicated by the target path information. Compared with a complex script file, the space occupied by the storage path information is smaller; and simulation is performed based on the node information indicated by the path information and the side information, so that the accuracy of interface positioning and operation behavior restoration in the simulation process can be improved, and the success rate of instruction execution can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an instruction execution method, a computer device, and a computer program product. Background Art

[0002] As the intelligence level of terminals continues to improve, more and more terminals begin to support the automatic execution of instructions.

[0003] In the related art, a user can first input a voice command, and then perform the operation to implement the voice command by manually recording. The terminal records the operation and generates a script file, and binds the script file to the voice command. When the voice command is received later, the terminal can perform the corresponding operation based on the script file. Summary of the invention

[0004] The embodiments of the present application provide an instruction execution method, a computer device, and a computer program product. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides an instruction execution method, the method comprising:

[0006] During the recording process of the target instruction, target path information of the target operation path for realizing the target instruction is generated based on the operation trajectory model, wherein the operation trajectory model includes node information of nodes corresponding to the interface in the application and edge information of behavior edges corresponding to the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes;

[0007] In response to an instruction execution operation on the target instruction, an operation behavior is simulated based on the target operation path indicated by the target path information.

[0008] On the other hand, an embodiment of the present application provides an instruction execution device, the device comprising:

[0009] A recording module, used to generate target path information of a target operation path for realizing the target instruction based on an operation trajectory model during the recording process of the target instruction, wherein the operation trajectory model includes node information of nodes corresponding to the interface in the application and edge information of behavior edges corresponding to the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes;

[0010] An execution module is used to simulate an operation behavior based on the target operation path indicated by the target path information in response to an instruction execution operation on the target instruction.

[0011] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the instruction execution method described in the above aspects.

[0012] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the instruction execution method described in the above aspect.

[0013] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the instruction execution method described in the above aspects.

[0014] In the embodiment of the present application, an operation trajectory model is used to record the operation process in the interface, wherein the operation trajectory model includes node information of the corresponding nodes of the interface and edge information of the corresponding behavior edges of the operation behavior in the interface. With the help of the node information and edge information in the operation trajectory model, the computer device can determine the nodes and behavior edges involved in the operation path of implementing the instruction in the instruction recording stage, and then generate the path information of the operation path based on the node information of the node and the edge information of the behavior edge, so as to simulate the corresponding operation behavior based on the path information in the subsequent instruction execution stage. Compared with complex script files, the space occupied by storing path information is smaller; and simulation based on the node information and edge information indicated by the path information can improve the accuracy of interface positioning and operation behavior restoration during the simulation process, which helps to improve the success rate of instruction execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of an operation trajectory model corresponding to an operation process shown in an exemplary embodiment of the present application;

[0016] Figure 2 A flowchart of an instruction execution method provided by an exemplary embodiment of the present application is shown;

[0017] Figure 3 is a flow chart of a path information generation process shown in an exemplary embodiment of the present application;

[0018] Figure 4 is a schematic diagram of a skeletonized interface layout shown in an exemplary embodiment of the present application;

[0019] Figure 5is a schematic diagram of an interface layout similarity calculation process shown in an exemplary embodiment of the present application;

[0020] Figure 6 is a schematic diagram of an operation path shown in an exemplary embodiment of the present application;

[0021] Figure 7 is a schematic diagram of an operation path shown in another exemplary embodiment of the present application;

[0022] Figure 8 is a schematic diagram of a recording control floating window shown in an exemplary embodiment of the present application;

[0023] Fig. 9 is a flowchart of a task setting and execution process shown in an exemplary embodiment of the present application;

[0024] Fig.10 is an interface schematic diagram of a current monitoring task setting process shown in an exemplary embodiment of the present application;

[0025] Fig.11 is a flow chart of an exemplary embodiment of the present application showing a simulation operation process based on target path information;

[0026] Fig.12 A structural block diagram of an instruction execution device provided by another exemplary embodiment of the present application is shown;

[0027] Fig.13 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0030] For ease of understanding, the operation trajectory model involved in the embodiments of the present application is first described below.

[0031] The operation trajectory model (Trajectory Structure) is a model (or data structure) used to record the operation process. Figure 1As shown, the operation trajectory model 13 corresponding to the operation process is composed of nodes 131, behavior edges 132 and operation paths 133. Among them, the node 131 corresponds to the interface 11 passed during the operation process; the behavior edge 132 corresponds to the operation behavior in the interface, which is used to connect different nodes (interface switching occurs after the operation) or connect the same node (no interface switching occurs after the operation) 12; the operation path 133 corresponds to the operation behavior sequence.

[0032] In addition, in order to ensure the accuracy of the operation process record, the node information 141 of the node 131 represents the interface features of the interface from different dimensions to ensure the accuracy of the subsequent interface retrieval and positioning; the edge information 142 of the behavior edge 132 indicates the operation behavior and the node jumps before and after the operation behavior to ensure the accurate restoration of the operation in the subsequent interface; the path information 143 of the operation path 133 represents the execution sequence between different operation behaviors to ensure that complex operations are executed in sequence.

[0033] Compared with screenshots and videos, the operation trajectory model has a lower data volume and can be automatically generated without manual triggering by the user. In addition, it can automatically backtrack and reproduce based on node information, edge information, and path information, improving the efficiency of operation backtracking and reproduction.

[0034] Regarding the method of generating the operation trajectory model, in one possible implementation, a computer device traverses the entire interface of the application and all interface elements in the interface, and performs operations on the interface elements, thereby generating an operation trajectory model that includes node information representing interface features, edge information representing operation behaviors, and path information representing operation paths.

[0035] node

[0036] The interface is a visual user interface, such as a desktop, an application program interface, a system interface, etc. In the embodiment of the present application, different interfaces in the operation process correspond to respective nodes, and different nodes represent the interface features of their respective corresponding interfaces through their respective node information, that is, different nodes can be distinguished based on the node information.

[0037] Optionally, dimensions characterizing interface features may include interface content dimension, interface layout dimension, interface function dimension, interface description dimension, the window to which the interface belongs, etc. The following embodiments will describe in detail the specific content contained in the node information.

[0038] The interface in this embodiment is not limited to the visible range of the screen, but includes the completed interface within and outside the visible range of the screen. Accordingly, before and after the interface content is scrolled, even if the displayed content within the visible range of the screen changes significantly, the interface before and after the scrolling will not be judged as a different interface.

[0039] In the embodiments of the present application, different interfaces refer to interfaces with significant differences in interface layout, content, and style. Usually, when an interface switch occurs, the interfaces before and after the switch are determined to be different interfaces. For example, when text or images in an interface change dynamically, or when list elements in an interface change, they are not determined to be different interfaces; when clicking on a control in an interface to trigger an interface jump, the interfaces before and after the jump are determined to be different interfaces.

[0040] Behavior Edge

[0041] The behavior edge (Edge) corresponding to the operation behavior is a directed edge, and the direction of the behavior edge is the interface jump direction, that is, the behavior edge points from the interface before the jump to the interface after the jump.

[0042] Optionally, the operation behavior refers to a manual operation behavior of the user, such as a trigger operation on a control in the interface; or, the operation behavior refers to a system behavior, such as an automatic interface jump operation after a preset time period has been reached.

[0043] In some embodiments, a plurality of operation behavior types are preset, and the edge information of the behavior edge includes the operation behavior type to which the operation behavior belongs.

[0044] Regarding the recording method of the node jump situation before and after the operation behavior, in a possible implementation, the side information includes information about the corresponding nodes of the interface before and after the operation behavior is performed. For example, the side information includes the node identifier of the node corresponding to the interface before the operation behavior is performed, and the node identifier of the node corresponding to the interface after the operation behavior is performed. Among them, the interface identifiers corresponding to the interfaces before and after the operation behavior can be different interface identifiers (i.e., the interface jump occurs after the operation), or they can be the same interface identifier (i.e., the operation does not cause an interface jump).

[0045] Operation Path

[0046] Since a series of operations may need to be performed during the operation process and multiple interfaces may need to be jumped, the computer device needs to sort the behavior edges corresponding to the operation behaviors according to the execution sequence between the operation behaviors to obtain the operation path (Path). Among them, the operation path is a directed path composed of multiple nodes and behavior edges between nodes.

[0047] Correspondingly, the path information of the operation path is composed of edge information, and can indicate the order relationship between the edge information and the corresponding behavior edge.

[0048] Please refer to Figure 2 , which shows a flowchart of an instruction execution method provided by an exemplary embodiment of the present application, and the method may include the following steps.

[0049] Step 201, during the recording process of the target instruction, target path information of the target operation path for realizing the target instruction is generated based on the operation trajectory model, the operation trajectory model includes node information of the corresponding nodes of the interface in the application and edge information of the corresponding behavior edges of the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes.

[0050] Optionally, the target instruction can be a voice instruction, a text instruction, a conditional instruction (an instruction triggered when a specific condition is met) or a gesture instruction (an instruction triggered when a specific gesture is recognized), and the target instruction is used to instruct the implementation of a specific function or a combined function (obtained by combining multiple functions).

[0051] For example, the target instruction is a voice instruction of "instructing to enable function B in application A", or the target instruction is a conditional instruction of "enabling function D when detecting being at location C". The embodiment of the present application does not limit the type of the target instruction.

[0052] In one possible implementation, a computer device determines the interfaces passed through on a target operation path and the operation behaviors performed on the target operation path, thereby generating target path information of the target operation path based on node information corresponding to the nodes passed through the interfaces and edge information corresponding to the behavior edges of the performed operation behaviors.

[0053] In some embodiments, the edge information in the operation trajectory model includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of a starting node corresponding to a starting interface before the operation behavior, an ending node identifier (toNodeID) of a terminating node corresponding to a terminating interface after the operation behavior, and an edge identifier (a unique identifier of the edge).

[0054] Since the interface before and after the operation is performed can be determined based on the edge information, in some embodiments, the target path information includes a behavior edge list. The behavior edge list contains a series of behavior edges corresponding to the operation behaviors, and the behavior edges are arranged in the time sequence of the operation behaviors. For example, the behavior edge at the head of the behavior edge list corresponds to the starting operation behavior in the target operation path, and the behavior edge at the end of the list corresponds to the ending operation behavior in the target operation path.

[0055] Compared with the script file, the target path information only needs to indicate the nodes and behavior edges recorded in the operation trajectory model, so the data volume is significantly smaller than the script file. For example, the target path information only needs to record the edge identifiers of the behavior edges on the target operation path.

[0056] In some embodiments, after completing the recording of the target instruction, the computer device associates and stores the target instruction with the target path information.

[0057] Step 202 , in response to an instruction execution operation on a target instruction, simulate an operation behavior based on a target operation path indicated by the target path information.

[0058] When receiving an instruction execution operation of a target instruction, the computer device determines the target path information corresponding to the target instruction, and then simulates the operation behavior based on the target path information to simulate the target operation path to implement the target instruction.

[0059] Optionally, the instruction execution operation may be triggered by a voice, a condition, or a gesture, etc. This embodiment does not limit the method of triggering the execution of the target instruction.

[0060] In one possible implementation, when the target path information includes a behavior edge list, the computer device sequentially obtains the edge information of the behavior edge from the behavior edge list, and then determines the starting interface before the operation behavior is executed, the behavior information of the operation behavior, and the termination interface after the operation behavior is executed from the operation trajectory model based on the edge information.

[0061] In some embodiments, when the computer device determines that the current interface is the starting interface, the computer device simulates the operation behavior based on the behavior information of the operation behavior, and detects whether to jump to the termination interface after the operation behavior is executed. In the case of jumping to the terminal interface, the computer device determines that the operation behavior corresponding to the edge information is successfully executed.

[0062] Compared with executing instructions based on script files, which strictly rely on the interface layout that is completely consistent with the instruction recording stage, that is, when the interface layout changes, the instruction execution based on the script file may fail; when executing the target instruction due to path information, the interface and the operation behavior within the interface can be dynamically determined according to the operation trajectory model, that is, it can automatically use the changes in the interface layout, has better adaptability, and helps to improve the success rate of instruction execution.

[0063] In summary, in the embodiment of the present application, an operation trajectory model is used to record the operation process in the interface, wherein the operation trajectory model includes node information of the corresponding nodes of the interface and edge information of the corresponding behavior edges of the operation behavior in the interface. With the help of the node information and edge information in the operation trajectory model, the computer device can determine the nodes and behavior edges involved in the operation path of implementing the instruction in the instruction recording stage, and then generate the path information of the operation path based on the node information of the node and the edge information of the behavior edge, so as to simulate the corresponding operation behavior based on the path information in the subsequent instruction execution stage. Compared with complex script files, the space occupied by storing path information is smaller; and simulation based on the node information and edge information indicated by the path information can improve the accuracy of interface positioning and operation behavior restoration during the simulation process, which helps to improve the success rate of instruction execution.

[0064] Path information generation process

[0065] When generating path information with the help of the operation trajectory model, for each interface on the operation path, the computer device determines the recorded interface that matches the real-time interface from the operation trajectory model based on the real-time interface; for each operation behavior on the operation path, the computer device determines the recorded operation behavior that matches the real-time operation behavior from the operation trajectory model based on the real-time operation behavior.

[0066] Furthermore, based on the matched recorded interface and recorded operation behavior, the computer device determines the nodes included in the operation path and the behavior edges between the nodes, and then generates path information of the operation path. Figure 3 As shown, the above step 201 may include the following sub-steps:

[0067] Step 201A: determine the target node corresponding to the target interface involved in the target operation path from the operation trajectory model.

[0068] The target interface is any interface on the target operation path. That is, for each interface on the operation path, the computer device needs to determine the recorded node corresponding to the interface from the operation trajectory model.

[0069] Since node information is used in the operation trajectory model to represent the interface features of the interface, and interface feature matching is usually used to identify whether two interfaces are the same interface, the computer device determines the target node (recorded node) corresponding to the target interface based on the node information of the recorded node corresponding to the recorded interface in the operation trajectory model.

[0070] In some embodiments, the computer device generates node information corresponding to the target interface in the same manner as the recorded interface, thereby determining the target node corresponding to the target interface by comparing the node information of the recorded node corresponding to the recorded interface and the node information corresponding to the target interface.

[0071] The following first describes the node information:

[0072] Node Information

[0073] In some embodiments, the node information of a node includes at least a node identifier and a node factor of the node.

[0074] The node identifier is a unique identifier assigned to a node (with no actual meaning), which can be represented in the form of a string. Different nodes have different node identifiers, while the same node has the same node identifier.

[0075] Optionally, a node factor is composed of several fields that characterize interface features from different dimensions, wherein the fields that characterize interface features can be referred to as node sub-factors. When determining whether two nodes are the same node, the computer device matches the field contents with the same field name in the two node factors, that is, matches the node sub-factors in different node factors, thereby determining whether the two nodes are the same node based on the matching results of multiple dimensions.

[0076] In some embodiments, the node factor may include at least one of the following node sub-factors:

[0077] First node subfactor (windowName)

[0078] Since the same interface is definitely located in the same window, and interfaces located in different windows are definitely not the same interface, the computer device can determine whether two nodes are the same node based on the dimension of the window to which the interface belongs.

[0079] In a possible implementation, the computer device determines the first node sub-factor based on the window name of the window to which the interface belongs.

[0080] Among them, the window name of the window to which the interface belongs can be extracted from the interface code.

[0081] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the first node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, since different interfaces may be located in the same window, the computer device cannot directly determine that the two nodes with the same window name belong to the same node, but needs to make further determinations based on other node sub-factors.

[0082] Second node sub-factor (operableElements)

[0083] The interface usually includes a large number of operable elements, such as button controls, expandable list controls, text input box controls, upload controls, etc., and these operable elements can reflect the interface characteristics of the interface to a certain extent.

[0084] Therefore, the computer device can determine whether two nodes are the same node based on the dimension of the operable elements contained in the interface.

[0085] Affected by factors such as version updates or dynamic changes in content, the element content of the operable elements in the same interface may change (for example, the button display text of a button control changes, or the list item content in an expandable list control changes), but the element type of the operable elements in the same interface usually does not change. Therefore, in a possible implementation, the computer device determines the element type of the operable element, and performs information summary processing on the string obtained by splicing the element type, thereby determining the information summary result as the second node sub-factor. Among them, the element type of the operable element can be extracted from the interface code.

[0086] Among them, since the character string after the information digest is of fixed length, it is convenient to directly compare the information digest results later.

[0087] Optionally, the computer device performs MD5 processing on the string obtained by concatenating the element types, and determines the obtained string as the second node sub-factor.

[0088] Illustratively, when the interface includes 2 button controls, 1 text box control, and 3 radio selection controls, the computer device performs MD5 processing on the concatenated string Button_TextBox_RadioButton, and determines the obtained 64-bit string as the second node sub-factor.

[0089] Of course, in addition to being represented by a character string obtained from the information digest, the computer device may also determine an element type list (list) composed of element types as the second node sub-factor, which is not limited in this embodiment.

[0090] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the second node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, in order to avoid judging two different simple interfaces (i.e., interfaces containing a small number of operable elements) as the same interface, the computer device needs to make further judgments based on other node sub-factors.

[0091] The third node subfactor (layoutVector)

[0092] Affected by factors such as version updates, the interface content of the same interface may change significantly, but the overall layout of the interface usually changes less. Therefore, the computer device can determine whether two nodes are the same node based on the dimension of interface layout.

[0093] In a possible implementation, the computer device determines the third node sub-factor based on the interface layout of the interface.

[0094] Optionally, in order to facilitate the subsequent comparison of interface layouts, the third node sub-factor is represented by a vector, wherein vector elements of different dimensions in the vector are used to represent different interface elements, and the vectors obtained by vectorizing the interface layouts of different interfaces have the same number of dimensions (for example, all are 512-dimensional vectors).

[0095] Regarding the vectorization method of the interface layout, in a possible implementation, the computer device extracts the interface layout information of the interface, performs vectorization processing on the interface layout information, and determines the obtained interface layout vector as the third node sub-factor. The interface layout information of the interface can be extracted from the interface code.

[0096] Optionally, the computer device generates an interface element skeleton based on the interface layout information, and the interface element skeleton is only used to characterize the position and size of different interface elements in the interface, and does not include the specific element content of the interface elements. Further, the computer device vectorizes the interface element skeleton to obtain an interface layout vector.

[0097] Indicatively, Figure 4 As shown, the computer device performs skeleton processing on the interface layout information of the interface 41 to obtain an interface element skeleton 42.

[0098] Fourth node sub-factor (screenVector)

[0099] In addition to determining different interfaces (nodes) from the dimension of interface layout, different interfaces can also be determined from a more intuitive visual level, that is, a computer device can determine whether two nodes are the same node from the dimension of the interface screen.

[0100] In a possible implementation, the computer device determines the fourth node sub-factor based on an interface screen of the interface.

[0101] In some embodiments, the computer device directly determines the interface screen as the fourth node sub-factor, and the fourth node sub-factor includes the pixel value of each pixel in the interface screen.

[0102] In order to reduce the amount of storage and simplify the amount of calculation in subsequent comparisons, in other embodiments, the computer device performs vector processing on the interface screen of the interface, and determines the obtained interface screen vector as the fourth node sub-factor. The number of dimensions of the interface screen vectors obtained by vector processing of different interface screens is consistent.

[0103] Regarding the method of vectorizing the interface screen, optionally, the computer device extracts image features of the interface screen through a neural network (such as a CNN network, a DNN network, etc.), and uses the extracted image feature vector as the fourth node sub-factor. Alternatively, the computer device can extract color features (such as obtained through a color histogram), texture features (such as obtained through texture analysis), and shape features (such as obtained through edge detection) of the interface screen, and obtain the fourth node sub-factor by vectorizing the color features, texture features, and shape features. The embodiments of the present application do not limit the specific method of screen vectorization.

[0104] Fifth node sub-factor (summaryFunction)

[0105] During the interface development phase, developers usually add interface function description information to the interface code. Even if the interface effect is subsequently updated, the interface function description information usually does not change. Therefore, the computer device can determine whether two nodes are the same node from the dimension of interface function.

[0106] In a possible implementation, the computer device determines the fifth node sub-factor based on interface function description information of the interface, wherein the interface function description information can be extracted from the interface code.

[0107] It should be noted that, since not all interfaces will add interface function description information, if the interface function description information cannot be obtained, the fifth node sub-factor may be empty.

[0108] Furthermore, when subsequently identifying whether two nodes are the same node, the computer device can compare whether the fifth node sub-factors in the node factors of the two nodes are the same. If they are the same, it is determined that the two nodes are the same node; if they are not the same, the computer device needs to make further determinations based on other node sub-factors.

[0109] In a possible implementation, the computer device may generate a node factor corresponding to the target interface, and compare the node factor with the node factor of the recorded node, thereby determining the target node corresponding to the target interface. This step may include the following sub-steps:

[0110] Step 201A1: Generate node factors corresponding to the target interface based on interface features of different dimensions of the target interface.

[0111] In a possible implementation, the computer device generates corresponding node sub-factors based on interface features of different dimensions of the target interface, and then determines the node factor corresponding to the target interface.

[0112] In some embodiments, the node factor corresponding to the target interface may include all or part of the above node sub-factors. For example, the computer device determines the first to fifth node sub-factors corresponding to the target interface.

[0113] Step 201A2: Determine a recorded interface that matches the target interface based on the node factors of the recorded nodes corresponding to the recorded interfaces in the operation trajectory model and the node factors corresponding to the target interface.

[0114] In a possible implementation, the computer device matches the node factors of the target interface with the node factors corresponding to the recorded interfaces in the operation trajectory model one by one, thereby determining the recorded interface that matches the target interface.

[0115] Optionally, the computer device may determine a recorded interface that matches the target interface based on a single node sub-factor in the node factor, or in combination with multiple node sub-factors in the node factor.

[0116] In some embodiments, when the node factor includes the first to fifth node sub-factors described above, the computer device may determine whether the target interface matches the recorded interface in the following cases.

[0117] Case 1: When the first node sub-factors corresponding to the target interface and the recorded interface are different, it is determined that the target interface does not match the recorded interface.

[0118] Since the same interface belongs to the same window, by comparing the first node sub-factors of the target interface and the recorded interface, and when the first node sub-factor of the target interface is different from the first node sub-factor of the recorded interface, the computer device determines that the target interface does not match the recorded interface.

[0119] When the first node sub-factor of the target interface is the same as the first node sub-factor of the recorded interface, the computer device needs to further determine whether the target interface matches the recorded interface in combination with other node sub-factors.

[0120] Optionally, when making subsequent determinations in combination with other node sub-factors, the computer device only needs to consider the recorded interfaces having the same first node sub-factor as the target interface, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0121] Case 2: When the second node sub-factors corresponding to the target interface and the recorded interface are different, it is determined that the target interface does not match the recorded interface.

[0122] Since the types of operable elements included in different interfaces are generally different, the computer device determines that the target interface does not match the recorded interface by comparing the second node sub-factors of the target interface and the recorded interface, and when the second node sub-factor of the target interface is different from the second node sub-factor of the recorded interface.

[0123] When the second node sub-factor of the target interface is the same as the second node sub-factor of the recorded interface (they may be two different interfaces containing very few operable elements), the computer device needs to combine other node sub-factors to further determine whether the target interface matches the recorded interface.

[0124] Optionally, when making subsequent determinations in combination with other node factors, the computer device only needs to consider the recorded interfaces that have the same second node sub-factor as the target interface, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0125] Case 3: When the fifth node sub-factors corresponding to the target interface and the recorded interface are the same, it is determined that the target interface matches the recorded interface.

[0126] If the functions of the two interfaces are completely the same, then the two interfaces should belong to the same interface. Therefore, in some embodiments, when the fifth node sub-factors of the target interface and the recorded interface are not empty, the computer device compares whether the fifth node sub-factors of the two are the same, and when the fifth node sub-factors of the two are the same, it is determined that the target interface matches the recorded interface.

[0127] Optionally, if at least one of the fifth node sub-factors of the target interface and the recorded interface is empty, or the fifth node sub-factors of the two are different, the computer device needs to combine other node sub-factors to further determine whether the target interface matches the recorded interface.

[0128] Case 4: When the first node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the second node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the fifth node sub-factor corresponding to the target interface is different from that of the recorded interface, and the similarity between the third node sub-factor corresponding to the target interface and the recorded interface is greater than the first similarity threshold, and / or the similarity between the fourth node sub-factor corresponding to the target interface and the recorded interface is greater than the second similarity threshold, it is determined that the target interface matches the recorded interface.

[0129] When it is impossible to determine whether the target interface matches the recorded interface based on the first node sub-factor, the second node sub-factor, and the fifth node sub-factor, since the same interface usually has a very similar interface layout and / or a very similar interface visual representation, the computer device can further combine at least one of the interface layout and the interface visual representation to determine whether the target interface matches the recorded interface.

[0130] Optionally, when the similarity between the interface layout of the target interface and the recorded interface, and / or the similarity between the interface visual presentation of the target interface and the recorded interface is greater than a threshold, the computer device determines that the target interface matches the recorded interface.

[0131] In some embodiments, when the third node sub-factor is represented by an interface layout vector, the computer device determines the vector distance between the target interface and the interface layout vector of the recorded interface as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiments of the present application.

[0132] In some embodiments, when the fourth node sub-factor is represented by an interface screen vector, the computer device determines the vector distance between the target interface and the interface screen vector of the recorded interface as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiment of the present application.

[0133] Indicatively, Figure 5 As shown, the computer device vectorizes the first interface element skeleton 51 corresponding to the target interface to obtain the first interface layout vector, and vectorizes the second interface element skeleton 52 corresponding to the recorded interface to obtain the second interface layout vector. By calculating the cosine distance between the first interface layout vector and the second interface layout vector, the interface layout similarity between the target interface and the recorded interface is 0.42490305.

[0134] Case 5: When the first node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the second node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the fifth node sub-factor corresponding to the target interface is different from that of the recorded interface, and the similarity between the third node sub-factor corresponding to the target interface and the recorded interface is less than the first similarity threshold, and the similarity between the fourth node sub-factor corresponding to the target interface and the recorded interface is less than the second similarity threshold, it is determined that the target interface does not match the recorded interface.

[0135] Optionally, when the similarity between the interface layout of the target interface and the recorded interface, and the similarity between the interface visual presentation of the target interface and the recorded interface are both less than a threshold, the computer device determines that the target interface does not match the recorded interface.

[0136] Step 201A3: determine the recorded node corresponding to the matched recorded interface as the target node corresponding to the target interface.

[0137] In some embodiments, when a single recorded interface is matched, the computer device determines the recorded node corresponding to the recorded interface as the target node corresponding to the target interface; when multiple recorded interfaces are matched, the computer device determines the recorded node corresponding to the recorded interface with the highest matching degree as the target node corresponding to the target interface.

[0138] Step 201B: determine the target behavior edge corresponding to the target operation behavior involved in the target operation path from the operation trajectory model.

[0139] The target operation behavior is any operation behavior on the target operation path. That is, for each operation behavior on the operation path, the computer device needs to determine the behavior edge corresponding to the operation behavior from the operation trajectory model.

[0140] In some embodiments, the operation trajectory model records the edge information of the behavior edge corresponding to the full amount of operation behaviors. During the instruction recording process, the computer device determines the target behavior edge corresponding to the target operation behavior based on the edge information corresponding to the target operation behavior and the edge information of the recorded operation behavior corresponding to the recorded behavior edge.

[0141] The side information in the operation trajectory model is explained below.

[0142] Side Information

[0143] When reproducing the operation behavior subsequently, it is necessary to know what operation was performed in the interface and the changes in the interface before and after the operation. Therefore, in order to ensure the accuracy of the operation behavior reproduction, the edge information corresponding to the operation behavior at least has the ability to indicate the operation behavior and the interface before and after the operation behavior is executed.

[0144] In some embodiments, the edge information includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of the starting node corresponding to the starting interface before the operation behavior, a terminating node identifier (toNodeID) of the terminating node corresponding to the terminating interface after the operation behavior, and an edge identifier (a unique identifier similar to a node identifier).

[0145] Among them, when the interface before and after the operation behavior does not switch, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are the same; when the interface before and after the operation behavior switches, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are different.

[0146] As an important content in the side information, the behavior information needs to be able to clearly indicate the operation behavior. In a possible implementation, the behavior information includes the action type (ActionType) of the operation behavior.

[0147] In some embodiments, a common action class (CommonAction) is defined, which includes all operation actions of users and systems, so as to ensure the accuracy of recorded operation actions. In addition, the behavior type of each operation behavior belongs to the behavior type class. Optionally, the behavior type class is an enumeration class.

[0148] In an illustrative example, taking the commonly used operations in a mobile phone as an example, the behavior type class may include at least one of the following behavior types: GoHome (return to the home page), RecentApps (recent applications), GoBack (return), StartApp (start application), Click (click), DoubleClick (double-click), LongClick (long press), InputText (input text), Scroll (scroll), Concurrent (collaborative operation, that is, operation behaviors that occur at the same time), Continuous (continuous operation, referring to operation behaviors performed in sequence), Stop (terminate), Wait (wait, a scenario in which the interface changes automatically over time), and Unknown (unknown behavior, as a fallback behavior).

[0149] Since some operation behaviors are directed to interface controls within the interface, in order to ensure that the subsequent operation can determine which interface control to operate on based on the behavior information, in some embodiments, when the behavior type of the operation behavior is determined as behavior information, if the operation behavior is related to the interface control within the interface, the computer device also determines the node element identifier of the node element corresponding to the interface control as the behavior information.

[0150] For example, when the operation behavior is a click operation, the behavior information corresponding to the operation behavior includes the operation type "Click" and the node element identifier of the node element corresponding to the clicked button; for another example, when the operation behavior is a text input operation, the behavior information corresponding to the operation behavior includes the operation type "InputText" and the node element identifier of the node element corresponding to the text input box where the text is entered.

[0151] In a possible implementation, determining the target behavior edge based on the edge information may include the following sub-steps:

[0152] Step 201B1, determining, from the operation trajectory model, a start node corresponding to the start interface before the target operation behavior, and a stop node corresponding to the stop interface after the target operation behavior.

[0153] In one possible implementation, the computer device determines the starting node corresponding to the starting interface based on the node information corresponding to the starting interface, and obtains the starting node identifier of the starting node; determines the terminating node corresponding to the terminating interface based on the node information corresponding to the terminating interface, and obtains the terminating node identifier of the terminating node.

[0154] In some embodiments, in order to ensure the accuracy of the determined termination node, when the duration of the target operation behavior is received and reaches a preset duration (because there may be a response delay in the interface jump), the computer device determines the current interface as the termination interface, and then determines the termination node corresponding to the termination interface.

[0155] Step 201B2: when the target operation behavior is related to the target interface control, determine the node element corresponding to the target interface control from the operation trajectory model.

[0156] In some embodiments, the computer device determines whether the target operation behavior is related to the target interface control. If the target operation behavior is related to the target interface control (i.e., the target operation behavior acts on the target interface control), the computer device determines the node element corresponding to the target interface control. If the target operation behavior is not related to the target interface control (for example, the target operation behavior is the system automatic interface jump behavior after reaching a specified time length), the computer device does not need to determine the node element.

[0157] In a possible implementation, the operation trajectory model includes node element information of node elements corresponding to each recorded interface control in the recorded interface. Accordingly, the computer device determines the node element information corresponding to the target interface control, and matches the node element information with the node element information of the node element corresponding to the recorded interface control, thereby determining the matched recorded node element as the node element corresponding to the target interface control.

[0158] In some embodiments, the computer device determines a node element factor corresponding to the target interface control (which is part of the node element information and is used to characterize the characteristics of the node element from multiple dimensions), and thereby determines the node element corresponding to the target interface control based on the degree of matching between the node element factor and the node element factor corresponding to the recorded interface control.

[0159] Step 201B3: determine a target behavior edge corresponding to the target operation behavior based on the start node, the end node, the node element, and the target behavior type of the target operation behavior.

[0160] In one possible implementation, the computer device matches the starting node identifier of the starting node, the ending node identifier of the ending node, the node element identifier of the node element, and the target behavior type of the target operation behavior with the edge information of the recorded behavior edge corresponding to the recorded operation behavior in the operation trajectory model, thereby determining the matched recorded behavior edge as the target behavior edge corresponding to the target operation behavior.

[0161] Step 201C: Generate target path information of the target operation path based on the target node information and the target edge information of the target behavior edge.

[0162] After determining each target node passed through on the target operation path and the target behavior edges between each target node from the operation trajectory model, the computer device can generate target path information of the target operation path. Based on the target path information, the computer device can determine the nodes on the target operation path and the behavior edges between the nodes.

[0163] In this embodiment, during the instruction recording process, since the target nodes corresponding to the target interface passed by the target operation path and the target behavior edges corresponding to the executed target operation behavior are determined from the recorded nodes and recorded behavior edges in the operation trajectory model, the accuracy and compliance of the operation path recording can be guaranteed; and, during the subsequent instruction execution process, the computer device can simulate the operation with the help of the node information and edge information recorded in the operation trajectory model, thereby improving the reliability of the operation simulation.

[0164] How to generate the target operation path

[0165] In a possible implementation, the target operation path is a path generated by manual operation, that is, during the instruction recording process, the user manually performs the operation required to implement the target instruction. During the user's manual operation, the computer device records each step of the operation and continuously updates the path information of the operation path. When the operation is completed, the generated path information is determined as the target path information.

[0166] Indicatively, Figure 6 As shown, during the command recording process, the user first clicks the first interface control 611 (corresponding node element identifier 48) in the first interface 61, triggering a jump from the first interface 61 to the second interface 62. The user further clicks the second interface control 621 (corresponding node element identifier 56) in the second interface 62, triggering a jump from the second interface 62 to the third interface 63.

[0167] During the user operation process, the computer device determines the nodes corresponding to the first interface 61, the second interface 62 and the third interface 63, and determines the behavior edges corresponding to the two operation behaviors of Click (48) and Click (56).

[0168] In another possible implementation, since the operation trajectory model records the entire interface and operation behavior within the application, the computer device can automatically generate a target operation path for implementing the target instruction based on the instruction requirements of the target instruction and the operation trajectory model without the need for manual operation by the user.

[0169] Regarding the method of automatically generating a target operation path, in one possible implementation, a computer device determines a first node corresponding to a first interface that implements the instruction requirement based on the instruction requirement of the target instruction and the node information of the corresponding nodes of the interface recorded in the operation trajectory model, thereby determining a target operation path to jump to the first interface based on the first node and the edge information in the operation trajectory model.

[0170] In a possible implementation, the node information of the node may also include atomic capabilities information (atomicCapabilities), where the atomic capabilities information is used to indicate the atomic capabilities of the interface, which are the smallest units of capabilities, such as turning on A, turning off A, viewing B, adding C, uploading D, and so on.

[0171] Regarding the method of generating atomic function information, in some embodiments, the computer device generates atomic function information of the interface through a large language model, and determines the atomic function information as node information of the node.

[0172] Optionally, the computer device inputs a screenshot of the interface and prompt information (prompt) into the large language model, instructing the large language model to extract atomic function information of the interface. The atomic function information can be in the form of a list.

[0173] It should be noted that the atomic function information can be generated synchronously with other node information, or it can be generated uniformly when the computer device is idle, or when the number of interfaces for which atomic function information has not been generated reaches a threshold value. This embodiment does not limit this.

[0174] In addition to the atomic function information, the fifth node sub-factor determined in the node information based on the interface function description information can also characterize the interface function.

[0175] Therefore, in one possible implementation, the computer device can determine the first interface that implements the instruction requirements based on the matching degree between the instruction requirements of the target instruction and the atomic function information in the node information, and / or the matching degree between the instruction requirements of the target instruction and the fifth node sub-factor in the node information, and then determine the first node corresponding to the first interface.

[0176] In some embodiments, the computer device may extract the function requirement text from the instruction requirement, and semantically match the function requirement text with the atomic function information, and / or semantically match the function requirement text with the fifth node sub-factor, thereby determining the interface to which the atomic function information and / or the fifth node sub-factor with the highest semantic matching degree belongs as the first interface to implement the instruction requirement. The embodiments of the present application do not limit the specific method for determining the matching degree.

[0177] Optionally, since it may be necessary to perform certain operations in the first interface when implementing the instruction requirements, such as triggering certain interface controls in the first interface, the computer device can further determine the target interface control for the required operation in the first interface and the behavior type of the operation performed on the target interface control based on the instruction requirements and the node element information corresponding to the interface controls in the first interface.

[0178] In a possible implementation, after determining the first interface, the computer device uses the first node as the end point, and determines the target operation path to jump to the first interface based on the jump relationship between nodes represented by the edge information in the operation trajectory model. The starting point of the target operation path can be a specified interface, such as a launcher interface (when executing the target instruction later, it is necessary to return to the specified interface first).

[0179] Optionally, the computer device performs a behavior edge search through a path search algorithm based on the directed connection relationship between different nodes represented by the edge information, and determines the set of behavior edges searched (directly or indirectly) connected to the path start node and the path end node (first node) as a behavior edge list, and the behavior edges in the behavior edge list are arranged according to the execution timing of the operation behaviors corresponding to the behavior edges.

[0180] Optionally, the path search algorithm may be a depth-first search algorithm, a breadth-first search algorithm, a Dijkstra algorithm, an A* search or a D* search algorithm, etc., which is not limited in the embodiments of the present application.

[0181] In an illustrative example, Figure 7As shown, the computer device determines that the interface for implementing the instruction requirement is interface 74 based on the instruction requirement of the target instruction. The computer device uses the launch interface as the starting interface, and the behavior edges of the behavior edge list determined by path search are startApp, Click (49) and Click (57) in sequence. The operation path indicated by the behavior edge is: click the application icon (startApp) in the launcher interface 71, and then click (click) the first interface control 721 in the interface 72 (the corresponding node element is identified as 49), and jump to display the second interface 73. Click the second interface control 731 in the interface 73 (the corresponding node element is identified as 57), and jump to display interface 74.

[0182] Recording control floating window

[0183] Typically, the target operation path for achieving instruction requirements usually includes multiple single-step operation behaviors. In order to facilitate users to understand the recording status of each step of the operation behavior (whether the recording is completed) in real time during the instruction recording process, and control the recording progress (start and end recording), in a possible implementation, during the recording of the target instruction, the computer device displays a recording control pop-up window.

[0184] Optionally, the recording control floating window is continuously displayed during the command recording process to display the real-time recording status during the recording process and control the recording progress.

[0185] In some embodiments, a recording status indicator is displayed in the recording control floating window, and the recording status indicator is used to indicate the recording status of the behavior edge corresponding to the single-step operation behavior, and the recording status includes recording and recording completed. That is, the computer device updates the recording status indicator in the recording control floating window based on the recording status of the behavior edge corresponding to the single-step operation behavior.

[0186] Optionally, the computer device indicates the recording status of the behavior edge by changing the display style of the recording status indicator. For example, a green recording status indicator is used to indicate that the behavior edge recording is completed, and a red recording status indicator indicates that the behavior edge is being recorded (not completed).

[0187] Of course, in addition to using color as a logo display style to represent different record states, the computer device may also use other logo display styles (such as logo size, logo shape, etc.) to represent different record states, which is not limited in this embodiment.

[0188] In some embodiments, in order to ensure that each operation behavior can be accurately recorded, the computer device can shield the user's manual operation during the behavior recording process, and unshield the user's manual operation after completing the behavior recording.

[0189] In some embodiments, a recording completion control is displayed in the recording control floating window, and the recording completion control is used to control the end of recording the target operation path. Accordingly, in response to the triggering operation of the recording completion control in the recording control floating window, the computer device determines that the target operation path recording has been completed and stores the target path information.

[0190] In one possible scenario, when the target operation path is a path generated by manual operation, when the user completes all operations to implement the instruction requirements, the user can click the recording completion control in the recording control floating window to trigger the completion of instruction recording.

[0191] Indicatively, Figure 8 As shown, when recording an instruction for a specific function implemented in the album, after the user opens the album application, a recording control floating window 81 is displayed on the upper layer of the main interface of the album application. When the user clicks the recording control 82 in the recording control floating window 81, the computer device starts to record the operation path. After the user completes the single-step operation in the interface, the computer device displays a recording status indicator 83 of a first color during the process of recording the single-step operation, and displays a recording status indicator 83 of a second color after the recording is completed. When all operations are performed, the user can click the recording completion control 84 in the recording control floating window 81 to end the instruction recording.

[0192] In this embodiment, a recording control floating window is displayed during the instruction recording process, so that the user can understand the real-time recording status of each step of the operation through the floating window, and then speed up or slow down the operation speed to ensure the accuracy of the recording of each step of the operation; in addition, the user can end the recording through the recording completion control in the floating window to ensure the accuracy and completeness of the recording of the operation path that implements the instruction requirements.

[0193] In addition to the recording status indication and recording progress control functions, in some embodiments, the recording control floating window also has a task setting function, wherein the task setting function is used to set a specific task to be executed synchronously during the execution of the target instruction.

[0194] In a possible implementation, Fig. 9 As shown, the process of setting tasks during instruction recording and executing tasks during instruction execution may include the following steps:

[0195] Step 901, in response to the task setting operation in the recording control floating window, determine the task information of the target task indicated by the task setting operation, the task information includes the task parameters of the target task and the execution node of the target task, and the execution node is the node in the target operation path.

[0196] In a possible implementation, the recording control floating window includes a task setting control, and in response to a trigger operation on the task setting control, the recording control floating window displays a task selection list, the task selection list including at least one candidate task. The user can select a target task to be executed when executing the target instruction from the task selection list.

[0197] In other possible implementations, before the command is recorded, the user selects the target task. During the recording process, the recording control pop-up window displays the setting controls corresponding to the target task. Through the setting controls, the user can control when to execute the target task.

[0198] Of course, the recording control floating window can also provide other forms of controls for users to set target tasks, which is not limited in the embodiments of the present application.

[0199] Optionally, the task parameters include at least a task type. In some embodiments, the task type includes at least one of an interface content verification task and an indicator monitoring task.

[0200] Among them, the interface content verification task refers to the task of verifying whether the interface meets specific interface characteristics. For example, verifying whether the interface contains specific interface elements (such as text, buttons), verifying whether the interface layout has changed (it can be compared with a specific interface layout to determine whether it has changed), etc.

[0201] The indicator monitoring task refers to the task of monitoring specific operating indicators, such as monitoring the current size, monitoring the jam rate, monitoring the hardware temperature, etc.

[0202] Of course, when using the task setting function in the recording control floating window during the application test, the interface content verification task and the indicator monitoring task can be tasks corresponding to other items to be tested, and this embodiment does not limit the specific task type.

[0203] Since the target operation path to implement the instruction requirements may pass through multiple nodes, and the target task may only need to be executed at a specific node, the task information of the target task also needs to include the execution node of the target task, which is at least one node in the target operation path.

[0204] In some embodiments, when the target task is an indicator monitoring task, and the indicator monitoring task needs to be executed on a sub-operation path in the target operation path, the execution node includes a start execution node and an end execution node, wherein the start execution node is the node for starting indicator monitoring, and the end execution node is the node for ending indicator monitoring, and at least one other node may be included between the start execution node and the end execution node, or no other nodes may be included.

[0205] Regarding the method for determining the start execution node and the end execution node, in a possible implementation, the recording control floating window includes a task start control, and when a trigger operation is received for the task start control, the computer device determines the node corresponding to the current interface as the start execution node, and switches the task start control to the task end control. When a trigger operation is received for the task end control, the computer device determines the node corresponding to the current interface as the end execution node.

[0206] Indicative, such as Fig.10 As shown, during the instruction recording process, the first interface 1001 of the application is currently displayed, and the upper layer of the first interface 1001 displays a recording control floating window 1011, and the recording control floating window 1011 further includes a task setting control 1012. When a trigger operation is received on the task setting control 1012, a task start control 1013 corresponding to the current monitoring task is displayed in the recording control floating window 1011. When a trigger operation is received on the task start control 1013, the computer device determines the node corresponding to the first interface 1001 as the starting execution node of the current monitoring task, and switches the task start control 1013 to the task end control 1014. After a series of interface jumps and jumps to the second interface 1002, when a trigger operation is received on the task end control 1014, the computer device determines the node corresponding to the second interface 1002 as the end execution node of the current monitoring task, and stops displaying the task end control 1014.

[0207] Step 902: Store the target path information and the task information in association with each other.

[0208] After the instruction recording is completed, in order to ensure that the target task can be executed synchronously when the subsequent instructions are executed, the computer device associates and stores the target path information and the task information.

[0209] Step 903 , in response to the instruction execution operation on the target instruction, in the case of being at the execution node indicated by the task information, executing the target task based on the task parameters of the task information.

[0210] During the process of executing the target instruction, the computer device obtains the task information bound to the target path information corresponding to the target instruction, and based on the task information, determines the task parameters and execution nodes required to execute the target task.

[0211] In some embodiments, during the instruction execution process, it is identified whether the current node matches the execution node, and if so, the target task is executed based on the task parameters.

[0212] Optionally, when the execution node includes a start execution node and an end execution node, the computer device starts executing the target task when identifying the current node as the start execution node; and ends executing the target task when identifying the current node as the end execution node.

[0213] Combination Fig.10 As shown, during the execution of the target instruction, when entering the first interface 1001, the computer device starts current monitoring, and when entering the second interface 1002, the computer device stops current monitoring.

[0214] Step 904: When the target instruction is executed, the task execution result of the target task is output.

[0215] In the case where a target task is set, when the target instruction is completed, the computer device outputs the task execution result of the target task.

[0216] In some embodiments, when the target task is an interface content verification task, the task execution result is an interface content verification result, for example, whether the interface contains specific interface elements, whether the interface layout has not changed, and so on.

[0217] When the target task is an indicator monitoring task, the task execution result is the indicator monitoring result, such as the average current after executing a specific operation, the freeze rate when displaying a specific interface, the CPU temperature when executing a specific operation, etc.

[0218] In the application testing scenario, developers can determine whether the application function is executed normally, the power consumption of the function, and other information based on the task execution results, and perform function debugging based on the task execution results.

[0219] In this embodiment, during the instruction recording process, by using the task setting function of the recording control floating window, the computer device can record the target task to be executed when running to a specific node, and store the task information of the target task in association with the target path information; when the instruction is subsequently executed, the target task can be executed at a specific node based on the task information, and the task execution result can be output. With the help of the task setting function, automated testing can be achieved in the test scenario without the need for testers to manually trigger test tasks, thereby improving the efficiency of automated testing.

[0220] Instruction execution process

[0221] In a possible implementation, Fig.11 As shown, the computer device may include the following steps to simulate the operation behavior based on the target operation path indicated by the target path information:

[0222] Step 202A: Determine target edge information corresponding to the target behavior edge between nodes on the target operation path based on the target path information.

[0223] In one possible implementation, in order to ensure the timing accuracy of the operation behavior, the computer device determines the target edge information corresponding to the target behavior edges between nodes on the target operation path in sequence according to the execution timing of the operation behavior indicated by the edge information in the target path information, thereby simulating the operation according to the target operation behavior indicated by the target edge information.

[0224] Step 202B, based on the target behavior information included in the target side information, simulate the target operation behavior indicated by the target behavior information in the interface.

[0225] The operation behavior in the interface can be a manual operation behavior of the user, such as a trigger operation on a control in the interface; or the operation behavior is a system behavior, such as an automatic interface jump operation after reaching a preset time. Accordingly, the target side information includes target behavior information, which is used to indicate the operation behavior in the interface.

[0226] In some embodiments, when the target operation behavior indicated by the target behavior information is related to the target interface control, the target behavior information includes the node element identifier of the target node element corresponding to the target interface control and the target behavior type of the target operation behavior.

[0227] The node element information is described below.

[0228] Node element information

[0229] In some embodiments, the node element information of the node element includes at least a node element identifier and a node element factor of the node element.

[0230] The node element identifier is a unique identifier assigned to the node element (with no actual meaning), which can be expressed in the form of a string. Different node elements have different node element identifiers.

[0231] Optionally, the node element factor (NodeElement Factor) is composed of several fields that characterize the control features of the interface control from different dimensions, wherein the fields that characterize the control features can be called node element sub-factors. Similar to the role of the node factor, the role of the node element factor is to retrieve and locate the interface control of the feature in the interface.

[0232] When determining whether two node elements are the same node element (i.e., determining whether two interface controls are the same interface control), the computer device matches the field contents with the same field name in the two node element factors, that is, matches the node element sub-factors in different node element factors, thereby determining whether the two node elements are the same node element based on the matching results of multiple dimensions.

[0233] In some embodiments, the node element factor may include at least one of the following node element sub-factors:

[0234] First node element subfactor (packageName)

[0235] In a possible implementation, the computer device determines the first node element sub-factor based on the package name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control package name dimension.

[0236] Optionally, the computer device obtains the package name of the control package referenced by the interface from the interface code, and thereby determines the package name of the control class to which the interface control belongs based on the package name of the control package.

[0237] In an illustrative example, the computer device determines the package name android.widget of the control class as the first node element sub-factor based on the package name of the widget package referenced by the interface.

[0238] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the first node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control package, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0239] Second node element sub-factor (className)

[0240] In a possible implementation, the computer device determines the second node element sub-factor based on the class name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control class name dimension.

[0241] Optionally, the computer device obtains the control code of the interface control and the package name of the control package referenced by the interface from the interface code, thereby determining the package name of the control class to which the interface control belongs based on the control code and the control package name.

[0242] In an illustrative example, the computer device determines that the class name of the control class in which the text box control in the widget is located, android.widget.TextView, is determined as the second node element sub-factor.

[0243] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the second node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control class at the same time, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0244] Third node element sub-factor (ResourceId)

[0245] The resources of the interface control can be layout, string, color, image, etc., and each interface control is assigned a unique ID, namely, resource ID, when it is defined in the layout file. The interface control can be referenced or operated in the code through the resource ID. Since the resource ID of the interface control is unique and generally does not change (it may be updated during the version iteration process), in a possible implementation, the computer device determines the third node element sub-factor based on the resource identifier corresponding to the interface control, that is, describes the control characteristics of the interface control from the resource dimension.

[0246] Optionally, the computer device may start from the interface root node of the interface, traverse each control under each interface root node, and obtain the resource ID of each interface control by means of code printing.

[0247] In an illustrative example, the computer device determines the resource ID "com.phonebrand.someapp:id / tv_name" of the interface control as the third node element sub-factor.

[0248] Fourth node element sub-factor (contentDescription)

[0249] The interface code may include a control description of the interface control (used to describe the control content, usually used for barrier-free functions). For example, for a text box control, the interface code may contain the following control description: text box, text content xxx. Since the control description generally does not change (it may be manually adjusted during version iteration), in a possible implementation, the computer device determines the fourth node element sub-factor based on the control description corresponding to the interface control, that is, describes the control characteristics of the interface control from the control description dimension.

[0250] Optionally, the computer device may extract a control description corresponding to the interface control from the interface code, wherein the control description of the interface control may be empty.

[0251] Fifth node element sub-factor (Text)

[0252] Some interface controls in the interface usually display text so that users can know the purpose of the interface control, such as the word "Upload" displayed on the upload button. Since the text content corresponding to the interface control usually does not change, in a possible implementation, the computer device determines the fifth node element sub-factor based on the text content corresponding to the interface control, that is, describes the control feature of the interface control from the dimension of the control's explicit text content.

[0253] Optionally, the computer device may traverse each control under the root node of the interface, and when the text attribute field of the control is not empty, determine the text content of the text attribute field as the fifth node element sub-factor.

[0254] Optionally, the computer device may also recognize the text displayed by the interface control through OCR, and use the recognized text as the text content corresponding to the interface control, which is not limited in this embodiment.

[0255] Sixth node element subfactor (listOrder)

[0256] For an interface control located in a list, the text content of the same interface control may change dynamically (for example, it may change dynamically with the time period or scene), but its absolute position or relative position in the list will not change. Therefore, in order to improve the comprehensiveness of the node element information and improve the accuracy of the subsequent positioning of the interface control, in a possible implementation, when the interface control is located in a list control, the computer device determines the sixth node element sub-factor based on the position of the interface control in the list control.

[0257] Optionally, the position of the interface control in the list control can be an absolute position, such as being located at the first position in the list, or being located at the last position in the list. The position can be a relative position, such as being located between two interface controls in the list control, or being located after an interface control in the list control.

[0258] Optionally, when a list control is identified in the interface code, the computer device may determine the absolute position or relative position of the interface control based on the arrangement order or rule of each interface control in the list control.

[0259] Seventh node element subfactor (parentChain)

[0260] Each interface control in the interface is in a control chain, where the upper interface control in the control chain is the parent control of the lower interface control, and the top interface control in the control chain is the root control corresponding to the root node of the interface. When the text content, position, size and other parameters of the interface control are fine-tuned, as long as the control function implemented by the interface control remains unchanged, the control chain to which it belongs will usually not change. Therefore, computer devices can describe interface controls from the dimension of the control chain corresponding to the interface control.

[0261] In a possible implementation, the computer device determines the seventh node element sub-factor based on the parent control chain corresponding to the interface control.

[0262] Among them, the parent control chain is composed of the parent controls of the interface controls, and the end point of the parent control chain is the root control corresponding to the interface root node, that is, the first and last controls of the parent control chain of the current interface control are the root control of the interface root node and the current interface control respectively.

[0263] Optionally, the computer device may determine the parent-child relationship between each interface control based on the interface code, and thereby construct a parent control chain corresponding to each interface control based on the parent-child relationship.

[0264] Eighth node element sub-factor (preNodeElementFactor & postNodeElementFactor)

[0265] In order to further improve the comprehensiveness of the control features, the current interface control may be described with the help of other interface controls surrounding the current interface control. In a possible implementation, the computer device determines the eighth node element sub-factor based on the adjacent interface controls of the interface control.

[0266] Optionally, the adjacent interface control may include other interface controls that are adjacent to the current interface control in at least one direction. For example, the adjacent interface control may include a left interface control and a right interface control of the current interface control.

[0267] Optionally, the computer device determines the relative position relationship between the interface controls based on the control layout information corresponding to each interface control in the interface code, or the computer device identifies the interface screen to determine the relative position relationship between the interface controls.

[0268] Optionally, the computer device may determine the node element factor corresponding to the adjacent interface control as the eighth node element sub-factor of the current interface control. For example, the eighth node element sub-factor of the current interface control includes the node tuple factor (preNodeElementFactor) of the left interface control and the node tuple factor (postNodeElementFactor) of the right interface control.

[0269] Regarding the timing of determining the above node element sub-factors, in a possible implementation manner, the above various node element sub-factors are all determined in real time.

[0270] In another possible implementation, the first to fifth node element sub-factors are determined in real time. Since the sixth to eighth node element sub-factors cannot be directly extracted from the interface code but need to be analyzed by certain calculations, the sixth to eighth node element sub-factors are determined by lazy count.

[0271] Among them, lazy calculation means that node element sub-factor calculation is performed only when node element sub-factor comparison is required. For example, when the node element cannot be uniquely located based on the first to fifth node element sub-factors, the computer device calculates the sixth to eighth node element sub-factors.

[0272] Optionally, when the sixth to eighth node element sub-factors adopt lazy calculation, the sixth to eighth node element sub-factors will not participate in serialization, and only the first to fifth node element sub-factors participate in serialization.

[0273] In a possible implementation, when the sixth to eighth node element sub-factors are calculated in a lazy manner, the computer device stores the interface code so as to subsequently calculate the sixth to eighth node element sub-factors based on the interface code analysis.

[0274] In some embodiments, the node element information may further include at least one of the following:

[0275] Control Properties

[0276] Optionally, the control attributes include the type, purpose, and coordinates of the interface control.

[0277] In some embodiments, the control properties may include at least one of the following: stateDescription (control state description), isPassword (whether it is a password input control), isHintText (whether it is a hint text), visible (visibility), checkable (whether it can be checked), editable (whether it can be edited), clickable (whether it can be clicked), scrollable (whether it can be scrolled), longClickable (whether it can be long pressed), focusable (whether it can get focus), enabled (whether it is available), focused (whether it is the focus), checked (whether it is checked), selected (whether it is selected) and bounds (coordinate range).

[0278] Of course, the node element information may also include other control attributes other than the above examples, which is not limited in this embodiment.

[0279] Child node element identifier (children)

[0280] In a possible implementation, the computer device determines a sub-control of the current interface control, and determines a sub-node element identifier of a sub-node element corresponding to the sub-control as the node element information of the current interface control.

[0281] Optionally, the computer device uses the child node element identifier of the direct child control of the current interface control as the node element information, wherein the direct child control refers to a lower-level child control directly connected to the current interface control in a parent-child relationship.

[0282] Optionally, the child node element identifiers can be in list form.

[0283] Parent node element identifier (parentNodeElementId)

[0284] In a possible implementation, the computer device determines the parent control of the current interface control, and determines the parent node element identifier of the child node element corresponding to the parent control as the node element information of the current interface control.

[0285] Optionally, the computer device uses the child node element identifier of the direct parent control of the current interface control as the node element information, wherein the direct parent control refers to the upper parent control directly connected to the current interface control in the parent-child relationship.

[0286] Since the interface controls in the application may be fine-tuned during version updates or development and debugging, or the interface controls may change slightly in different scenarios, in order to ensure the accuracy of the operation of the interface controls, in one possible implementation, when the computer device simulates the operation behavior of the target interface control in the interface, it is necessary to determine the node element factor of the target node element based on the node element identifier of the target node element, and then locate the target interface control from the interface based on the node element factor and the node element factors corresponding to each actual interface control in the interface. Optionally, this step may include the following sub-steps:

[0287] Step 202B1, based on the node element identifier in the target behavior information, obtain the target node element factor of the target node element, the target node element factor includes node element sub-factors that characterize the control features of the target interface control from different dimensions.

[0288] In some embodiments, the computer device obtains the node element factor of the node element corresponding to the node element identifier from the operation trajectory model based on the node element identifier. Optionally, the node element factor may include one or more of the first to eighth node element sub-factors mentioned above.

[0289] Step 202B2: Determine the target interface control from the interface based on the target node element factor and the node element factors corresponding to each interface control in the interface.

[0290] Before performing node element factor comparison, the computer device needs to generate node element factors corresponding to each interface control in the interface. The generated node element factors may include one or more of the first to eighth node element sub-factors, and different node element sub-factors may be generated synchronously or under the request of comparison requirements.

[0291] The process of generating the node element factor may refer to the above embodiment, which will not be described in detail in this embodiment.

[0292] Optionally, the computer device may locate the target interface control from the interface based on a single node element sub-factor in the node element factor, or a combination of multiple node element sub-factors.

[0293] In some embodiments, the node element sub-factors that characterize the control features from different dimensions can be divided into first precision sub-factors, second precision sub-factors, and third precision sub-factors. In the process of locating the interface control based on the node element factor, the interface control can be located based on all or part of the precision sub-factors in descending order of precision.

[0294] Optionally, the first node element subfactor and the second node element subfactor belong to the first precision subfactor, the third node element subfactor, the fourth node element subfactor and the fifth node element subfactor belong to the second precision subfactor, and the sixth node element subfactor, the seventh node element subfactor and the eighth node element subfactor belong to the third precision subfactor.

[0295] In the process of positioning the interface control based on the node element factor, the priority of the first precision sub-factor is higher than the priority of the second precision sub-factor, and the priority of the second precision sub-factor is higher than the priority of the third precision sub-factor.

[0296] In the process of locating the interface control, if a unique interface control can be located based on the first precision sub-factor, there is no need to perform positioning based on the second precision sub-factor and the third precision sub-factor.

[0297] If the unique interface control cannot be located based on the first precision sub-factor, the interface control location is continued based on the second precision. If the unique interface control can be located based on the second precision sub-factor, there is no need to locate it based on the third precision sub-factor.

[0298] If the only interface control cannot be located based on the second precision sub-factor, the interface control is located based on the third precision.

[0299] Optionally, the first precision subfactors of two identical interface controls must be exactly the same. In other words, if the first precision subfactors of two interface controls are different, the two interface controls must not be the same interface control.

[0300] Optionally, when the first precision subfactors of the two interface controls are completely identical, the computer device further determines whether the two interface controls are identical based on the second precision subfactors of the two interface controls.

[0301] Optionally, if the second precision subfactors of the two interface controls are exactly the same, then the two interface controls are determined to be the same interface control. If the second precision subfactors of the two interface controls are completely different, then the two interface controls are determined to be different interface controls. If the second precision subfactors of the two interface controls are partially the same and partially different, the computer device further determines whether the two interface controls are the same based on the third precision subfactors of the two interface controls.

[0302] Optionally, if the third precision subfactors of the two interface controls are exactly the same, the two interface controls are determined to be the same interface control; if the third precision subfactors of the two interface controls are different, the two interface controls are determined to be different interface controls. Alternatively, if the matching degree of the third precision subfactors of the two interface controls is higher than a matching degree threshold, the two interface controls are determined to be the same interface control; if the matching degree of the third precision subfactors of the two interface controls is lower than the matching degree threshold, the two interface controls are determined to be different interface controls.

[0303] Step 202B3: simulating the target operation behavior on the target interface control in the interface based on the target behavior type included in the target behavior information.

[0304] After locating the target interface control in the interface, the computer device generates an operation behavior event for the target interface control based on the target behavior type in the target behavior information, thereby simulating the target operation behavior for the target interface control in the interface based on the operation behavior event.

[0305] In this embodiment, node element factors that describe the characteristics of the interface control in multiple dimensions are generated from dimensions such as the package name of the control class to which the interface control belongs, the type, the resource identifier corresponding to the interface control, the control description of the interface control, the text content displayed by the interface control, the position of the interface control in the list, the control chain in which the interface control is located, and the positional relationship between the interface control and other interface controls. This improves the comprehensiveness and accuracy of the description of the interface control, and is conducive to improving the accuracy of subsequent retrieval and positioning of the interface control based on the node element sub-factors.

[0306] Furthermore, compared to executing instructions based on script files, if the interface content changes slightly, the instruction will fail. In the embodiment of the present application, since the interface elements are dynamically positioned based on the node tuple factor, even if the interface content changes slightly, the interface elements for the required operation can be retrieved and located, thereby improving the success rate of instruction execution.

[0307] In some embodiments, in order to further improve the accuracy and success rate of instruction execution, after each single-step operation is completed, the computer device detects whether the node corresponding to the interface is the expected node. If it is the expected node, the subsequent operation behavior will continue to be executed. If it is not the expected node, it will further determine whether it is necessary to re-dynamically plan the path.

[0308] In a possible implementation, when the target operation behavior is executed, the computer device determines the second node corresponding to the current interface.

[0309] In the case where the second node and the termination node indicated by the target edge information are the same node, the computer device determines that it is at the desired node and continues to perform subsequent operation behaviors.

[0310] In some scenarios, when the switching relationship between interfaces is fine-tuned due to an application version update, it may happen that after performing an operation in the interface corresponding to the starting node indicated by the target edge information, the interface jumps to another interface other than the interface corresponding to the ending node indicated by the target edge information. For example, during the command recording process, the target operation path recorded is A→B→C→D, and during the command execution process, after performing an operation in interface A, it does not jump to interface B, but directly jumps to interface C.

[0311] In the case where the second node and the termination node indicated by the target edge information are different nodes, the computer device needs to further determine whether the operation path needs to be dynamically planned based on whether the second node is located in the target operation path and its position in the target operation path.

[0312] Optionally, when the second node and the termination node indicated by the target edge information are different nodes, and the second node is located after the termination node indicated by the target edge information in the target operation path, the computer device continues to simulate the operation behavior based on the target operation path.

[0313] When the second node is different from the terminating node indicated by the target edge information, and the second node is located in the target operation path and after the terminating node, even if the interface jump relationship indicated by the target edge information is not followed, subsequent operations in the target operation path can continue to be executed starting from the second node. Therefore, the computer device does not need to dynamically plan the operation path, but starts from the second node and continues to simulate the operation behavior according to the target operation path.

[0314] Schematically, the target operation path recorded during the instruction recording process is A→B→C→D, and during the instruction execution process, after the operation behavior is performed in interface A, it does not jump to interface B, but directly jumps to interface C. Since the node corresponding to interface C is located after the node corresponding to interface B, the computer device continues to simulate the operation behavior from the node corresponding to interface C.

[0315] Optionally, when the second node is different from the terminating node indicated by the target edge information, and the second node is located in the target operation path and before the terminating node, if starting from the second node and continuing to simulate the operation behavior according to the target operation path, it will result in an infinite loop, so the computer device needs to re-plan the path.

[0316] When the second node is different from the termination node indicated by the target edge information, and the second node is outside the target operation path (i.e., the second node does not match any node in the target operation path), the computer device cannot continue to perform simulation operations according to the target operation path. In order to ensure the normal execution of the instruction, the computer device needs to dynamically plan the operation path.

[0317] Optionally, when the second node and the termination node indicated by the target edge information are different nodes and the second node does not belong to the target operation path, a dynamic operation path is determined based on the second node and the last node of the target operation path; and the operation behavior is simulated based on the dynamic operation path.

[0318] In one possible implementation, the computer device uses the second node as the path start node and the last node in the target operation path as the path end node, and based on the directed connection relationship between different nodes represented by the edge information of the recorded behavior edges in the operation trajectory model, performs a behavior edge search through a path search algorithm to obtain a set of behavior edges that are (directly or indirectly) connected to the path start node and the path end node, and then determines the dynamic operation path based on the set of behavior edges.

[0319] When the dynamic operation path is determined, the computer device simulates the operation behavior based on the dynamic operation path.

[0320] Schematically, the target operation path recorded during the instruction recording process is A→B→C→D, and during the instruction execution process, after the operation behavior is performed in interface A, it does not jump to interface B, but jumps to interface E. Since the node corresponding to interface E does not belong to the target operation path, the computer device takes interface E as the starting point and interface D as the end point, and determines the dynamic operation path as E→F→D through the path search algorithm.

[0321] In this embodiment, when the computer device simulates the operation of the interface control, it locates the target interface control for the required operation by matching the node element factors corresponding to each interface control in the current interface and the target node element factors of the target node elements in the target edge information, thereby ensuring the accuracy of the interface control operation; and, in the event that the target interface control is fine-tuned, the fine-tuned target interface control can still be located by factor matching, which helps to improve the success rate of instruction execution.

[0322] In addition, when the computer device is executing the target instruction, if the actual execution path deviates from the target operation path, it can re-plan the dynamic operation path, further improving the success rate of instruction execution.

[0323] Command self-healing

[0324] Since the interface content or the jump relationship between interfaces may be fine-tuned during the application version update process, in order to improve the availability of recorded instructions during subsequent application version upgrades, the computer device can reversely correct the operation trajectory model during the instruction execution process based on the path information of the actual operation path used in the actual operation process, so that the instruction has self-healing capabilities.

[0325] In a possible implementation, in a process of simulating an operation behavior based on a target operation path indicated by the target path information, the computer device determines actual path information of the actual operation path.

[0326] When the target instruction is executed successfully, the computer device updates the operation trajectory model based on difference information between the actual path information and the target path information, where the difference information includes at least one of difference node information and difference edge information.

[0327] Optionally, after updating the operation trajectory model based on the difference node information, the impact of fine-tuning the interface and the elements in the interface during the application version update can be reduced; after updating the operation trajectory model based on the difference edge information, the impact of the jump relationship between interfaces during the application version update can be reduced. Through this instruction self-healing capability, the operation trajectory model can be adaptively updated during the application version update process to improve the execution success rate of instructions under different versions of applications.

[0328] Please refer to Fig.12 , which shows a structural block diagram of an instruction execution device provided by an exemplary embodiment of the present application. The device includes:

[0329] The recording module 1201 is used to generate target path information of a target operation path for realizing the target instruction based on an operation trajectory model during the recording process of the target instruction, wherein the operation trajectory model includes node information of nodes corresponding to the interface in the application and edge information of behavior edges corresponding to the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes;

[0330] The execution module 1202 is used to simulate an operation behavior based on the target operation path indicated by the target path information in response to an instruction execution operation on the target instruction.

[0331] Optionally, the recording module 1201 is used to:

[0332] Determine a target node corresponding to a target interface involved in the target operation path from the operation trajectory model;

[0333] Determine, from the operation trajectory model, a target behavior edge corresponding to the target operation behavior involved in the target operation path;

[0334] Based on the target node information of the target node and the target edge information of the target behavior edge, target path information of the target operation path is generated.

[0335] Optionally, the recording module 1201 is used to:

[0336] Based on interface features of different dimensions of the target interface, generating node factors corresponding to the target interface;

[0337] Determining the recorded interface that matches the target interface based on the node factor of the recorded node corresponding to the recorded interface in the operation trajectory model and the node factor corresponding to the target interface;

[0338] The recorded node corresponding to the matched recorded interface is determined as the target node corresponding to the target interface.

[0339] Optionally, the recording module 1201 is used to:

[0340] Determine, from the operation trajectory model, a starting node corresponding to a starting interface before the target operation behavior and an ending node corresponding to a terminating interface after the target operation behavior;

[0341] In a case where the target operation behavior is related to a target interface control, determining a node element corresponding to the target interface control from the operation trajectory model;

[0342] The target behavior edge corresponding to the target operation behavior is determined based on the start node, the end node, the node element, and the target behavior type of the target operation behavior.

[0343] Optionally, the target operation path is a path generated by manual operation, or,

[0344] The target operation path is a path automatically generated based on the instruction requirement of the target instruction and the operation trajectory model.

[0345] Optionally, when the target operation path is a path automatically generated based on the instruction requirement of the target instruction and the operation trajectory model, the recording module 1201 is further used to:

[0346] Determine, based on the instruction requirement of the target instruction and the node information of the nodes corresponding to the recorded interfaces in the operation trajectory model, a first node corresponding to the first interface that implements the instruction requirement;

[0347] Based on the first node and the edge information in the operation trajectory model, determine the target operation path for jumping to the first interface.

[0348] Optionally, the target operation path includes multiple single-step operation behaviors;

[0349] The recording module 1201 is further used for:

[0350] During the recording process of the target instruction, displaying a recording control floating window;

[0351] Based on the recording status of the behavior edge corresponding to the single-step operation behavior, updating the recording status indicator in the recording control floating window, wherein the recording status includes recording in progress and recording completed;

[0352] In response to a triggering operation on a recording completion control in the recording control floating window, the target path information is stored.

[0353] Optionally, the recording module 1201 is further used for:

[0354] In response to a task setting operation in the recording control floating window, determining task information of a target task indicated by the task setting operation, the task information including task parameters of the target task and an execution node of the target task, the execution node being a node in the target operation path;

[0355] Associating and storing the target path information and the task information;

[0356] The execution module 1202 is further configured to:

[0357] In response to the instruction execution operation on the target instruction, executing the target task based on the task parameters of the task information when in the execution node indicated by the task information;

[0358] When the target instruction is executed, the task execution result of the target task is output.

[0359] Optionally, the task parameter includes a task type, and the task type includes at least one of an interface content verification task and an indicator monitoring task;

[0360] Among them, when the task type is an indicator monitoring task, the execution node includes a start execution node and an end execution node, the start execution node is a node for starting indicator monitoring, and the end execution node is a node for ending indicator monitoring.

[0361] Optionally, the execution module 1202 is used to:

[0362] Based on the target path information, determine target edge information corresponding to the target behavior edge between nodes on the target operation path;

[0363] Based on the target behavior information included in the target side information, the target operation behavior indicated by the target behavior information is simulated in the interface.

[0364] Optionally, when the target operation behavior indicated by the target behavior information is related to a target interface control, the target behavior information includes a node element identifier of a target node element corresponding to the target interface control and a target behavior type of the target operation behavior;

[0365] The execution module 1202 is used to:

[0366] Based on the node element identifier in the target behavior information, a target node element factor of the target node element is acquired, wherein the target node element factor includes node element sub-factors that characterize control features of the target interface control from different dimensions;

[0367] Determine the target interface control from the interface based on the target node element factor and the node element factors corresponding to each interface control in the interface;

[0368] Based on the target behavior type included in the target behavior information, the target operation behavior on the target interface control is simulated in the interface.

[0369] Optionally, the execution module 1202 is further configured to:

[0370] When the target operation behavior is executed, determining a second node corresponding to the current interface;

[0371] When the second node and the termination node indicated by the target edge information are different nodes, and the second node is located after the termination node indicated by the target edge information in the target operation path, continue to simulate the operation behavior based on the target operation path;

[0372] When the second node and the termination node indicated by the target edge information are different nodes, and the second node does not belong to the target operation path, a dynamic operation path is determined based on the second node and the last node of the target operation path; and the operation behavior is simulated based on the dynamic operation path.

[0373] Optionally, the execution module 1202 is further configured to:

[0374] In a process of simulating an operation behavior based on the target operation path indicated by the target path information, determining actual path information of an actual operation path;

[0375] An updating module is used to update the operation trajectory model based on difference information between the actual path information and the target path information when the target instruction is successfully executed, wherein the difference information includes at least one of difference node information and difference edge information.

[0376] In summary, in the embodiment of the present application, an operation trajectory model is used to record the operation process in the interface, wherein the operation trajectory model includes node information of the corresponding nodes of the interface and edge information of the corresponding behavior edges of the operation behavior in the interface. With the help of the node information and edge information in the operation trajectory model, the computer device can determine the nodes and behavior edges involved in the operation path of implementing the instruction in the instruction recording stage, and then generate the path information of the operation path based on the node information of the node and the edge information of the behavior edge, so as to simulate the corresponding operation behavior based on the path information in the subsequent instruction execution stage. Compared with complex script files, the space occupied by storing path information is smaller; and simulation based on the node information and edge information indicated by the path information can improve the accuracy of interface positioning and operation behavior restoration during the simulation process, which helps to improve the success rate of instruction execution.

[0377] See also Fig.13 , Fig.13 13 is a schematic diagram of a computer device provided by an exemplary embodiment of the present application. The computer device may also include one or more of the following components: a processor 1310 and a memory 1320 .

[0378] Optionally, the processor 1310 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1320, and calling data stored in the memory 1320. Optionally, the processor 1310 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1310 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above baseband chip may not be integrated into the processor 1310, but may be implemented by a separate chip.

[0379] The memory 1320 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1320 includes a non-transitory computer-readable storage medium. The memory 1320 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0380] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device, and the computer device may include more (such as communication components, display components) or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0381] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the instruction execution method described in the above embodiment.

[0382] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the instruction execution method described in the above embodiment.

[0383] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0384] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for executing an instruction, characterized in that: The method comprises: During the recording process of the target instruction, target path information of the target operation path for realizing the target instruction is generated based on the operation trajectory model, wherein the operation trajectory model includes node information of nodes corresponding to the interface in the application and edge information of behavior edges corresponding to the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes; In response to an instruction execution operation on the target instruction, an operation behavior is simulated based on the target operation path indicated by the target path information.

2. The method according to claim 1, characterized in that The generating target path information of the target operation path for realizing the target instruction based on the operation trajectory model includes: Determine a target node corresponding to a target interface involved in the target operation path from the operation trajectory model; Determine, from the operation trajectory model, a target behavior edge corresponding to the target operation behavior involved in the target operation path; Based on the target node information of the target node and the target edge information of the target behavior edge, target path information of the target operation path is generated.

3. The method according to claim 2, characterized in that The step of determining, from the operation trajectory model, a target node corresponding to a target interface involved in the target operation path, comprises: Based on interface features of different dimensions of the target interface, generating node factors corresponding to the target interface; Determining the recorded interface that matches the target interface based on the node factor of the recorded node corresponding to the recorded interface in the operation trajectory model and the node factor corresponding to the target interface; The recorded node corresponding to the matched recorded interface is determined as the target node corresponding to the target interface.

4. The method according to claim 2, characterized in that: The determining, from the operation trajectory model, a target behavior edge corresponding to the target operation behavior involved in the target operation path includes: Determine, from the operation trajectory model, a starting node corresponding to a starting interface before the target operation behavior and an ending node corresponding to a terminating interface after the target operation behavior; In a case where the target operation behavior is related to a target interface control, determining a node element corresponding to the target interface control from the operation trajectory model; The target behavior edge corresponding to the target operation behavior is determined based on the start node, the end node, the node element, and the target behavior type of the target operation behavior.

5. The method according to any one of claims 1 to 4, characterized in that: The target operation path is a path generated by manual operation, or, The target operation path is a path automatically generated based on the instruction requirement of the target instruction and the operation trajectory model.

6. The method according to claim 5, characterized in that In the case where the target operation path is a path automatically generated based on the instruction requirement of the target instruction and the operation trajectory model, the method further includes: Determine, based on the instruction requirement of the target instruction and the node information of the nodes corresponding to the recorded interfaces in the operation trajectory model, a first node corresponding to the first interface that implements the instruction requirement; Based on the first node and the edge information in the operation trajectory model, determine the target operation path for jumping to the first interface.

7. The method according to any one of claims 1 to 4, characterized in that: The target operation path includes multiple single-step operation behaviors; The method further comprises: During the recording process of the target instruction, displaying a recording control floating window; Based on the recording status of the behavior edge corresponding to the single-step operation behavior, updating the recording status indicator in the recording control floating window, wherein the recording status includes recording in progress and recording completed; In response to a triggering operation on a recording completion control in the recording control floating window, the target path information is stored.

8. The method according to claim 7, characterized in that The method further comprises: In response to a task setting operation in the recording control floating window, determining task information of a target task indicated by the task setting operation, the task information including task parameters of the target task and an execution node of the target task, the execution node being a node in the target operation path; Associating and storing the target path information and the task information; In response to the instruction execution operation on the target instruction, executing the target task based on the task parameters of the task information when in the execution node indicated by the task information; When the target instruction is executed, the task execution result of the target task is output.

9. The method according to claim 8, characterized in that The task parameters include a task type, and the task type includes at least one of an interface content verification task and an indicator monitoring task; Among them, when the task type is an indicator monitoring task, the execution node includes a start execution node and an end execution node, the start execution node is a node for starting indicator monitoring, and the end execution node is a node for ending indicator monitoring.

10. The method according to any one of claims 1 to 4, characterized in that: The target operation path simulation operation behavior based on the target path information indicated includes: Based on the target path information, determine target edge information corresponding to the target behavior edge between nodes on the target operation path; Based on the target behavior information included in the target side information, the target operation behavior indicated by the target behavior information is simulated in the interface.

11. The method according to claim 10, characterized in that In the case where the target operation behavior indicated by the target behavior information is related to a target interface control, the target behavior information includes a node element identifier of a target node element corresponding to the target interface control and a target behavior type of the target operation behavior; The simulating, in the interface, the target operation behavior indicated by the target behavior information based on the target behavior information included in the target side information includes: Based on the node element identifier in the target behavior information, a target node element factor of the target node element is acquired, wherein the target node element factor includes node element sub-factors that characterize control features of the target interface control from different dimensions; Determine the target interface control from the interface based on the target node element factor and the node element factors corresponding to each interface control in the interface; Based on the target behavior type included in the target behavior information, the target operation behavior on the target interface control is simulated in the interface.

12. The method according to claim 10, characterized in that The method further comprises: When the target operation behavior is executed, determining a second node corresponding to the current interface; When the second node and the termination node indicated by the target edge information are different nodes, and the second node is located after the termination node indicated by the target edge information in the target operation path, continue to simulate the operation behavior based on the target operation path; When the second node and the termination node indicated by the target edge information are different nodes, and the second node does not belong to the target operation path, a dynamic operation path is determined based on the second node and the last node of the target operation path; and the operation behavior is simulated based on the dynamic operation path.

13. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In a process of simulating an operation behavior based on the target operation path indicated by the target path information, determining actual path information of an actual operation path; When the target instruction is executed successfully, the operation trajectory model is updated based on difference information between the actual path information and the target path information, where the difference information includes at least one of difference node information and difference edge information.

14. An instruction execution device, characterized in that: The device comprises: A recording module, used to generate target path information of a target operation path for realizing the target instruction based on an operation trajectory model during the recording process of the target instruction, wherein the operation trajectory model includes node information of nodes corresponding to the interface in the application and edge information of behavior edges corresponding to the operation behavior in the interface, and the target operation path is composed of nodes and behavior edges between nodes; An execution module is used to simulate an operation behavior based on the target operation path indicated by the target path information in response to an instruction execution operation on the target instruction.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the instruction execution method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the instruction execution method according to any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the instruction execution method as described in any one of claims 1 to 13.

Images