Operation recording method, computer equipment and computer program product

Abstract

The embodiment of the invention discloses an operation recording method, computer equipment and a computer program product, and belongs to the technical field of artificial intelligence. The method comprises the steps that node information of nodes corresponding to interfaces in the operation process is determined, different interfaces correspond to different nodes, and the node information comprises information representing interface characteristics of the interfaces from different dimensions; determining side information of a behavior side corresponding to an operation behavior in the interface in an operation process, wherein the side information comprises information used for representing the operation behavior and node jump conditions before and after the operation behavior; on the basis of the side information, path information of an operation path corresponding to the operation process is determined, and the path information is used for representing a time sequence between the operation behaviors; and generating an operation track model based on the node information, the side information and the path information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an operation recording method, a computer device, and a computer program product. Background Art

[0002] In the daily use of the terminal, users can record specific operation processes in the application by taking screenshots or recording the screen. However, taking screenshots and recording the screen require users to actively trigger them, the efficiency of operation recording is low, and it is not convenient to trace back and reproduce them later. Summary of the invention

[0003] The embodiment of the present application provides an operation recording method, a computer device and a computer program product. The technical solution is as follows:

[0004] On the one hand, an embodiment of the present application provides an operation recording method, the method comprising:

[0005] Determine node information of nodes corresponding to the interface during the operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interface from different dimensions;

[0006] Determine edge information of a behavior edge corresponding to the operation behavior in the interface during the operation process, wherein the edge information includes information for characterizing the operation behavior and node jump conditions before and after the operation behavior;

[0007] Based on the edge information, determine path information of an operation path corresponding to the operation process, where the path information is used to characterize a time sequence between the operation behaviors;

[0008] An operation trajectory model is generated based on the node information, the edge information, and the path information.

[0009] On the other hand, an embodiment of the present application provides an operation recording device, the device comprising:

[0010] A first determination module is used to determine node information of nodes corresponding to interfaces during operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interfaces from different dimensions;

[0011] A second determination module is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process, wherein the edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior;

[0012] A third determination module, configured to determine path information of an operation path corresponding to the operation process based on the edge information, wherein the path information is used to characterize a time sequence between the operation behaviors;

[0013] A generation module is used to generate an operation trajectory model based on the node information, the edge information and the path information.

[0014] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the operation recording method described in the above aspects.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the operation recording method described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the operation recording method described in the above aspects.

[0017] In the embodiment of the present application, an operation trajectory model is used to record the operation process. Among them, the nodes in the operation trajectory model correspond to the interface during the operation process, the edges correspond to the operation behaviors in the interface, and the operation path characterizes the execution timing of the operation behaviors during the operation process. When recording the operation process, the interface during the operation process is recorded using node information that characterizes the interface characteristics from different dimensions, thereby ensuring the accuracy of the interface record and facilitating the subsequent interface positioning; the operation behaviors in the interface and the interface jumps before and after the operation are recorded using edge information, and the timing between the operation behaviors is recorded using path information, thereby ensuring the accuracy of the operation process record and facilitating the subsequent operation reproduction. Since the operation recording process does not require manual triggering by the user, the efficiency of the operation recording is relatively high, and compared to screen recording, the information in the operation trajectory model can be used to backtrack and reproduce, reducing the difficulty of reproducing historical operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an operation trajectory model corresponding to an operation process shown in an exemplary embodiment of the present application;

[0019] Figure 2 A flowchart of an operation recording method provided by an exemplary embodiment of the present application is shown;

[0020] Figure 3 is a schematic diagram of a skeletonized interface layout shown in an exemplary embodiment of the present application;

[0021] Figure 4is a schematic diagram of an interface layout similarity calculation process shown in an exemplary embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the structure of node information shown in an exemplary embodiment of the present application;

[0023] Figure 6 is a schematic diagram of the structure of node element information shown in an exemplary embodiment of the present application;

[0024] Figure 7 is a schematic diagram of the structure of side information shown in an exemplary embodiment of the present application;

[0025] Figure 8 is a schematic diagram of an operation path shown in an exemplary embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of the structure of path information shown in an exemplary embodiment of the present application;

[0027] Fig.10 is a schematic diagram showing the relationship between node groups and nodes according to an exemplary embodiment of the present application;

[0028] Fig.11 is a schematic diagram of the structure of node group information shown in an exemplary embodiment of the present application;

[0029] Fig.12 A structural block diagram of an operation recording device provided by another exemplary embodiment of the present application is shown;

[0030] Fig.13 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0033] In the related art, users usually take screenshots or screen recordings to record a certain operation interface or a series of interface operation processes. When a certain operation needs to be performed again later, the user can review the screenshots or screen recordings and manually reproduce the operation.

[0034] However, screenshots and screen recordings require manual triggering by users and take up a large amount of storage space to store the generated screenshots and videos. On the other hand, operation backtracking based on screenshots and videos is inefficient and requires manual execution by users.

[0035] In the embodiment of the present application, an operation trajectory model (Trajectory Structure) is used to record the operation process. Figure 1 As shown, the operation trajectory model 13 corresponding to the operation process is composed of nodes 131, behavior edges 132 and operation paths 133. Among them, the node 131 corresponds to the interface 11 passed through during the operation; the behavior edge 132 corresponds to the operation behavior in the interface, which is used to connect the edges 12 of different nodes (interface switching occurs after the operation) or the same node (no interface switching occurs after the operation); the operation path 133 corresponds to the operation behavior sequence.

[0036] In addition, in order to ensure the accuracy of the operation process record, the node information 141 of the node 131 represents the interface features of the interface from different dimensions to ensure the accuracy of the subsequent interface retrieval and positioning; the edge information 142 of the behavior edge 132 indicates the operation behavior and the node jumps before and after the operation behavior to ensure the accurate restoration of the operation in the subsequent interface; the path information 143 of the operation path 133 represents the execution sequence between different operation behaviors to ensure that complex operations are executed in sequence.

[0037] Compared with screenshots and videos, the operation trajectory model has a lower data volume and can be automatically generated without manual triggering by the user. In addition, it can automatically backtrack and reproduce based on node information, edge information, and path information, improving the efficiency of operation backtracking and reproduction.

[0038] The operation recording method provided in the embodiment of the present application can be executed by a computer device with an interface display function. The computer device can be a smart phone, a tablet computer, a smart wearable device, a personal computer, etc. In the following embodiments, the operation recording method is described by taking the computer device as an example.

[0039] Please refer to Figure 2 , which shows a flow chart of an operation recording method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device as an example, and the method may include the following steps:

[0040] Step 201 : determining node information of nodes corresponding to interfaces during operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interfaces from different dimensions.

[0041] The interface is a visual user interface, such as a desktop, an application program interface, a system interface, etc. In the embodiment of the present application, different interfaces in the operation process correspond to respective nodes, and different nodes represent the interface features of their respective corresponding interfaces through their respective node information, that is, different nodes can be distinguished based on the node information.

[0042] Optionally, dimensions characterizing interface features may include interface content dimension, interface layout dimension, interface function dimension, interface description dimension, the window to which the interface belongs, etc. The following embodiments will describe in detail the specific content contained in the node information.

[0043] The interface in this embodiment is not limited to the visible range of the screen, but includes the completed interface within and outside the visible range of the screen. Accordingly, before and after the interface content is scrolled, even if the displayed content within the visible range of the screen changes significantly, the interface before and after the scrolling will not be judged as a different interface.

[0044] The different interfaces in the embodiments of the present application refer to interfaces with large differences in interface layout, content, and style. Usually, when an interface switch occurs, the interfaces before and after the switch are determined to be different interfaces. For example, when the text or images in the interface change dynamically, or when the list elements in the interface change, they are not determined to be different interfaces; when clicking on a control in the interface to trigger an interface jump, the interfaces before and after the jump are determined to be different interfaces. The process of determining different interfaces will be described in detail in the following embodiments.

[0045] Step 202 : determining edge information of a behavior edge corresponding to an operation behavior in the interface during the operation process, wherein the edge information includes information for characterizing the operation behavior and node jump conditions before and after the operation behavior.

[0046] Among them, the behavior edge (Edge) corresponding to the operation behavior is a directed edge, and the direction of the behavior edge is the interface jump direction, that is, the behavior edge points from the interface before the jump to the interface after the jump.

[0047] Optionally, the operation behavior refers to a manual operation behavior of the user, such as a trigger operation on a control in the interface; or, the operation behavior refers to a system behavior, such as an automatic interface jump operation after a preset time period has been reached.

[0048] In some embodiments, a plurality of operation behavior types are preset, and the edge information of the behavior edge includes the operation behavior type to which the operation behavior belongs.

[0049] Regarding the recording method of the node jump situation before and after the operation behavior, in a possible implementation, the side information includes information about the corresponding nodes of the interface before and after the operation behavior is performed. For example, the side information includes the node identifier of the node corresponding to the interface before the operation behavior is performed, and the node identifier of the node corresponding to the interface after the operation behavior is performed. Among them, the interface identifiers corresponding to the interfaces before and after the operation behavior can be different interface identifiers (i.e., the interface jump occurs after the operation), or they can be the same interface identifier (i.e., the operation does not cause an interface jump).

[0050] Step 203: Based on the side information, determine the path information of the operation path corresponding to the operation process, where the path information is used to characterize the time sequence between the operation behaviors.

[0051] Since a series of operations may need to be performed during the operation process and multiple interfaces may need to be jumped, the computer device needs to sort the behavior edges corresponding to the operation behaviors according to the execution sequence between the operation behaviors to obtain the operation path (Path). Among them, the operation path is a directed path composed of multiple nodes and behavior edges between nodes.

[0052] Correspondingly, the path information of the operation path is composed of edge information, and can indicate the order relationship between the edge information and the corresponding behavior edge.

[0053] Step 204: Generate an operation trajectory model based on the node information, edge information, and path information.

[0054] Based on the node information of the nodes involved in the operation process, the edge information of the behavior edges involved, and the path information of the operation paths involved, the computer device generates an operation trajectory model corresponding to the operation process.

[0055] In some embodiments, the computer device stores the generated operation trajectory model so as to subsequently perform interface retrieval, positioning, and operation reproduction based on the operation trajectory model.

[0056] In summary, the operation trajectory model is used to record the operation process in the embodiment of the present application. Among them, the nodes in the operation trajectory model correspond to the interface during the operation process, the edges correspond to the operation behaviors in the interface, and the operation path characterizes the execution timing of the operation behaviors during the operation process. When recording the operation process, the interface during the operation process is recorded using node information that characterizes the interface characteristics from different dimensions, thereby ensuring the accuracy of the interface record and facilitating the subsequent interface positioning; the operation behaviors in the interface and the interface jumps before and after the operation are recorded using edge information, and the timing between the operation behaviors is recorded using path information, thereby ensuring the accuracy of the operation process record and facilitating the subsequent operation reproduction. Since the operation recording process does not require manual triggering by the user, the efficiency of the operation recording is relatively high, and compared to screen recording, the information in the operation trajectory model can be used to backtrack and reproduce, reducing the difficulty of reproducing historical operations.

[0057] Node Information

[0058] In some embodiments, the node information of a node includes at least a node identifier and a node factor of the node.

[0059] Accordingly, in a possible implementation, determining the node information of the node corresponding to the interface during the operation process may include: determining the node factor of the node corresponding to the interface during the operation process, the node factor including node sub-factors that characterize the interface characteristics of the interface from different dimensions; determining the node identifier of the node; and determining the node factor and the node identifier as the node information of the node.

[0060] The node identifier is a unique identifier assigned to a node (with no actual meaning), which can be represented in the form of a string. Different nodes have different node identifiers, while the same node has the same node identifier.

[0061] Optionally, a node factor is composed of several fields that characterize interface features from different dimensions, wherein the fields that characterize interface features can be referred to as node sub-factors. When determining whether two nodes are the same node, the computer device matches the field contents with the same field name in the two node factors, that is, matches the node sub-factors in different node factors, thereby determining whether the two nodes are the same node based on the matching results of multiple dimensions.

[0062] In some embodiments, the node factor may include at least one of the following node sub-factors:

[0063] First node subfactor (windowName)

[0064] Since the same interface is definitely located in the same window, and interfaces located in different windows are definitely not the same interface, the computer device can determine whether two nodes are the same node based on the dimension of the window to which the interface belongs.

[0065] In a possible implementation, the computer device determines the first node sub-factor based on the window name of the window to which the interface belongs.

[0066] Among them, the window name of the window to which the interface belongs can be extracted from the interface code.

[0067] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the first node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, since different interfaces may be located in the same window, the computer device cannot directly determine that the two nodes with the same window name belong to the same node, but needs to make further determinations based on other node sub-factors.

[0068] Second node sub-factor (operableElements)

[0069] The interface usually includes a large number of operable elements, such as button controls, expandable list controls, text input box controls, upload controls, etc., and these operable elements can reflect the interface characteristics of the interface to a certain extent.

[0070] Therefore, the computer device can determine whether two nodes are the same node based on the dimension of the operable elements contained in the interface.

[0071] Affected by factors such as version updates or dynamic changes in content, the element content of the operable elements in the same interface may change (for example, the button display text of a button control changes, or the list item content in an expandable list control changes), but the element type of the operable elements in the same interface usually does not change. Therefore, in a possible implementation, the computer device determines the element type of the operable element, and performs information summary processing on the string obtained by splicing the element type, thereby determining the information summary result as the second node sub-factor. Among them, the element type of the operable element can be extracted from the interface code.

[0072] Among them, since the character string after the information digest is of fixed length, it is convenient to directly compare the information digest results later.

[0073] Optionally, the computer device performs MD5 processing on the string obtained by concatenating the element types, and determines the obtained string as the second node sub-factor.

[0074] Illustratively, when the interface includes 2 button controls, 1 text box control, and 3 radio selection controls, the computer device performs MD5 processing on the concatenated string Button_TextBox_RadioButton, and determines the obtained 64-bit string as the second node sub-factor.

[0075] Of course, in addition to being represented by a character string obtained from the information digest, the computer device may also determine an element type list (list) composed of element types as the second node sub-factor, which is not limited in this embodiment.

[0076] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the second node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, in order to avoid judging two different simple interfaces (i.e., interfaces containing a small number of operable elements) as the same interface, the computer device needs to make further judgments based on other node sub-factors.

[0077] The third node subfactor (layoutVector)

[0078] Affected by factors such as version updates, the interface content of the same interface may change significantly, but the overall layout of the interface usually changes less. Therefore, the computer device can determine whether two nodes are the same node based on the dimension of interface layout.

[0079] In a possible implementation, the computer device determines the third node sub-factor based on the interface layout of the interface.

[0080] Optionally, in order to facilitate the subsequent comparison of interface layouts, the third node sub-factor is represented by a vector, wherein vector elements of different dimensions in the vector are used to represent different interface elements, and the vectors obtained by vectorizing the interface layouts of different interfaces have the same number of dimensions (for example, all are 512-dimensional vectors).

[0081] Regarding the vectorization method of the interface layout, in a possible implementation, the computer device extracts the interface layout information of the interface, performs vectorization processing on the interface layout information, and determines the obtained interface layout vector as the third node sub-factor. The interface layout information of the interface can be extracted from the interface code.

[0082] Optionally, the computer device generates an interface element skeleton based on the interface layout information, and the interface element skeleton is only used to characterize the position and size of different interface elements in the interface, and does not include the specific element content of the interface elements. Further, the computer device vectorizes the interface element skeleton to obtain an interface layout vector.

[0083] Indicatively, Figure 3 As shown, the computer device performs skeleton processing on the interface layout information of the interface 31 to obtain an interface element skeleton 32.

[0084] Fourth node sub-factor (screenVector)

[0085] In addition to determining different interfaces (nodes) from the dimension of interface layout, different interfaces can also be determined from a more intuitive visual level, that is, a computer device can determine whether two nodes are the same node from the dimension of the interface screen.

[0086] In a possible implementation, the computer device determines the fourth node sub-factor based on an interface screen of the interface.

[0087] In some embodiments, the computer device directly determines the interface screen as the fourth node sub-factor, and the fourth node sub-factor includes the pixel value of each pixel in the interface screen.

[0088] In order to reduce the amount of storage and simplify the amount of calculation in subsequent comparisons, in other embodiments, the computer device performs vector processing on the interface screen of the interface, and determines the obtained interface screen vector as the fourth node sub-factor. The number of dimensions of the interface screen vectors obtained by vector processing of different interface screens is consistent.

[0089] Regarding the method of vectorizing the interface screen, optionally, the computer device extracts image features of the interface screen through a neural network (such as a CNN network, a DNN network, etc.), and uses the extracted image feature vector as the fourth node sub-factor. Alternatively, the computer device can extract color features (such as obtained through a color histogram), texture features (such as obtained through texture analysis), and shape features (such as obtained through edge detection) of the interface screen, and obtain the fourth node sub-factor by vectorizing the color features, texture features, and shape features. The embodiments of the present application do not limit the specific method of screen vectorization.

[0090] Fifth node sub-factor (summaryFunction)

[0091] During the interface development phase, developers usually add interface function description information to the interface code. Even if the interface effect is subsequently updated, the interface function description information usually does not change. Therefore, the computer device can determine whether two nodes are the same node from the dimension of interface function.

[0092] In a possible implementation, the computer device determines the fifth node sub-factor based on interface function description information of the interface, wherein the interface function description information can be extracted from the interface code.

[0093] It should be noted that, since not all interfaces will add interface function description information, if the interface function description information cannot be obtained, the fifth node sub-factor may be empty.

[0094] Furthermore, when subsequently identifying whether two nodes are the same node, the computer device can compare whether the fifth node sub-factors in the node factors of the two nodes are the same. If they are the same, it is determined that the two nodes are the same node; if they are not the same, the computer device needs to make further determinations based on other node sub-factors.

[0095] In an embodiment of the present application, node factors that describe interface features in multiple dimensions are generated from the dimensions of the window to which the interface belongs, the dimensions of the operable elements within the interface, the interface layout, the interface screen, and the interface function description, thereby improving the comprehensiveness and accuracy of the description of the interfaces corresponding to different nodes, thereby facilitating the accuracy of subsequent interface retrieval and positioning based on the node factors and improving the accuracy of operation reproduction.

[0096] Since the node identifier of a node is unique, in order to avoid assigning different node identifiers to the same node, in one possible implementation, the computer device identifies whether the current interface belongs to a recorded interface based on the node factor of the node corresponding to the current interface and the node factor of the recorded node corresponding to the recorded interface.

[0097] When it is determined based on the node factor that the interface belongs to a recorded interface, the recorded node identifier of the recorded node corresponding to the recorded interface is determined as the node identifier of the node corresponding to the interface.

[0098] When it is determined based on the node factor that the interface does not belong to a recorded interface, a node identifier is assigned to a node corresponding to the interface, wherein the assigned node identifier is different from a recorded node identifier of a recorded node corresponding to the recorded interface.

[0099] Optionally, the recorded interface is the interface corresponding to the recorded node in the operation trajectory model. For example, the recorded interface is the interface involved in the previous operation process. Accordingly, when it is determined that the current interface does not belong to the recorded interface, the computer device determines that the current interface is a newly appeared interface during the operation process; when it is determined that the current interface belongs to the recorded interface, the computer device determines that the current interface is an interface that has appeared in the previous operation process.

[0100] Optionally, the computer device may determine whether the current interface is a recorded interface based on a single node sub-factor in the node factor, or in combination with multiple node sub-factors in the node factor.

[0101] Optionally, when the current interface is a recorded interface, the computer device may update the node factor of the node corresponding to the recorded interface to the node factor of the node corresponding to the current interface, thereby improving the timeliness of the node factor in the node information.

[0102] In some embodiments, when the node factor includes the first to fifth node sub-factors described above, the computer device may identify whether the current interface is a recorded interface in the following cases.

[0103] Case 1: When the first node subfactor of the node is different from that of the recorded node, it is determined that the interface does not belong to the recorded interface.

[0104] Since the same interface belongs to the same window, by comparing the first node subfactors of the current node and the recorded node, and when the first node subfactor of the current node is different from the first node subfactor of the recorded node, the computer device determines that the current interface does not belong to the recorded interface.

[0105] When the first node sub-factor of the current node is the same as the first node sub-factor of the recorded node, the computer device needs to further determine whether the current interface belongs to the recorded interface in combination with other node sub-factors.

[0106] Optionally, when making subsequent determinations in combination with other node sub-factors, the computer device only needs to consider the recorded nodes that have the same first node sub-factor as the current node, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0107] Case 2: When the second node sub-factor of the node is different from that of the recorded node, it is determined that the interface does not belong to the recorded interface.

[0108] Since the types of operable elements included in different interfaces are usually different, by comparing the second node subfactors of the current node with those of the recorded nodes, and when the second node subfactor of the current node is different from the second node subfactor of the recorded node, the computer device determines that the current interface does not belong to the recorded interface.

[0109] When the second node sub-factor of the current node is the same as the second node sub-factor of the recorded node (they may be two different interfaces containing very few operable elements), the computer device needs to combine other node sub-factors to further determine whether the current interface belongs to the recorded interface.

[0110] Optionally, when making subsequent determinations in combination with other node factors, the computer device only needs to consider the recorded nodes that have the same second node sub-factor as the current node, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0111] Case 3: When the fifth node sub-factor of the node is the same as that of the recorded node, it is determined that the interface belongs to the recorded interface.

[0112] If the functions of the two interfaces are completely the same, then the two interfaces should belong to the same interface. Therefore, in some embodiments, when the fifth node sub-factors of the current node and the recorded node are both not empty, the computer device compares whether the fifth node sub-factors of the two are the same, and when the fifth node sub-factors of the two are the same, it is determined that the current interface belongs to the recorded interface.

[0113] Optionally, if at least one of the fifth node sub-factors of the current node and the recorded node is empty, or the fifth node sub-factors of the two are different, the computer device needs to combine other node sub-factors to further determine whether the current interface belongs to the recorded interface.

[0114] Case 4: When the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is greater than the first similarity threshold, and / or the similarity between the fourth node sub-factor of the node and the recorded node is greater than the second similarity threshold, it is determined that the interface belongs to the recorded interface.

[0115] When it is impossible to determine whether the current interface is a recorded interface based on the first node sub-factor, the second node sub-factor, and the fifth node sub-factor, since the same interface usually has a very similar interface layout and / or a very similar interface visual representation, the computer device can further combine at least one of the interface layout and the interface visual representation to determine whether the current interface is a recorded interface.

[0116] Optionally, when the similarity between the interface layout of the current interface and the recorded interface, and / or the similarity between the interface visual presentation of the current interface and the recorded interface is greater than a threshold, the computer device determines that the current interface belongs to the recorded interface.

[0117] In some embodiments, when the third node sub-factor is represented by an interface layout vector, the computer device determines the vector distance between the interface layout vector of the current node and the recorded node as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiments of the present application.

[0118] In some embodiments, when the fourth node sub-factor is represented by an interface screen vector, the computer device determines the vector distance between the interface screen vector of the current node and the recorded node as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiment of the present application.

[0119] Indicative, such as Figure 4 As shown, the computer device vectorizes the first interface element skeleton 41 corresponding to the first interface to obtain the first interface layout vector, and vectorizes the second interface element skeleton 42 corresponding to the second interface to obtain the second interface layout vector. By calculating the cosine distance between the first interface layout vector and the second interface layout vector, the interface layout similarity between the first interface and the second interface is obtained to be 0.42490305.

[0120] Case 5: When the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is less than the first similarity threshold, and the similarity between the fourth node sub-factor of the node and the recorded node is less than the second similarity threshold, it is determined that the interface does not belong to the recorded interface.

[0121] Optionally, when the similarity between the interface layout of the current interface and the recorded interface, and the similarity between the interface visual presentation of the current interface and the recorded interface are both less than a threshold, the computer device determines that the current interface does not belong to the recorded interface.

[0122] It should be noted that the difference between the fifth node subfactor of the node and the recorded node may include that the fifth node subfactor of at least one of the node and the recorded node is empty, or that the fifth node subfactors of both the node and the recorded node are not empty and are different.

[0123] In this embodiment, the computer device determines whether the current node and the recorded node belong to the same node based on the node factors of the current node and the recorded node, thereby avoiding assigning different node identifiers to the same node when generating the operation trajectory model, thereby improving the recording accuracy of the operation trajectory model.

[0124] In order to facilitate subsequent interface retrieval based on functions, such as being able to retrieve the hotspot settings subpage under the settings page based on the "hotspot settings" function indicated by "I want to set up a mobile phone hotspot", in a possible implementation, the node information of the node may also include atomic capabilities information (atomicCapabilities). The atomic capabilities information is used to indicate the atomic capabilities of the interface, which are the smallest units of functions, such as turning on A, turning off A, viewing B, adding C, uploading D, and so on.

[0125] Regarding the method of generating atomic function information, in some embodiments, the computer device generates atomic function information of the interface through a large language model, and determines the atomic function information as node information of the node.

[0126] Optionally, the computer device inputs a screenshot of the interface and prompt information (prompt) into the large language model, instructing the large language model to extract atomic function information of the interface. The atomic function information can be in the form of a list.

[0127] It should be noted that the atomic function information can be generated synchronously with other node information, or it can be generated uniformly when the computer device is idle, or when the number of interfaces for which atomic function information has not been generated reaches a threshold value. This embodiment does not limit this.

[0128] In other possible implementations, in order to further improve the efficiency of subsequent interface retrieval, in addition to using the atomic function information as part of the node information, the computer device may use at least one of the following methods to mine more node information:

[0129] 1. Perform OCR (Optical Character Recognition) on the images in the interface and extract the text information contained in the images as node information.

[0130] 2. Generate the image description corresponding to the image in the interface as node information. For example, you can use the Image2Text model to generate the image description corresponding to the image.

[0131] 3. Use the TAG (tag) system to generate image classification labels as node information.

[0132] 4. Use the large language model to generate an interface information summary and / or an interface summary description as node information.

[0133] 5. Get the interface text information from all text type controls in the interface and use it as node information.

[0134] As an important part of the interface, the interface control carries the mission of receiving user operations and triggering interface content updates or interface jumps. In the operation process, most of the operation behaviors are for the interface controls. In order to accurately record the operation behaviors for the interface controls during the operation process, the computer device also needs to determine the node element information of the node element (NodeElement) corresponding to each interface control in the interface.

[0135] Among them, the node element belongs to the node. In addition, in order to improve the accuracy of subsequent positioning of the interface control based on the node element information, the node element information includes information representing the control features of the interface control from different dimensions.

[0136] Furthermore, in order to associate the node element with the node to which it belongs, so that the node element contained in the node can be determined based on the node information of the node, in a possible implementation, the computer device determines the node element corresponding to the interface root node (rootView) of the interface as the node information of the node. Among them, each interface control in the interface is a child node of the interface root node.

[0137] Optionally, the computer device extracts the interface root node from the interface code, and then determines the node element corresponding to the interface root node as the node information.

[0138] Combined with the above description of node information, in an illustrative example, the node class can be defined as:

[0139] class Node(

[0140] var id:String=generateUUID(),

[0141] val nodeFactor:NodeFactor,

[0142] var atomicCapabilities:List <string>? = null,

[0143] var rootView:NodeElement? =null )

[0145] The node factor class can be defined as:

[0146] data class NodeFactor(

[0147] val windowName: String? =null,

[0148] val operableElement: String? =DEFAULT_OPERABLE_ELEMEMTS,

[0149] val screenVector: String? =null,

[0150] val layoutVector:List <string>? = null,

[0151] val summaryFunction: String? =null )

[0153] Based on the above definitions of node class and node factor class, the structure of node information is as follows: Figure 5 shown.

[0154] Node element information

[0155] In some embodiments, the node element information of the node element includes at least a node element identifier and a node element factor of the node element.

[0156] Accordingly, in a possible implementation, determining the node element information may include: determining a node element identifier of a node element corresponding to an interface control in the interface; determining a node element factor of the node element, the node element factor including node element sub-factors that characterize control features of the interface control from different dimensions; and determining the node element identifier and the node element factor as the node element information of the node element.

[0157] The node element identifier is a unique identifier assigned to the node element (with no actual meaning), which can be expressed in the form of a string. Different node elements have different node element identifiers.

[0158] Optionally, the node element factor (NodeElement Factor) is composed of several fields that characterize the control features of the interface control from different dimensions, wherein the fields that characterize the control features can be called node element sub-factors. Similar to the role of the node factor, the role of the node element factor is to retrieve and locate the interface control of the feature in the interface.

[0159] When determining whether two node elements are the same node element (i.e., determining whether two interface controls are the same interface control), the computer device matches the field contents with the same field name in the two node element factors, that is, matches the node element sub-factors in different node element factors, thereby determining whether the two node elements are the same node element based on the matching results of multiple dimensions.

[0160] In some embodiments, the node element factor may include at least one of the following node element sub-factors:

[0161] First node element subfactor (packageName)

[0162] In a possible implementation, the computer device determines the first node element sub-factor based on the package name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control package name dimension.

[0163] Optionally, the computer device obtains the package name of the control package referenced by the interface from the interface code, and thereby determines the package name of the control class to which the interface control belongs based on the package name of the control package.

[0164] In an illustrative example, the computer device determines the package name android.widget of the control class as the first node element sub-factor based on the package name of the widget package referenced by the interface.

[0165] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the first node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control package, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0166] Second node element sub-factor (className)

[0167] In a possible implementation, the computer device determines the second node element sub-factor based on the class name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control class name dimension.

[0168] Optionally, the computer device obtains the control code of the interface control and the package name of the control package referenced by the interface from the interface code, thereby determining the package name of the control class to which the interface control belongs based on the control code and the control package name.

[0169] In an illustrative example, the computer device determines that the class name of the control class in which the text box control in the widget is located, android.widget.TextView, is determined as the second node element sub-factor.

[0170] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the second node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control class at the same time, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0171] Third node element sub-factor (ResourceId)

[0172] The resources of the interface control can be layout, string, color, image, etc., and each interface control is assigned a unique ID, namely, resource ID, when it is defined in the layout file. The interface control can be referenced or operated in the code through the resource ID. Since the resource ID of the interface control is unique and generally does not change (it may be updated during the version iteration process), in a possible implementation, the computer device determines the third node element sub-factor based on the resource identifier corresponding to the interface control, that is, describes the control characteristics of the interface control from the resource dimension.

[0173] Optionally, the computer device may start from the interface root node of the interface, traverse each control under each interface root node, and obtain the resource ID of each interface control by means of code printing.

[0174] In an illustrative example, the computer device determines the resource ID "com.phonebrand.someapp:id / tv_name" of the interface control as the third node element sub-factor.

[0175] Fourth node element sub-factor (contentDescription)

[0176] The interface code may include a control description of the interface control (used to describe the control content, usually used for barrier-free functions). For example, for a text box control, the interface code may contain the following control description: text box, text content xxx. Since the control description generally does not change (it may be manually adjusted during version iteration), in a possible implementation, the computer device determines the fourth node element sub-factor based on the control description corresponding to the interface control, that is, describes the control characteristics of the interface control from the control description dimension.

[0177] Optionally, the computer device may extract a control description corresponding to the interface control from the interface code, wherein the control description of the interface control may be empty.

[0178] Fifth node element sub-factor (Text)

[0179] Some interface controls in the interface usually display text so that users can know the purpose of the interface control, such as the word "Upload" displayed on the upload button. Since the text content corresponding to the interface control usually does not change, in a possible implementation, the computer device determines the fifth node element sub-factor based on the text content corresponding to the interface control, that is, describes the control feature of the interface control from the dimension of the control's explicit text content.

[0180] Optionally, the computer device may traverse each control under the root node of the interface, and when the text attribute field of the control is not empty, determine the text content of the text attribute field as the fifth node element sub-factor.

[0181] Optionally, the computer device may also recognize the text displayed by the interface control through OCR, and use the recognized text as the text content corresponding to the interface control, which is not limited in this embodiment.

[0182] Sixth node element subfactor (listOrder)

[0183] For an interface control located in a list, the text content of the same interface control may change dynamically (for example, it may change dynamically with the time period or scene), but its absolute position or relative position in the list will not change. Therefore, in order to improve the comprehensiveness of the node element information and improve the accuracy of the subsequent positioning of the interface control, in a possible implementation, when the interface control is located in a list control, the computer device determines the sixth node element sub-factor based on the position of the interface control in the list control.

[0184] Optionally, the position of the interface control in the list control can be an absolute position, such as being located at the first position in the list, or being located at the last position in the list. The position can be a relative position, such as being located between two interface controls in the list control, or being located after an interface control in the list control.

[0185] Optionally, when a list control is identified in the interface code, the computer device may determine the absolute position or relative position of the interface control based on the arrangement order or rule of each interface control in the list control.

[0186] Seventh node element subfactor (parentChain)

[0187] Each interface control in the interface is in a control chain, where the upper interface control in the control chain is the parent control of the lower interface control, and the top interface control in the control chain is the root control corresponding to the root node of the interface. When the text content, position, size and other parameters of the interface control are fine-tuned, as long as the control function implemented by the interface control remains unchanged, the control chain to which it belongs will usually not change. Therefore, computer devices can describe interface controls from the dimension of the control chain corresponding to the interface control.

[0188] In a possible implementation, the computer device determines the seventh node element sub-factor based on the parent control chain corresponding to the interface control.

[0189] Among them, the parent control chain is composed of the parent controls of the interface controls, and the end point of the parent control chain is the root control corresponding to the interface root node, that is, the first and last controls of the parent control chain of the current interface control are the root control of the interface root node and the current interface control respectively.

[0190] Optionally, the computer device may determine the parent-child relationship between each interface control based on the interface code, and thereby construct a parent control chain corresponding to each interface control based on the parent-child relationship.

[0191] Eighth node element sub-factor (preNodeElementFactor & postNodeElementFactor)

[0192] In order to further improve the comprehensiveness of the control features, the current interface control may be described with the help of other interface controls surrounding the current interface control. In a possible implementation, the computer device determines the eighth node element sub-factor based on the adjacent interface controls of the interface control.

[0193] Optionally, the adjacent interface control may include other interface controls that are adjacent to the current interface control in at least one direction. For example, the adjacent interface control may include a left interface control and a right interface control of the current interface control.

[0194] Optionally, the computer device determines the relative position relationship between the interface controls based on the control layout information corresponding to each interface control in the interface code, or the computer device identifies the interface screen to determine the relative position relationship between the interface controls.

[0195] Optionally, the computer device may determine the node element factor corresponding to the adjacent interface control as the eighth node element sub-factor of the current interface control. For example, the eighth node element sub-factor of the current interface control includes the node tuple factor (preNodeElementFactor) of the left interface control and the node tuple factor (postNodeElementFactor) of the right interface control.

[0196] Regarding the timing of determining the above node element sub-factors, in a possible implementation manner, the above various node element sub-factors are all determined in real time.

[0197] In another possible implementation, the first to fifth node element sub-factors are determined in real time. Since the sixth to eighth node element sub-factors cannot be directly extracted from the interface code but need to be analyzed by certain calculations, the sixth to eighth node element sub-factors are determined by lazy count.

[0198] Among them, lazy calculation means that node element sub-factor calculation is performed only when node element sub-factor comparison is required. For example, when the node element cannot be uniquely located based on the first to fifth node element sub-factors, the computer device calculates the sixth to eighth node element sub-factors.

[0199] Optionally, when the sixth to eighth node element sub-factors adopt lazy calculation, the sixth to eighth node element sub-factors will not participate in serialization, and only the first to fifth node element sub-factors participate in serialization.

[0200] In a possible implementation, when the sixth to eighth node element sub-factors are calculated in a lazy manner, the computer device stores the interface code so as to subsequently calculate the sixth to eighth node element sub-factors based on the interface code analysis.

[0201] In this embodiment, node element factors that describe the characteristics of the interface control in multiple dimensions are generated from dimensions such as the package name of the control class to which the interface control belongs, the type, the resource identifier corresponding to the interface control, the control description of the interface control, the text content displayed by the interface control, the position of the interface control in the list, the control chain in which the interface control is located, and the positional relationship between the interface control and other interface controls. This improves the comprehensiveness and accuracy of the description of the interface control, and is conducive to improving the accuracy of subsequent retrieval and positioning of the interface control based on the node element sub-factors.

[0202] In some embodiments, the node element sub-factors that characterize the control features from different dimensions can be divided into first precision sub-factors, second precision sub-factors, and third precision sub-factors. In the process of locating the interface control based on the node element factor, the interface control can be located based on all or part of the precision sub-factors in descending order of precision.

[0203] Optionally, the first node element subfactor and the second node element subfactor belong to the first precision subfactor, the third node element subfactor, the fourth node element subfactor and the fifth node element subfactor belong to the second precision subfactor, and the sixth node element subfactor, the seventh node element subfactor and the eighth node element subfactor belong to the third precision subfactor.

[0204] In the process of positioning the interface control based on the node element factor, the priority of the first precision sub-factor is higher than the priority of the second precision sub-factor, and the priority of the second precision sub-factor is higher than the priority of the third precision sub-factor.

[0205] In the process of locating the interface control, if a unique interface control can be located based on the first precision sub-factor, there is no need to perform positioning based on the second precision sub-factor and the third precision sub-factor.

[0206] If the unique interface control cannot be located based on the first precision sub-factor, the interface control location is continued based on the second precision. If the unique interface control can be located based on the second precision sub-factor, there is no need to locate it based on the third precision sub-factor.

[0207] If the only interface control cannot be located based on the second precision sub-factor, the interface control is located based on the third precision.

[0208] Optionally, the first precision subfactors of two identical interface controls must be exactly the same. In other words, if the first precision subfactors of two interface controls are different, the two interface controls must not be the same interface control.

[0209] Optionally, when the first precision subfactors of the two interface controls are completely identical, the computer device further determines whether the two interface controls are identical based on the second precision subfactors of the two interface controls.

[0210] Optionally, if the second precision subfactors of the two interface controls are exactly the same, then the two interface controls are determined to be the same interface control. If the second precision subfactors of the two interface controls are completely different, then the two interface controls are determined to be different interface controls. If the second precision subfactors of the two interface controls are partially the same and partially different, the computer device further determines whether the two interface controls are the same based on the third precision subfactors of the two interface controls.

[0211] Optionally, if the third precision subfactors of the two interface controls are exactly the same, the two interface controls are determined to be the same interface control; if the third precision subfactors of the two interface controls are different, the two interface controls are determined to be different interface controls. Alternatively, if the matching degree of the third precision subfactors of the two interface controls is higher than a matching degree threshold, the two interface controls are determined to be the same interface control; if the matching degree of the third precision subfactors of the two interface controls is lower than the matching degree threshold, the two interface controls are determined to be different interface controls.

[0212] In some embodiments, the node element information may further include at least one of the following:

[0213] Control properties

[0214] Optionally, the control attributes include the type, purpose, and coordinates of the interface control.

[0215] In some embodiments, the control properties may include at least one of the following: stateDescription (control state description), isPassword (whether it is a password input control), isHintText (whether it is a hint text), visible (visibility), checkable (whether it can be checked), editable (whether it can be edited), clickable (whether it can be clicked), scrollable (whether it can be scrolled), longClickable (whether it can be long pressed), focusable (whether it can get focus), enabled (whether it is available), focused (whether it is the focus), checked (whether it is checked), selected (whether it is selected) and bounds (coordinate range).

[0216] Of course, the node element information may also include other control attributes other than the above examples, which is not limited in this embodiment.

[0217] Child node element identifier (children)

[0218] In a possible implementation, the computer device determines a sub-control of the current interface control, and determines a sub-node element identifier of a sub-node element corresponding to the sub-control as the node element information of the current interface control.

[0219] Optionally, the computer device uses the child node element identifier of the direct child control of the current interface control as the node element information, wherein the direct child control refers to a lower-level child control directly connected to the current interface control in a parent-child relationship.

[0220] Optionally, the child node element identifiers can be in list form.

[0221] Parent node element identifier (parentNodeElementId)

[0222] In a possible implementation, the computer device determines the parent control of the current interface control, and determines the parent node element identifier of the child node element corresponding to the parent control as the node element information of the current interface control.

[0223] Optionally, the computer device uses the child node element identifier of the direct parent control of the current interface control as the node element information, wherein the direct parent control refers to the upper parent control directly connected to the current interface control in the parent-child relationship.

[0224] Combined with the above description of node information, in an illustrative example, the node element class can be defined as:

[0225] class NodeElement(

[0226] val id:String=generateUUID(),

[0227] val nodeElementFactor:NodeElementFactor,

[0228] val stateDescription:String?=null,

[0229] val isPassword:Boolean=false,

[0230] val isHintText:Boolean=false,

[0231] val visible:Boolean=false,

[0232] val checkable:Boolean=false,

[0233] valeditable:Boolean=false,

[0234] val clickable:Boolean=false,

[0235] val scrollable:Boolean=false,

[0236] val longClickable:Boolean=false,

[0237] val focusable:Boolean=false,

[0238] val enabled:Boolean=false,

[0239] val focused:Boolean=false,

[0240] val selected:Boolean=false,

[0241] val bounds:Rect=Rect(),

[0242] var children:List <nodeelement>=listof(),

[0243] var parentNodeElementId: String? =null, )

[0245] The node element factor class can be defined as:

[0246] data class NodeElementFactor(

[0247] val packageName:String,

[0248] var className:String,

[0249] var resourceId: String? null,

[0250] val contentDescription: String? =null,

[0251] var text: String? =null,

[0252] ){

[0253] var listOrder: Int? = null

[0254] var parentChain:Set <nodeelementfactor>? =null

[0255] var preNodeElementFactor:NodeElementFactor? =null

[0256] var postNodeElementFactor:NodeElementFactor? =null

[0257] }

[0258] Based on the above definitions of node element class and node element factor class, the structure of node element information is as follows: Figure 6 shown.

[0259] Side Information

[0260] When reproducing the operation behavior subsequently, it is necessary to know what operation was performed in the interface and the changes in the interface before and after the operation. Therefore, in order to ensure the accuracy of the operation behavior reproduction, the edge information corresponding to the operation behavior at least has the ability to indicate the operation behavior and the interface before and after the operation behavior is executed.

[0261] In some embodiments, the edge information includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of the starting node corresponding to the starting interface before the operation behavior, an ending node identifier (toNodeID) of the ending node corresponding to the ending interface after the operation behavior, and an edge identifier (a unique identifier similar to a node identifier and a node element identifier).

[0262] Correspondingly, when determining the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process, the computer device determines the behavior information of the operation behavior in the interface during the operation process, and determines the starting node identifier of the starting node corresponding to the starting interface before the operation behavior, and the ending node identifier of the ending node corresponding to the ending interface after the operation behavior, thereby determining the behavior information, the starting node identifier and the ending node identifier as the edge information of the behavior edge corresponding to the operation behavior.

[0263] Among them, when the interface before and after the operation behavior does not switch, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are the same; when the interface before and after the operation behavior switches, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are different.

[0264] As an important content in the side information, the behavior information needs to be able to clearly indicate the operation behavior. In a possible implementation, the behavior information includes the action type (ActionType) of the operation behavior.

[0265] In some embodiments, a common action class (CommonAction) is defined, which includes all operation actions of users and systems, so as to ensure the accuracy of recorded operation actions. In addition, the behavior type of each operation behavior belongs to the behavior type class. Optionally, the behavior type class is an enumeration class.

[0266] In an illustrative example, taking the commonly used operations in a mobile phone as an example, the behavior type class may include at least one of the following behavior types: GoHome (return to the home page), RecentApps (recent applications), GoBack (return), StartApp (start application), Click (click), DoubleClick (double-click), LongClick (long press), InputText (input text), Scroll (scroll), Concurrent (collaborative operation, that is, operation behaviors that occur at the same time), Continuous (continuous operation, referring to operation behaviors performed in sequence), Stop (terminate), Wait (wait, a scenario in which the interface changes automatically over time), and Unknown (unknown behavior, as a fallback behavior).

[0267] Since some operation behaviors are directed to interface controls within the interface, in order to ensure that the subsequent operation can determine which interface control to operate on based on the behavior information, in some embodiments, when the behavior type of the operation behavior is determined as behavior information, if the operation behavior is related to the interface control within the interface, the computer device also determines the node element identifier of the node element corresponding to the interface control as the behavior information.

[0268] For example, when the operation behavior is a click operation, the behavior information corresponding to the operation behavior includes the operation type "Click" and the node element identifier of the node element corresponding to the clicked button; for another example, when the operation behavior is a text input operation, the behavior information corresponding to the operation behavior includes the operation type "InputText" and the node element identifier of the node element corresponding to the text input box where the text is entered.

[0269] In addition, since the operation process may involve a series of continuous operation behaviors, in order to facilitate the tracing back of the previous operation behavior before the current operation behavior, in a possible implementation, when the operation behavior has a previous operation, the computer device determines the operation information of the previous operation as the edge information of the behavior edge corresponding to the operation behavior, that is, the edge information also includes the previous operation information (preAction).

[0270] Illustratively, for the operation of clicking the login button, the preceding operations may include entering an account number in a first text box and entering a password in a second text box.

[0271] Optionally, the preceding operation information may be identified in the form of a list, where each item in the list corresponds to a preceding operation, and the preceding operation may be represented by a jump between nodes, such as a starting node identifier → an ending node identifier (the starting node identifier and the ending node identifier may be the same).

[0272] Combined with the above description of edge information, in an illustrative example, the behavior edge class can be defined as:

[0273] class Edge(

[0274] val fromNodeID: String,

[0275] val toNodeID: String,

[0276] val action: CommonAction,

[0277] var id: String=generatedUUID(),

[0278] var preAction:List <string>? null, )

[0280] Based on the above definition of behavior edge class, the structure of edge information is as follows Figure 7 shown.

[0281] In this embodiment, the starting node identifier of the starting node before the operation behavior, the ending node identifier of the ending node after the operation behavior, and the behavior information of the operation behavior are used as side information, thereby improving the accuracy of the operation behavior record; and the behavior information includes the behavior type of the operation behavior, and contains the node element identifier of the targeted interface control, thereby ensuring the accuracy of subsequent restoration of the interface control operation.

[0282] Path information

[0283] In some embodiments, the path information of the operation path includes at least a path identifier and a behavior edge list.

[0284] In one possible implementation, when determining the path information of the operation path corresponding to the operation process based on the edge information, the computer device determines the path identifier of the operation path corresponding to the operation process, and determines the behavior edge list based on the edge information, thereby determining the path identifier and the behavior edge list as the path information of the operation path.

[0285] Similar to the node identifier, node element identifier and behavior edge identifier, the path identifier is a unique identifier assigned to the operation path (with no actual meaning) and can be represented in the form of a string, where the path identifiers of different operation paths are different.

[0286] The behavior edge list contains a series of behavior edges corresponding to the operation behaviors, and the behavior edges are arranged in the time sequence of the operation behaviors. For example, the behavior edge at the head of the list corresponds to the start operation behavior, and the behavior edge at the tail of the list corresponds to the end operation behavior.

[0287] In some embodiments, the behavior edge list includes the behavior edge identifier in the edge information corresponding to the behavior edge.

[0288] Since the edge information of the behavior edge can indicate the nodes (interfaces) involved before and after the operation behavior and the behavior information of the executed operation behavior, the edge information of the behavior edge arranged in time sequence in the behavior edge list can accurately describe the execution order of a series of operation behaviors and what kind of operation behaviors are executed in the interface.

[0289] In an illustrative example, Figure 8 As shown, the user first starts the system setting application (com.android.setting). When the user clicks the first interface control 811 in the first interface 81 (the corresponding node element is identified as 48), the terminal jumps to display the second interface 82. When the user clicks the second interface control 821 in the second interface 82 (the corresponding node element is identified as 56), the terminal jumps to display the third interface 83. Since the third interface 83 is the target interface expected by the user, further operations are stopped. Based on this series of operations, the behavior edges contained in the generated behavior edge list are: StartApp(com.android.setting), Click(48), Click(56) and Stop(true).

[0290] In some embodiments, in order to facilitate subsequent operation path retrieval based on a series of operations to achieve functions, the path information also includes a path function. Accordingly, the computer device determines the path function of the operation path and determines the path function as the path information of the operation path.

[0291] Regarding the method of determining the path function, in one possible implementation, the computer device can generate a path function based on interface elements in the interface (such as interface text, the package name of the application to which the interface belongs) and the text content corresponding to the interface controls triggered during the operation.

[0292] For example, the application package name of the interface involved in the operation process is (com.android.setting), which is the system setting application, and the text contents corresponding to the interface controls triggered during the operation are "Display and Brightness" and "Screen Color Mode", and the path function generated by the computer device is "Set the screen color mode in the system application".

[0293] In another possible implementation, the computer device may input interface images of interfaces involved in the operation process into a large language model, and generate a path function through the large language model. This embodiment does not limit the specific method of generating the path function.

[0294] Combined with the above description of path information, in an illustrative example, the path class can be defined as:

[0295] class Path(

[0296] val id: String=generateUUID(),

[0297] val pathFunction: String? =null,

[0298] valedges:MutableList <edge>= mutableListOf(), )

[0300] Based on the above definition of path class, the structure of path information is as follows Fig. 9 shown.

[0301] In this embodiment, a behavior edge list is generated based on the behavior edges corresponding to the operation behaviors involved in the operation process, and the behavior edge list is used as the path information of the operation path, thereby ensuring the accuracy of subsequent restoration of a series of operations based on the path information; and, using the path function of the operation path as part of the path information is conducive to subsequent operation path retrieval based on the functions that need to be implemented by the operation, thereby expanding the application scenarios of the operation trajectory model.

[0302] Node Group Information

[0303] The interface of a computer device is usually represented by an activity, but in actual scenarios, there are cases where a single activity corresponds to multiple fragments, or a single activity corresponds to multiple pages. In this case, it is difficult to distinguish the two interfaces based on the name of the activity, so it is impossible to distinguish the nodes directly based on the name of the activity.

[0304] In order to solve this problem, in the embodiments of the present application, based on the concept of node, the concept of node group (NodeGroup) is extended.

[0305] In some embodiments, the node group corresponds to the activity at the code level, that is, different activities correspond to different node groups. The node group is composed of at least one node, and the interface of the node at the visual level corresponds.

[0306] Indicatively, Fig.10 As shown, during the operation, switching between three activities is involved, which correspond to node group 1010, node group 1020 and node group 1030. Node group 1010 includes three nodes corresponding to three different interfaces (all the same activity), namely node 1011, node 1012 and node 1013; node group 1020 includes node 1021 corresponding to the interface; node group 1030 includes two nodes corresponding to two different interfaces (all the same activity), namely node 1031 and node 1032.

[0307] In a possible implementation, when generating the operation trajectory model, the computer device determines node group information of the node group based on the node information, and generates the operation trajectory model based on the node group information, the edge information, and the path information.

[0308] In order to enable the node group information to have the ability to characterize each node included in the node group and distinguish different node groups, the node group information at least includes a node list and a node group factor (NodeGroupFactor).

[0309] The node list contains nodes belonging to a node group, and the node group factor is used for the node group characteristics of the node group.

[0310] In a possible implementation, when determining the node group information, the computer device determines the node group factor of the node group, and based on the node information, determines the node list (childNode) of the node group, thereby determining the node group factor and the node list as the node group information of the node group.

[0311] Optionally, the computer device groups nodes having the same activity name into the same node group.

[0312] In a possible implementation, at the code level, the node group is uniquely indicated by a combination of an application package name + an activity name, that is, the node group factor includes the application package name (packageName) and the activity name (activityName) of the application.

[0313] In another possible implementation, at the code level, the node group is uniquely indicated by a combination of application package name+activity name+activity title, that is, the node group factor includes the application package name (packageName), activity name (activityName) and activity title (activityTitle) of the application.

[0314] Optionally, since the activity titles of some activities may change dynamically, the computer device determines the application package name and activity name as the node group factors of the activity when the activity title supports dynamic change; and determines the application package name, activity name and activity title as the node group factors of the activity when the activity title does not support dynamic change.

[0315] In some embodiments, in order to facilitate subsequent node group retrieval based on the functions implemented by the activities, the computer device determines the node group function (summaryFunction) of the node group based on the interface function of the interface corresponding to the node in the node group, thereby determining the node group function as the node group information of the node group.

[0316] In a possible implementation, the computer device may summarize the interface functions corresponding to each node in the node group through a large language model, thereby obtaining the node group function.

[0317] Optionally, the node group function can be generated asynchronously, rather than in real time, such as when the computer device is idle, or when the number of node groups reaches a threshold.

[0318] Combined with the above description of node group information, in an illustrative example, the node group class can be defined as:

[0319] class NodeGroup(

[0320] val nodeGroupFactor: NodeGroupFactor,

[0321] val summaryFunction: String? =null,

[0322] val childNodes:CopyOnWriteArrayList <node>=CopyOnWriteArrayList(), )

[0324] The node group factor class can be defined as:

[0325] data class NodeGroupFactor(

[0326] var packageName:String,

[0327] var activityName: String? =null,

[0328] var activityTitle: String? =null, )

[0330] Based on the above definitions of node group class and node group factor class, the structure of node group information is as follows: Fig.11 shown.

[0331] In this embodiment, by introducing the concept of node group and using the node group factor to distinguish the node groups corresponding to different activities, the problem of multiple interfaces being difficult to distinguish under the same activity is solved; and, using the node group function as part of the node group information is conducive to subsequent function-based interface retrieval, further expanding the application scenarios of the operation trajectory model.

[0332] In combination with the above description of the operation trajectory model, the application scenario of the operation trajectory model is described.

[0333] Scenario 1: Interface Search

[0334] In response to the search instruction, the target node information is retrieved from the operation trajectory model, and the target interface corresponding to the target node to which the target node information belongs meets the interface search condition indicated by the search instruction; the interface search result is fed back based on the target node information.

[0335] Optionally, the interface search condition indicated by the search instruction may include at least one of voice, text and image. For example, the interface search condition includes voice or text describing the interface. Schematically, the interface search condition is "Which interface can set the wallpaper dynamic effect?"; for another example, the interface search condition includes a reference interface image and a voice indicating to perform interface search based on the reference interface image. Schematically, the interface search condition includes a reference interface image similar to the target interface and a voice of "Help me find an interface similar to this interface".

[0336] In a possible implementation, the computer device matches the node information corresponding to the node in the operation trajectory model with the interface search condition, thereby determining the target node that meets the interface search condition, and then provides search result feedback based on the target node information of the target node.

[0337] In a possible implementation, the computer device determines the target function of the target interface to be retrieved based on the interface retrieval condition, thereby matching the target function with the atomic function information in the node information, thereby determining the node to which the atomic function information with the highest matching degree belongs as the target node. And / or, the computer device matches the reference interface image indicated by the interface retrieval condition with the node factor in the node information, thereby determining the node to which the node factor with the highest matching degree belongs as the target node. In general, the computer device can determine the matching target node based on matching one or more information in the node information with various forms of interface retrieval conditions.

[0338] Optionally, the fed-back search result may include the application to which the target interface corresponding to the target node belongs, the level to which it belongs, etc. The search result may be fed back in the form of text, image or voice, which is not limited in this embodiment.

[0339] Scenario 2: Path positioning

[0340] In response to the positioning instruction, a target interface is determined; based on the target interface, target path information is determined from an operation trajectory model, and a target operation path to which the target path information belongs is an operation path to reach the target interface; and positioning result feedback is performed based on the target path information.

[0341] Optionally, the positioning instruction is used to indicate the target operation path for positioning to reach the target interface. The target interface can be clearly indicated in the positioning instruction, or the target interface can be determined by the computer device based on the interface search condition (refer to the interface search process of scenario 1).

[0342] In one possible implementation, after determining the target interface indicated by the positioning instruction, the computer device determines an operation trajectory model that includes a target node corresponding to the target interface, thereby taking the target node corresponding to the target node as the end point in the operation trajectory model, and determining the target operation path to reach the target node from the operation trajectory model through a breadth-first traversal algorithm.

[0343] Optionally, when the computer device performs positioning result feedback based on the target path information of the target operation path, the positioning result may include interface information of each interface on the target operation path and operation information of the operation performed in the interface. The operation information is determined based on the behavior information in the edge information corresponding to the behavior edge on the target operation path, and the interface information is determined based on the start node identifier and the end node identifier in the edge information corresponding to the behavior edge.

[0344] Based on the interface information, the user can know which intermediate interfaces need to be passed to reach the target interface, and based on the operation information, the user can know the operations that need to be performed in each intermediate interface.

[0345] Optionally, the computer device may feed back the positioning result in the form of text, image or voice, which is not limited in this embodiment.

[0346] Scenario 3: Operation playback

[0347] In response to a playback instruction, a target interface is determined; based on the target interface, target path information is determined from an operation trajectory model, and a target operation path to which the target path information belongs is an operation path to reach the target interface; based on target side information in the target path information, a target operation behavior indicated by the target side information is simulated in the current interface.

[0348] Optionally, the playback instruction is used to instruct to automatically execute a series of operations to reach the target interface. The target interface can be clearly indicated in the playback instruction, or the target interface can be determined by the computer device based on the interface search condition (refer to the interface search process of scenario 1).

[0349] In some embodiments, when a computer device receives a playback instruction, it first performs path positioning based on the playback instruction to determine the target operation path to reach the target interface (refer to the path positioning process of scenario two), and then performs simulated operations in the interface based on the target path information of the target operation path.

[0350] In a possible implementation, the computer device extracts the behavior information from the side information included in the target path information, and executes the target operation behavior indicated by the behavior information. The computer device can determine the behavior type of the target operation behavior and the interface control affected by the behavior information (in the case of affecting the interface control, it is indicated by the node element identifier).

[0351] Since the interface content may be iteratively updated (for example, the position of the interface control and the displayed content may change), when determining the interface control affected by the target operation behavior, the computer device obtains the node element factor corresponding to the node element identifier based on the node element identifier, and matches the node element factor with the node element factors corresponding to each node element in the current interface. When a unique node element is matched in the current interface, the computer device simulates the target operation behavior on the interface control corresponding to the node element.

[0352] In a possible implementation, when a unique node element is matched based on the first precision subfactor, the computer device determines the node element as a target node element to be operated in the current interface.

[0353] In the case where at least two node elements are matched based on the first precision sub-factor, the computer device further performs matching based on the second precision sub-factor. If no node element is matched based on the second precision sub-factor, or at least two node elements are matched, the computer device dynamically calculates a third precision sub-factor and performs matching based on the third precision sub-factor.

[0354] Optionally, when no node element is matched based on the first precision subfactor, the second precision subfactor, and the third precision subfactor, the computer device may input the node information of the node indicated by the edge information, the node element information of the node element indicated by the node element identifier, and the node information of the current interface into the large language model, and the large language model may select a target node element for the required operation from the current interface.

[0355] It should be noted that in addition to being applied to the above-mentioned scenarios, the operation trajectory model can also be used in other scenarios with operation trajectory usage requirements, such as use case generation and accessibility mode in automated testing processes. The embodiments of the present application do not limit the specific application scenarios.

[0356] Please refer to Fig.12 , which shows a structural block diagram of an operation recording device provided by an exemplary embodiment of the present application. The device includes:

[0357] The first determination module 1201 is used to determine node information of nodes corresponding to interfaces during operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interfaces from different dimensions;

[0358] The second determination module 1202 is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process, wherein the edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior;

[0359] A third determination module 1203 is used to determine path information of an operation path corresponding to the operation process based on the edge information, wherein the path information is used to characterize a time sequence between the operation behaviors;

[0360] The generating module 1204 is configured to generate an operation trajectory model based on the node information, the edge information and the path information.

[0361] Optionally, the first determining module 1201 is configured to:

[0362] Determine a node factor of a node corresponding to the interface during the operation, wherein the node factor includes node sub-factors that characterize interface features of the interface from different dimensions;

[0363] Determining a node identifier of the node;

[0364] The node factor and the node identifier are determined as the node information of the node.

[0365] Optionally, the first determining module 1201 is configured to:

[0366] Determine a first node sub-factor based on the window name of the window to which the interface belongs;

[0367] Determining a second node sub-factor based on the operable elements in the interface;

[0368] determining a third node sub-factor based on the interface layout of the interface;

[0369] Determining a fourth node sub-factor based on the interface screen of the interface;

[0370] Determining a fifth node sub-factor based on the interface function description information of the interface;

[0371] At least one of the first node sub-factor, the second node sub-factor, the third node sub-factor, the fourth node sub-factor, and the fifth node sub-factor is determined as the node factor of the node.

[0372] Optionally, the first determining module 1201 is configured to:

[0373] Determining the element type of the operable element in the interface;

[0374] Perform information digest processing on the character string obtained by concatenating the element types, and determine the information digest result as the second node sub-factor.

[0375] Optionally, the first determining module 1201 is configured to:

[0376] Extracting interface layout information of the interface;

[0377] The interface layout information is vectorized, and the obtained interface layout vector is determined as the third node sub-factor.

[0378] Optionally, the first determining module 1201 is configured to:

[0379] Vectorization is performed on the interface picture of the interface, and the obtained interface picture vector is determined as the fourth node sub-factor.

[0380] Optionally, the first determining module 1201 is configured to:

[0381] In the case where it is determined based on the node factor that the interface belongs to a recorded interface, determining the recorded node identifier of the recorded node corresponding to the recorded interface as the node identifier of the node corresponding to the interface;

[0382] When it is determined based on the node factor that the interface does not belong to a recorded interface, the node identifier is assigned to the node corresponding to the interface, wherein the assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

[0383] Optionally, the first determining module 1201 is configured to:

[0384] In a case where the first node sub-factor of the node is different from that of the recorded node, determining that the interface does not belong to the recorded interface;

[0385] In a case where the node and the second node sub-factor of the recorded node are different, determining that the interface does not belong to the recorded interface;

[0386] In a case where the node and the fifth node sub-factor of the recorded node are the same, determining that the interface belongs to the recorded interface;

[0387] In the case where the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is greater than a first similarity threshold, and / or the similarity between the fourth node sub-factor of the node and the recorded node is greater than a second similarity threshold, it is determined that the interface belongs to the recorded interface;

[0388] In the case where the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is less than a first similarity threshold, and the similarity between the fourth node sub-factor of the node and the recorded node is less than a second similarity threshold, it is determined that the interface does not belong to the recorded interface;

[0389] Among them, the first node sub-factor is determined based on the window name of the window to which the interface belongs, the second node sub-factor is determined based on the operable elements in the interface, the third node sub-factor is determined based on the interface layout of the interface, the fourth node sub-factor is determined based on the interface screen of the interface, and the fifth node sub-factor is determined based on the interface function description information of the interface.

[0390] Optionally, the first determining module 1201 is configured to:

[0391] Generate atomic function information of the interface through a large language model, where the atomic function information is used to indicate the atomic functions of the interface;

[0392] The atomic function information is determined as the node information of the node.

[0393] Optionally, the device further includes a fourth determining module, configured to:

[0394] Determine node element information of a node element corresponding to each interface control in the interface, wherein the node element belongs to the node, and the node element information includes information characterizing control features of the interface control from different dimensions;

[0395] The first determining module 1201 is used to:

[0396] The node element corresponding to the interface root node of the interface is determined as the node information of the node.

[0397] Optionally, the fourth determining module is used to:

[0398] Determine a node element identifier of a node element corresponding to the interface control in the interface;

[0399] Determine a node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize control features of the interface control from different dimensions;

[0400] The node element identifier and the node element factor are determined as the node element information of the node element.

[0401] Optionally, the fourth determining module is used to:

[0402] Determine the first node element sub-factor based on the package name of the control class to which the interface control belongs;

[0403] Determine a second node element sub-factor based on the class name of the control class to which the interface control belongs;

[0404] Determine a third node element sub-factor based on the resource identifier corresponding to the interface control;

[0405] Determine a fourth node element sub-factor based on a control description corresponding to the interface control;

[0406] Determine a fifth node element sub-factor based on the text content corresponding to the interface control;

[0407] At least one of the first node element sub-factor, the second node element sub-factor, the third node element sub-factor, the fourth node element sub-factor, and the fifth node element sub-factor is determined as the node element factor of the node element.

[0408] Optionally, the fourth determining module is used to:

[0409] In the case where the interface control is located in the list control, determining a sixth node element sub-factor based on the position of the interface control in the list control;

[0410] Determining a seventh node element sub-factor based on a parent control chain corresponding to the interface control, wherein the parent control chain is composed of parent controls of the interface control, and an end point of the parent control chain is a root control corresponding to the interface root node;

[0411] Determining an eighth node element sub-factor based on adjacent interface controls of the interface control;

[0412] At least one of the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor is determined as the node element factor of the node element.

[0413] Optionally, the sixth node element sub-factor, the seventh node element sub-factor and the eighth node element sub-factor are determined by lazy calculation.

[0414] Optionally, the first node element subfactor and the second node element subfactor belong to a first precision subfactor, the third node element subfactor, the fourth node element subfactor and the fifth node element subfactor belong to a second precision subfactor, and the sixth node element subfactor, the seventh node element subfactor and the eighth node element subfactor belong to a third precision subfactor;

[0415] In the process of positioning the interface control based on the node element factor, the priority of the first precision sub-factor is higher than the priority of the second precision sub-factor, and the priority of the second precision sub-factor is higher than the priority of the third precision sub-factor.

[0416] Optionally, the fourth determining module is used to:

[0417] Determining control properties of the interface control;

[0418] Determine the sub-node element identifier of the sub-node element corresponding to the sub-control of the interface control;

[0419] Determine the parent node element identifier of the parent node element corresponding to the parent control of the interface control;

[0420] At least one of the control attribute, the child node element identifier and the parent node element identifier is determined as the node element information of the node element.

[0421] Optionally, the second determining module 1202 is configured to:

[0422] Determine behavior information of the operation behavior in the interface during the operation;

[0423] Determine a starting node identifier of a starting node corresponding to a starting interface before the operation behavior, and a terminating node identifier of a terminating node corresponding to a terminating interface after the operation behavior;

[0424] The behavior information, the start node identifier, and the end node identifier are determined as the edge information of the behavior edge corresponding to the operation behavior.

[0425] Optionally, the second determining module 1202 is used to: determine the behavior type of the operation behavior as the behavior information;

[0426] In the case where the operation behavior is related to an interface control in the interface, a node element identifier of a node element corresponding to the interface control is determined as the behavior information.

[0427] Optionally, the second determining module 1202 is configured to:

[0428] In the case that the operation behavior has a preceding operation, the operation information of the preceding operation is determined as the edge information of the operation behavior corresponding to the behavior edge.

[0429] Optionally, the third determining module 1203 is used to:

[0430] Determine a path identifier of the operation process corresponding to the operation path;

[0431] Determine a behavior edge list based on the edge information, wherein the behavior edge list includes the behavior edges corresponding to the operation behaviors arranged in time sequence;

[0432] The path identifier and the behavior edge list are determined as the path information of the operation path.

[0433] Optionally, the third determining module 1203 is configured to:

[0434] determining a path function of the operation path;

[0435] The path function is determined as the path information of the operation path.

[0436] Optionally, the generating module 1204 is used to:

[0437] Based on the node information, determine node group information of a node group, wherein the node group includes at least one node;

[0438] The operation trajectory model is generated based on the node group information, the edge information, and the path information.

[0439] Optionally, the generating module 1204 is used to:

[0440] determining a node group factor of the node group, the node group factor being used for a node group feature of the node group;

[0441] Based on the node information, determining a node list of the node group, the node list including the nodes belonging to the node group;

[0442] The node group factor and the node list are determined as the node group information of the node group.

[0443] Optionally, the generating module 1204 is used to:

[0444] An application package name of an application and an activity name of an activity are determined as the node group factor of the node group, wherein nodes belonging to the same node group correspond to the same application package name and activity name.

[0445] Optionally, the generating module 1204 is used to:

[0446] An activity title of the activity is determined as the node group factor of the node group.

[0447] Optionally, the generating module 1204 is used to:

[0448] Determining a node group function of the node group based on interface functions of interfaces corresponding to nodes in the node group;

[0449] The node group function is determined as the node group information of the node group.

[0450] Optionally, the device further includes:

[0451] A retrieval module, configured to, in response to a retrieval instruction, retrieve target node information from the operation trajectory model, wherein the target interface corresponding to the target node to which the target node information belongs meets the interface retrieval condition indicated by the retrieval instruction; and provide interface retrieval result feedback based on the target node information;

[0452] A positioning module, configured to determine a target interface in response to a positioning instruction; based on the target interface, determine target path information from the operation trajectory model, wherein the target operation path to which the target path information belongs is an operation path to reach the target interface; and provide positioning result feedback based on the target path information;

[0453] A playback module is used to determine a target interface in response to a playback instruction; based on the target interface, determine target path information from the operation trajectory model, where the target operation path to which the target path information belongs is the operation path to reach the target interface; based on the target edge information in the target path information, simulate the target operation behavior indicated by the target edge information in the current interface.

[0454] In summary, the operation trajectory model is used to record the operation process in the embodiment of the present application. Among them, the nodes in the operation trajectory model correspond to the interface during the operation process, the edges correspond to the operation behaviors in the interface, and the operation path characterizes the execution timing of the operation behaviors during the operation process. When recording the operation process, the interface during the operation process is recorded using node information that characterizes the interface characteristics from different dimensions, thereby ensuring the accuracy of the interface record and facilitating the subsequent interface positioning; the operation behaviors in the interface and the interface jumps before and after the operation are recorded using edge information, and the timing between the operation behaviors is recorded using path information, thereby ensuring the accuracy of the operation process record and facilitating the subsequent operation reproduction. Since the operation recording process does not require manual triggering by the user, the efficiency of the operation recording is relatively high, and compared to screen recording, the information in the operation trajectory model can be used to backtrack and reproduce, reducing the difficulty of reproducing historical operations.

[0455] See also Fig.13 , Fig.13 13 is a schematic diagram of a computer device provided by an exemplary embodiment of the present application. The computer device may also include one or more of the following components: a processor 1310 and a memory 1320 .

[0456] Optionally, the processor 1310 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1320, and calling data stored in the memory 1320. Optionally, the processor 1310 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1310 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above baseband chip may not be integrated into the processor 1310, but may be implemented by a separate chip.

[0457] The memory 1320 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1320 includes a non-transitory computer-readable storage medium. The memory 1320 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0458] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device, and the computer device may include more (such as communication components, display components) or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0459] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the operation recording method described in the above embodiment.

[0460] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the operation recording method described in the above embodiment.

[0461] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0462] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / node> < / edge> < / string> < / nodeelementfactor> < / nodeelement> < / string> < / string>

Claims

1. An operation recording method, characterized in that: The method comprises: Determine node information of nodes corresponding to the interface during the operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interface from different dimensions; Determine edge information of a behavior edge corresponding to the operation behavior in the interface during the operation process, wherein the edge information includes information for characterizing the operation behavior and node jump conditions before and after the operation behavior; Based on the edge information, determining path information of an operation path corresponding to the operation process, wherein the path information is used to characterize a time sequence between the operation behaviors; An operation trajectory model is generated based on the node information, the edge information, and the path information.

2. The method according to claim 1, characterized in that The node information of the node corresponding to the interface during the determination operation includes: Determine a node factor of a node corresponding to the interface during the operation, wherein the node factor includes node sub-factors that characterize interface features of the interface from different dimensions; Determining a node identifier of the node; The node factor and the node identifier are determined as the node information of the node.

3. The method according to claim 2, characterized in that The determining of the node factor of the node comprises: Determine a first node sub-factor based on the window name of the window to which the interface belongs; Determining a second node sub-factor based on the operable elements in the interface; determining a third node sub-factor based on the interface layout of the interface; Determining a fourth node sub-factor based on the interface screen of the interface; Determining a fifth node sub-factor based on the interface function description information of the interface; At least one of the first node sub-factor, the second node sub-factor, the third node sub-factor, the fourth node sub-factor, and the fifth node sub-factor is determined as the node factor of the node.

4. The method according to claim 3, characterized in that The determining of the second node sub-factor based on the operable elements in the interface includes: Determining the element type of the operable element in the interface; Perform information digest processing on the character string obtained by concatenating the element types, and determine the information digest result as the second node sub-factor.

5. The method according to claim 3, characterized in that: The determining of the third node sub-factor based on the interface layout of the interface includes: Extracting interface layout information of the interface; The interface layout information is vectorized, and the obtained interface layout vector is determined as the third node sub-factor.

6. The method according to claim 3, characterized in that The determining of the fourth node sub-factor based on the interface screen of the interface includes: Vectorization is performed on the interface picture of the interface, and the obtained interface picture vector is determined as the fourth node sub-factor.

7. The method according to claim 2, characterized in that The determining the node identifier of the node includes: In the case where it is determined based on the node factor that the interface belongs to a recorded interface, determining the recorded node identifier of the recorded node corresponding to the recorded interface as the node identifier of the node corresponding to the interface; When it is determined based on the node factor that the interface does not belong to a recorded interface, the node identifier is assigned to the node corresponding to the interface, wherein the assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

8. The method according to claim 7, characterized in that The node factor comprises a first node sub-factor, a second node sub-factor, a third node sub-factor, a fourth node sub-factor and a fifth node sub-factor; The method comprises: In a case where the first node sub-factor of the node is different from that of the recorded node, determining that the interface does not belong to the recorded interface; In a case where the node and the second node sub-factor of the recorded node are different, determining that the interface does not belong to the recorded interface; In a case where the node and the fifth node sub-factor of the recorded node are the same, determining that the interface belongs to the recorded interface; In the case where the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is greater than a first similarity threshold, and / or the similarity between the fourth node sub-factor of the node and the recorded node is greater than a second similarity threshold, it is determined that the interface belongs to the recorded interface; In the case where the first node sub-factor of the node is the same as that of the recorded node, and the second node sub-factor of the node is the same as that of the recorded node, and the fifth node sub-factor of the node is different from that of the recorded node, and the similarity between the third node sub-factor of the node and the recorded node is less than a first similarity threshold, and the similarity between the fourth node sub-factor of the node and the recorded node is less than a second similarity threshold, it is determined that the interface does not belong to the recorded interface; Among them, the first node sub-factor is determined based on the window name of the window to which the interface belongs, the second node sub-factor is determined based on the operable elements in the interface, the third node sub-factor is determined based on the interface layout of the interface, the fourth node sub-factor is determined based on the interface screen of the interface, and the fifth node sub-factor is determined based on the interface function description information of the interface.

9. The method according to claim 2, characterized in that: The node information of the node corresponding to the interface during the determination operation also includes: Generate atomic function information of the interface through a large language model, where the atomic function information is used to indicate the atomic functions of the interface; The atomic function information is determined as the node information of the node.

10. The method according to claim 2, characterized in that The method further comprises: Determine node element information of a node element corresponding to each interface control in the interface, wherein the node element belongs to the node, and the node element information includes information characterizing control features of the interface control from different dimensions; The node information of the node corresponding to the interface during the determination operation also includes: The node element corresponding to the interface root node of the interface is determined as the node information of the node.

11. The method according to claim 10, characterized in that The determining of the node element information of the node element corresponding to each interface control in the interface includes: Determine a node element identifier of a node element corresponding to the interface control in the interface; Determine a node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize control features of the interface control from different dimensions; The node element identifier and the node element factor are determined as the node element information of the node element.

12. The method according to claim 11, characterized in that The determining of the node element factor of the node element comprises: Determine the first node element sub-factor based on the package name of the control class to which the interface control belongs; Determine a second node element sub-factor based on the class name of the control class to which the interface control belongs; Determine a third node element sub-factor based on the resource identifier corresponding to the interface control; Determine a fourth node element sub-factor based on a control description corresponding to the interface control; Determine a fifth node element sub-factor based on the text content corresponding to the interface control; At least one of the first node element sub-factor, the second node element sub-factor, the third node element sub-factor, the fourth node element sub-factor, and the fifth node element sub-factor is determined as the node element factor of the node element.

13. The method according to claim 12, characterized in that The determining of the node element factor of the node element further includes: In the case where the interface control is located in the list control, determining a sixth node element sub-factor based on the position of the interface control in the list control; Determining a seventh node element sub-factor based on a parent control chain corresponding to the interface control, wherein the parent control chain is composed of parent controls of the interface control, and an end point of the parent control chain is a root control corresponding to the interface root node; Determining an eighth node element sub-factor based on adjacent interface controls of the interface control; At least one of the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor is determined as the node element factor of the node element.

14. The method according to claim 13, characterized in that The sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor are determined by lazy calculation.

15. The method according to claim 13, characterized in that The first node element subfactor and the second node element subfactor belong to the first precision subfactor, the third node element subfactor, the fourth node element subfactor and the fifth node element subfactor belong to the second precision subfactor, and the sixth node element subfactor, the seventh node element subfactor and the eighth node element subfactor belong to the third precision subfactor; In the process of positioning the interface control based on the node element factor, the priority of the first precision sub-factor is higher than the priority of the second precision sub-factor, and the priority of the second precision sub-factor is higher than the priority of the third precision sub-factor.

16. The method according to claim 11, characterized in that The determining of the node element information of the node element corresponding to each interface control in the interface also includes: Determining control properties of the interface control; Determine the sub-node element identifier of the sub-node element corresponding to the sub-control of the interface control; Determine the parent node element identifier of the parent node element corresponding to the parent control of the interface control; At least one of the control attribute, the child node element identifier and the parent node element identifier is determined as the node element information of the node element.

17. The method according to claim 1, characterized in that The edge information of the behavior edge corresponding to the operation behavior in the interface during the determination operation includes: Determine behavior information of the operation behavior in the interface during the operation; Determine a starting node identifier of a starting node corresponding to a starting interface before the operation behavior, and a terminating node identifier of a terminating node corresponding to a terminating interface after the operation behavior; The behavior information, the start node identifier, and the end node identifier are determined as the edge information of the behavior edge corresponding to the operation behavior.

18. The method according to claim 17, characterized in that The behavior information of the operation behavior in the interface during the determination operation includes: determining a behavior type of the operation behavior as the behavior information; In the case where the operation behavior is related to an interface control in the interface, a node element identifier of a node element corresponding to the interface control is determined as the behavior information.

19. The method according to claim 17, characterized in that The edge information of the behavior edge corresponding to the operation behavior in the interface during the determination operation process also includes: In the case that the operation behavior has a preceding operation, the operation information of the preceding operation is determined as the edge information of the operation behavior corresponding to the behavior edge.

20. The method according to claim 1, characterized in that The determining, based on the edge information, path information of an operation path corresponding to the operation process includes: Determine a path identifier of the operation process corresponding to the operation path; Determine a behavior edge list based on the edge information, wherein the behavior edge list includes the behavior edges corresponding to the operation behaviors arranged in time sequence; The path identifier and the behavior edge list are determined as the path information of the operation path.

21. The method according to claim 20, characterized in that The determining, based on the edge information, path information of the operation path corresponding to the operation process further includes: determining a path function of the operation path; The path function is determined as the path information of the operation path.

22. The method according to claim 1, characterized in that The generating the operation trajectory model based on the node information, the edge information and the path information includes: Based on the node information, determine node group information of a node group, where the node group includes at least one node; The operation trajectory model is generated based on the node group information, the edge information, and the path information.

23. The method according to claim 22, characterized in that The determining the node group information of the node group based on the node information includes: determining a node group factor of the node group, the node group factor being used for a node group feature of the node group; Based on the node information, determining a node list of the node group, the node list including the nodes belonging to the node group; The node group factor and the node list are determined as the node group information of the node group.

24. The method according to claim 23, characterized in that The determining of the node group factor of the node group includes: An application package name of an application and an activity name of an activity are determined as the node group factor of the node group, wherein nodes belonging to the same node group correspond to the same application package name and activity name.

25. The method according to claim 24, characterized in that The determining of the node group factor of the node group further includes: An activity title of the activity is determined as the node group factor of the node group.

26. The method according to claim 23, characterized in that The determining the node group information of the node group based on the node information includes: Determining a node group function of the node group based on interface functions of interfaces corresponding to nodes in the node group; The node group function is determined as the node group information of the node group.

27. The method according to claim 1, characterized in that The method further comprises at least one of the following: In response to the search instruction, the target node information is retrieved from the operation trajectory model, and the target interface corresponding to the target node to which the target node information belongs meets the interface search condition indicated by the search instruction; Feedback of interface search results based on the target node information; In response to the positioning instruction, determining a target interface; Based on the target interface, determining target path information from the operation trajectory model, wherein the target operation path to which the target path information belongs is an operation path to reach the target interface; Providing positioning result feedback based on the target path information; In response to the playback instruction, determining a target interface; Based on the target interface, determining target path information from the operation trajectory model, wherein the target operation path to which the target path information belongs is an operation path to reach the target interface; Based on the target side information in the target path information, a target operation behavior indicated by the target side information is simulated in the current interface.

28. An operation recording device, characterized in that: The device comprises: A first determination module is used to determine node information of nodes corresponding to interfaces during operation, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interfaces from different dimensions; A second determination module is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process, wherein the edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior; A third determination module, configured to determine path information of an operation path corresponding to the operation process based on the edge information, wherein the path information is used to characterize a time sequence between the operation behaviors; A generation module is used to generate an operation trajectory model based on the node information, the edge information and the path information.

29. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the operation recording method according to any one of claims 1 to 27.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the operation recording method according to any one of claims 1 to 27.

31. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the operation recording method as described in any one of claims 1 to 27.

Images