Operation track access method, computer equipment and computer program product

Abstract

The embodiment of the invention discloses an operation track access method, computer equipment and a computer program product, and belongs to the technical field of man-machine interaction. The method comprises the following steps: reading a basic file of the operation track; the operation track is used for recording an operation behavior in at least one interface and an interface jump condition before and after the operation behavior; the basic file is used for indicating at least one object instance contained in the operation track; reading at least one feature file of the operation track; the feature file is used for storing feature information of the at least one object instance in different dimensions; and deserializing the basic file and the at least one feature file into the at least one object instance of the operation track. According to the method, the data related to the operation track is classified and stored based on the object instance in the operation track, and the access efficiency of the operation track is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction technology, and more particularly to a method for storing and accessing an operation trajectory, a computer device, and a computer program product. Background Art

[0002] When a user uses a terminal, the operation track of the operation behavior performed in the application can be recorded; based on the recorded operation track, functions such as searching historical browsing information, reproducing historical operations, and debugging the application can be realized.

[0003] In the related art, the operation track access system stores the recorded operation track in a file. The file stores the interface and operation behavior in the operation track, for example, the file stores interface information such as interface control information, interface text information, interface function information, and interface screenshots contained in the interface, and the file also stores operation behavior information such as the operation behavior type, the location where the operation behavior is executed, the time when the operation behavior is executed, and the interface control triggered by the operation behavior.

[0004] In the methods of the related art, the storage file data of the operation track is huge and the information is mixed, which is not convenient for the management of the operation track (for example, data reading, information search, content modification, etc.). Summary of the invention

[0005] The embodiment of the present application provides a method for accessing an operation track, a computer device, and a computer program product. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a method for accessing an operation track, the method comprising:

[0007] Reading a basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track;

[0008] Reading at least one feature file of the operation trajectory; the feature file is used to store feature information of the at least one object instance in different dimensions;

[0009] The basic file and the at least one feature file are deserialized into the at least one object instance of the operation track.

[0010] On the other hand, an embodiment of the present application provides an operation track access device, the device comprising:

[0011] A file module, used to read a basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track;

[0012] The file module is used to read at least one feature file of the operation trajectory; the feature file is used to store feature information of the at least one object instance in different dimensions;

[0013] The object module is used to deserialize the basic file and the at least one feature file into the at least one object instance of the operation track.

[0014] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the method for accessing the operation trajectory as described in the above aspects.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for accessing the operation trajectory as described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for accessing the operation trajectory as described in the above aspects.

[0017] The embodiment of the present application stores the operation traces in categories, uses the basic file to store the object instances contained in the operation traces, and uses the feature file to store the feature information of the object instances in different dimensions, thereby reducing the data volume of a single file and improving the file reading speed. In addition, the feature information of the object instance can be stored in different feature files in the storage format that best suits the feature information, ensuring the storage integrity of the feature information and facilitating the deserialization and restoration of the object instance. Using this method to access the operation trace, the basic file simply and clearly records the various object instances contained in the operation trace, and the feature file stores the feature information of each object instance in different dimensions in detail, making the storage structure of the operation trace clearer; different types of operation traces are stored in different files, respectively, so that the data volume of a single file is smaller, the reading speed is faster, the execution efficiency of the deserialization process is higher, and the reading of the operation trace is faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an operation trajectory model shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application;

[0020] Figure 3 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram showing a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0022] Figure 5 A schematic diagram of a skeleton diagram provided by an exemplary embodiment of the present application is shown;

[0023] Figure 6 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0024] Figure 7 is a preview diagram of an operation track shown in an exemplary embodiment of the present application;

[0025] Figure 8 is a preview diagram of an operation track shown in an exemplary embodiment of the present application;

[0026] Fig. 9 A schematic diagram of an access system for an operation track provided by an exemplary embodiment of the present application is shown;

[0027] Fig.10 A schematic diagram of an access system for an operation track provided by an exemplary embodiment of the present application is shown;

[0028] Fig.11 A schematic diagram of an access system for an operation track provided by an exemplary embodiment of the present application is shown;

[0029] Fig.12 is a preview diagram of an operation track shown in an exemplary embodiment of the present application;

[0030] Fig.13 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0031] Fig.14 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0032] Fig.15 A schematic diagram of a compression interface tree provided by an exemplary embodiment of the present application is shown;

[0033] Fig.16 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0034] Fig.17 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0035] Fig.18 A flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application is shown;

[0036] Fig.19 A schematic diagram of flashback provided by an exemplary embodiment of the present application is shown;

[0037] Fig. 20 A structural block diagram of an access device for operation traces provided by an exemplary embodiment of the present application is shown;

[0038] Fig.21 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0040] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0041] First, the nouns involved in the embodiments are briefly described.

[0042] 1. Operation trajectory.

[0043] The operation track is a path record of at least one operation behavior of the user in the application, including the operation behavior performed by the user in the application and the interface jump situation before and after the operation behavior is performed.

[0044] 2. Object instance.

[0045] Object instances are used to form operation traces. In order to facilitate reading and calling of key parts in the operation trace, the key parts in the operation trace are created as object instances. For example, an object instance includes at least one of the following: a node, a node group, a behavior edge, an operation path, a factor, and an operation behavior type. In an optional embodiment, an object instance includes at least a node and a behavior edge. Each object instance is described below.

[0046] 1) Node, which is used to represent the interface in the operation behavior. One interface is created as a node. The interface is a visual user interface, such as a desktop, application interface, system interface, etc. Different interfaces in the operation track correspond to their own nodes, and different nodes represent the interface features of their corresponding interfaces through their own node information, that is, different nodes can be distinguished based on node information.

[0047] 2) A node group, including at least one node. Optionally, each node corresponds to a node group, or each node corresponds to at least one node group. In order to facilitate the management of multiple nodes, at least two nodes with an associated relationship can be divided into the same node group as required. Alternatively, at least one node of the same type is divided into the same node group. For example, at least one interface (node) belonging to the same activity is divided into a node group. Alternatively, at least one interface (node) belonging to the same application is divided into a node group.

[0048] 3) Node elements, which are used to identify key information in the interface. Exemplarily, for key information in the interface, a separate object instance can be created for storage. For example, interface controls, pictures, links, text boxes, charts, and videos in the interface can all be identified as independent node elements for storage. The embodiment of the present application only takes the example of node elements including interface controls in the interface, and creates a separate node element object instance for each interface control in the interface.

[0049] 4) Behavior edge, which is used to characterize the operation behavior in the operation trajectory. The behavior edge includes the node jump situation before and after the operation behavior, that is, the behavior edge can be used to indicate: the operation behavior, the starting node before the operation behavior, and the ending node after the operation behavior. The behavior edge corresponding to the operation behavior is a directed edge, and the direction of the behavior edge is the interface jump direction, that is, the behavior edge points from the interface before the jump (that is, the starting node) to the interface after the jump (that is, the ending node). Among them, when the starting node and the ending node are the same node, the behavior edge includes a directed edge from the node pointing back to the node.

[0050] 5) Operation path (Path), which is used to indicate the timing between at least two operation behaviors. Since the operation trajectory may require the execution of a series of operation behaviors and the jumping between multiple interfaces, the computer device needs to sort the behavior edges corresponding to the operation behaviors according to the execution timing between the operation behaviors to obtain the operation path. Among them, the operation path is a directed path composed of multiple nodes and behavior edges between nodes.

[0051] 6) Factors, including features used to identify nodes or node elements. Node factors can be used to uniquely identify a node, and node element factors can be used to uniquely identify a node element.

[0052] 7) Operation behavior type, including multiple predefined operation behavior types, for example, the operation behavior type may include at least one of the following: click, double-click, slide, scroll, drag, shortcut key, etc. The operation behavior type of the behavior edge may be indicated from the predefined operation behavior types.

[0053] 3. Basic documents.

[0054] The basic file is used to store the basic information of the object instances included in the operation trace. Based on the basic file, the list of object instances included in the operation trace and the basic information of each object instance can be clearly defined. Basic information refers to information with a data volume less than a threshold. For example, basic information can include simple information such as identification, name, and summary.

[0055] Optionally, one operation track corresponds to one basic file, or at least one operation track in an application corresponds to one basic file, or at least one operation track of a user corresponds to one basic file.

[0056] 4. Feature file.

[0057] The feature file is used to store the feature information of the object instance in different dimensions. The feature information of an object instance in one dimension is stored as a feature file. For example, the feature file may include at least one of the following: a root node file of a node (used to store the complete information of each interface control in the interface corresponding to the node), a screenshot file of a node (used to store the screenshot of the interface corresponding to the node), a view tree file of a node (used to store the summary information and association relationship of each interface control in the interface corresponding to the node), a function file of a node or a function file of a node element (used to store the function description text of the node or node group), a skeleton diagram file of a node (used to store the skeleton diagram of the interface corresponding to the node).

[0058] The feature file may be stored in different file formats according to the data characteristics of the stored feature information, for example, storing images in picture format and text in text format.

[0059] 5. Skeleton diagram.

[0060] A skeleton diagram is a visual framework used to illustrate the distribution of interface elements (e.g., text, images, interface controls, etc.) and the overall structure of the page. The skeleton diagram uses lines, rectangles, or other basic graphics to depict the layout of the page before data or content is loaded in a concise and clear way. The skeleton diagram does not show specific content or details, but focuses on presenting the arrangement of interface elements, their size ratios, and their relative positional relationships.

[0061] 6. Serialization.

[0062] Serialization is the process of converting the information of an object instance into a form that can be stored or transmitted. During serialization, an object instance writes its current state to a temporary or persistent storage area. Later, the object instance can be recreated by reading or deserializing the state of the object instance from the storage area. Serialization is used to save the state of an object instance so that the object instance can be recreated when needed.

[0063] Serialization is used to convert the current state of an object instance (eg, the current state includes at least one of basic information and feature information) into a storage file in a storable form (eg, the storage file includes at least one of a basic file and a feature file).

[0064] An object instance is a specific object created according to the class definition when the program is running. An object instance has the properties and methods defined by the class, and can have its own state (i.e., attribute value). Serialization refers to converting the state of an object instance in a running program into a form that can be stored or transmitted (such as a byte stream) so that the same object instance can be reconstructed later when needed.

[0065] 7. Deserialization.

[0066] Deserialization is the reverse process of serialization. Deserialization refers to the process of decoding and restoring serialized data to the original object instance. Deserialization is used to create an object instance before serialization based on the base file and feature file. The object instance obtained after deserialization has the same state as the object instance before serialization.

[0067] In the process of daily use of applications, users can record a certain operation interface or a series of interface operation tracks. Later, when they need to perform a certain operation again, they can review the operation track and manually reproduce the operation.

[0068] In the embodiment of the present application, an operation trajectory model (Trajectory Structure) is used to record the operation trajectory. Figure 1 As shown, the operation trajectory model 13 corresponding to the operation trajectory is composed of at least one object instance. At least one object instance may include: a node 131, a behavior edge 132, and an operation path 133. Among them, the node 131 corresponds to the interface passed in the operation trajectory; the behavior edge 132 corresponds to the operation behavior in the interface, which is used to connect different nodes (interface jump occurs after operation) or connect the same node (no interface jump occurs after operation); the operation path 133 corresponds to the operation behavior sequence.

[0069] In addition, in order to ensure the accuracy of the operation trajectory record, the node information 141 of the node 131 represents the interface features of the interface from different dimensions (for example, interface screenshots, interface function description text, interface control information, skeleton diagrams, interface logos, interface node factors, etc.) to ensure the accuracy of subsequent interface retrieval and positioning; the edge information 142 of the behavior edge 132 indicates the operation behavior and the node jumps before and after the operation behavior to ensure the accurate restoration of the operation in the subsequent interface; the path information 143 of the operation path 133 represents the execution sequence between different operation behaviors to ensure that complex operations are executed in sequence.

[0070] The operation trajectory model has the characteristics of low data volume and can be automatically generated without manual triggering by the user. In addition, it can automatically backtrack and reproduce based on node information, edge information, and path information, which improves the efficiency of operation backtracking and reproduction.

[0071] like Figure 3 As shown, the operation track access method provided in the embodiment of the present application can be executed by a computer device 110 with an interface display function. The computer device 110 can be a mobile phone, a tablet computer, a notebook computer, a laptop computer, a desktop computer, an all-in-one computer, an Internet of Things device, an intelligent robot workstation, a TV, a set-top box, smart glasses, a smart watch, a digital camera, an MP4 player, an MP5 player, a learning machine, a point-reading machine, an electronic paper book, an electronic dictionary, a vehicle-mounted device, a virtual reality (VR) player or an augmented reality (AR) player, etc.

[0072] The computer device 110 includes a memory and a processor; the memory may include one or more computer-readable storage media. The computer-readable storage media include at least one of a random access memory (RAM), a read-only memory (ROM), and a flash memory.

[0073] The computer device 110 installs and runs an access system with operation traces, and the access system is used to access the operation traces. In some embodiments, a client of the access system can be run on the computer device 110, and the access system can also include a server running on the server 140. The server 140 can be any one of multiple servers, virtual cloud storage or cloud computing center. The server 140 is used to provide background services for the computer device 110.

[0074] In some embodiments, the computer device 110 and the server 140 are connected via a wired or wireless network. In the following embodiments, the execution subject of the method for accessing the operation track is taken as an example to illustrate.

[0075] Please refer to Figure 3 , which shows a flow chart of a method for accessing an operation track provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device as an example, and the method may include the following steps:

[0076] Step 210, read the basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track.

[0077] The operation track is used to record a series of operation behaviors performed by the user during the use of the application and the corresponding interface jumps. The operation track can be used to reproduce user operations, search for historical operations or historical browsing information, and debug the application. For example, the operation track stores relevant information such as the interface, operation behavior, and interface jump relationship.

[0078] In order to facilitate the sorting of the various objects involved in the operation track (for example, the interface, the controls in the interface, the interface group composed of multiple associated interfaces, a single operation behavior, the operation track composed of multiple operation behaviors, etc.), and improve the access efficiency of the operation track, the embodiment of the present application classifies and stores the various objects involved in the operation track (that is, object instances), stores the basic information of the operation track as a basic file, and stores the characteristic information of each object as a characteristic file. Based on the basic file, the objects contained in the operation track and the relationship between each object (for example, the jump relationship of the interface, the operation behavior and its corresponding interface jump relationship, the timing of the operation behavior, etc.) can be clarified, and the operation track can be simply restored. Based on the characteristic file, the detailed information of each object involved in the operation track (for example, the interface screenshot, the controls contained in the interface, the functional description of the interface, the skeleton diagram of the interface, etc.) can be supplemented to completely restore the data contained in the operation track. Using this method to store the operation track, the various dimensional information contained in the operation track can be simply and directly clarified from the data storage structure, which is convenient for data storage and data maintenance of the operation track.

[0079] Object instances are created to classify and encapsulate the data of operation tracks so that they can be repeatedly called, maintained and modified. According to the storage and application requirements of the operation tracks, the data in the operation tracks can be encapsulated according to different classification methods to create various types of object instances. For example, an interface in the operation track is created as an object instance (node), an operation behavior is created as an object instance (behavior edge), at least one interface is created as an object instance (node ​​group), and so on. The embodiments of the present application only use this classification method as an example. In other scenarios, new object instances can also be created according to scenario requirements. For example, when it is necessary to analyze the operation tracks of different applications, an application can also be created as an object instance. When it is necessary to focus on displaying the interface jump animation between interfaces, each interface jump animation can also be created as an object instance.

[0080] It should be noted that different interface screenshots recorded in the operation trace may actually belong to the same interface. Figure 4 As shown, when a user swipes the setting interface, multiple interface screenshots showing different setting information are obtained (for example, Figure 4In the example of the interface screenshot 101 and the interface screenshot 102 in the operation track, the method in the related art will store all these interface screenshots and identify each interface screenshot showing different information as an independent interface, which not only leads to a large amount of redundant data in the storage data of the operation track, but also the interface jump relationship is chaotic in the preview image rendered based on this operation track data. For example, the interface screenshot 101 and the interface screenshot 102 belong to the same setting interface, but because the method in the related art identifies them as different interfaces, the interface screenshot 101 and the interface screenshot 102 will be displayed as a node in the preview image of the operation track, resulting in a large number of nodes in the preview image, and the jump relationship between the nodes is chaotic.

[0081] For example, in order to make the storage and preview of the operation track more clearly show the actual jump logic of the operation track and accurately illustrate the jump relationship between interfaces, the method provided in the embodiment of the present application does not simply identify the interface screenshots with different display contents as different interfaces, but calculates the similarity based on the feature information contained in each interface screenshot, identifies the interface screenshots with higher similarity as belonging to the same interface, and stores each interface of the application as an object instance. The preview generated based on this storage method can simply and clearly show the jump logic of each interface in the application.

[0082] In an optional embodiment, the basic file is used to store the list of object instances included in the operation track, and the information that the data volume corresponding to each object instance is less than a threshold; the feature file is used to store the information that the data volume corresponding to each object instance is greater than the threshold. And / or, the basic file is used to store the information of the first type of format corresponding to the object instance, and the first type of format may include text; the feature file is used to store the information of the second type of format corresponding to the object instance, and the second type of format may include an image.

[0083] The feature information of an object instance in one dimension is stored as a feature file. The classification dimension of the feature file can also be set arbitrarily. For example, the classification dimension can include at least one of the following interface screenshots, skeleton diagrams, control information, and function description texts, and the feature information with larger amounts of data is stored as a feature file. Of course, other classification dimensions can also be used to store other feature information of the object instance separately as a feature file.

[0084] In summary, the computer device stores the operation track as a basic file and a feature file. The basic file lists the object instances contained in the operation track and the simplified information of the object instances, and the simplified information may be information whose data volume is less than a threshold. The feature file stores the feature information of the object instance in at least one dimension, and the feature information may include information whose data volume is greater than a threshold. Storing these feature information separately as feature files can reduce the data volume of each file, speed up the reading speed of a single file, and improve the access efficiency of the operation track.

[0085] Optionally, feature information of different dimensions is stored in different feature files, and the feature information can be stored in a format that is most suitable for the feature information, for example, using picture format to store interface screenshots and skeleton diagrams, and using json format to store interface control information.

[0086] Exemplarily, the operation track can be used to record a user's operation, or to record the interface jump relationship of the application. For example, the operation track can be obtained by recording a user's operation process. For example, the user records the operation process of playing music and obtains the operation track of playing music. The operation track can also be obtained by recording multiple operation processes of the user. Alternatively, the entire operation process of the user using the first application is recorded to obtain the operation track of the user using the first application. For example, the user records the entire process of applying the first application to obtain the jump relationship between the various interfaces of the first application.

[0087] The operation trace can also be obtained by recording the operation process of one user, or by recording the operation process of multiple users. For example, when the operation trace is used to trace back user operations, or to search for user historical operations, the computer device can record the operation process of the user to obtain the operation trace of the user. For another example, when the operation trace is used to debug an application, the developer of the application can, after obtaining user authorization, record the operation process of multiple users to obtain the operation trace of the user using the application, and obtain the interface jump relationship of the application based on the operation trace.

[0088] In an optional embodiment, the operation trace of an application can be stored in a set of files, which includes a basic file and at least one feature file. The basic file can store all interfaces of the application (each interface is a node) and the jump relationship between each interface. The feature file can store detailed information on each interface in at least one feature dimension. Through this storage method, the jump relationship of all interfaces of an application can be shown.

[0089] In another optional embodiment, a user's operation track can be stored in a set of files, which includes a basic file and at least one feature file. The basic file can store the interface of at least one application used by the user (each interface is a node), as well as the jump relationship between each interface or between each application. The feature file can store detailed information on each interface in at least one feature dimension. Through this storage method, the jump relationship between multiple applications used by the user can be analyzed.

[0090] In another optional embodiment, the operation trace of a user using an application can be stored in a set of files, which includes a basic file and a feature file. The basic file can store the interface of the application used by the user (each interface is a node), as well as the jump relationship between the interfaces used by the user. The feature file can store detailed information on each interface in at least one feature dimension. This storage method facilitates the analysis and retrieval of the user's historical operations using an application.

[0091] The basic file (OUTS.json) combines information such as nodes, node groups, node elements, behavior edges, operation paths, and factors, which is equivalent to merging the basic information part of OUTS into the same file with the most concise fields. Exemplarily, the basic file can store concise information of at least one of the following object instances: nodes, node groups, node elements, behavior edges, operation paths, and factors. The above object instances are described below.

[0092] 1) Node corresponding to the interface.

[0093] The interface is a visual user interface, such as a desktop, an application program interface, a system interface, etc. In the embodiment of the present application, different interfaces in the operation track correspond to respective nodes, and different nodes represent the interface features of their respective corresponding interfaces through their respective node information, that is, different nodes can be distinguished based on the node information.

[0094] Optionally, the dimensions that characterize the interface features may include interface content dimensions, interface layout dimensions, interface function dimensions, interface description dimensions, interface-to-window dimensions, and the like. Interface content is information captured from the interface (for example, information displayed in the interface such as text, pictures, videos, audio, links, tables, or interface background information that is not displayed in the interface). The interface layout is used to characterize the location distribution of each content in the interface, and the interface layout can be represented by a skeleton diagram. Interface functions are used to indicate the functions supported or provided by the interface. The interface description is used to describe the interface content and / or interface functions. The interface-to-window is used to indicate the parent interface to which the interface belongs.

[0095] The interface in this embodiment is not limited to the visible range of the screen, but includes the complete interface within and outside the visible range of the screen. For example, the display screen of the computer device cannot fully display the first interface, and the user can view different parts of the first interface by dragging. In this embodiment, the interface refers to the complete first interface, rather than a small area of ​​the first interface displayed on the display screen. Accordingly, before and after the interface content is scrolled, even if the display content within the visible range of the screen changes significantly, the interfaces before and after the scrolling will not be judged as different interfaces.

[0096] In the embodiments of the present application, different interfaces refer to interfaces with significant differences in interface layout, content, and style. Usually, when an interface jump occurs, the interfaces before and after the jump are determined to be different interfaces. For example, when text or images in an interface change dynamically, or when list elements in an interface change, they are not determined to be different interfaces; when clicking on a control in an interface to trigger an interface jump, the interfaces before and after the jump are determined to be different interfaces.

[0097] The nodes stored in the basic file may include at least one of the following node information: node identification, interface name, interface type, affiliated application identification, interface summary, interface level, whether there is an upper-level interface, the node identification of the upper-level interface, whether there is a lower-level interface, the node identification of the lower-level interface, the identification of the interface controls (i.e., node elements) contained in the interface, etc.

[0098] Among them, the node identifier is a unique identifier assigned to the node (it has no actual meaning) and can be represented in the form of a string. Different nodes have different node identifiers, and the same node has the same node identifier. The interface name can be directly obtained from the captured interface information, or it can be generated by the access system based on the interface content. The interface summary can be a text that briefly describes the interface content, or it can be a text that briefly describes the interface function. The interface level is used to indicate the hierarchical relationship of the interface. For example, the first interface is a first-level interface. If the second interface is opened through the first interface, the second interface is the second-level interface of the first interface.

[0099] Exemplarily, information with a relatively small amount of data related to the interface can be stored in the basic file, and information with a relatively large amount of data related to the interface can be stored in the feature file. For example, information with a relatively large amount of data, such as screenshots of the interface, skeleton diagrams of the interface, information of each control in the interface, pictures in the interface, and videos in the interface, can be stored in the feature file.

[0100] 2) A node group consisting of at least one node.

[0101] Optionally, the node group includes at least one node. Optionally, each node corresponds to a node group, or each node corresponds to at least one node group.

[0102] For example, in order to facilitate the management of multiple nodes, at least one node with an associated relationship can be divided into a node group as required. For example, at least one interface (node) belonging to the same activity is divided into a node group. Or, at least one interface (node) belonging to the same application is divided into a node group.

[0103] The node group stored in the basic file may include at least one of the following node group information: node group identifier, node identifier of at least one node belonging to the node group, node group name, node group type, application identifier to which it belongs, functional description of the node group, association relationship with other node groups, etc.

[0104] 3) The node contains node elements, and the node elements include interface controls.

[0105] Exemplarily, the key information in the interface can be separately identified as an object instance for storage. For example, interface controls, pictures, links, etc. in the interface can be identified as independent node elements for storage. The embodiment of the present application is only described by taking the example that the node elements include interface controls.

[0106] The node elements stored in the basic file may include at least one of the following node element information: node element identifier, node element type, node identifier of the interface to which it belongs, position coordinates of the interface control on the interface, type of operation behavior to which the interface control can respond, node identifier of the interface to which the node element can jump when triggered, interface control name, etc., and functional description of the interface control.

[0107] Among them, the node element identifier is a unique identifier assigned to the node element (with no actual meaning), which can be expressed in the form of a string. The node element identifiers of different node elements are different. The node element type is used to indicate the control type of the interface control. For example, the node element type may include at least one of the following: interactive controls, non-interactive controls, click controls, sliding controls, rocker controls, etc. The type of operation behavior that the interface control can respond to may include at least one of the following: click operation, double-click operation, sliding operation, dragging operation, gesture operation, and voice command.

[0108] 4) The behavior edge corresponding to the operation behavior, which is used to indicate the node jump situation before and after the operation behavior.

[0109] The behavior edge corresponding to the operation behavior is a directed edge, and the direction of the behavior edge is the interface jump direction, that is, the behavior edge points from the interface before the jump (ie, the starting node) to the interface after the jump (ie, the ending node).

[0110] Optionally, the operation behavior refers to a manual operation behavior of the user, such as a trigger operation on a control in the interface; or, the operation behavior refers to a system behavior, such as an automatic interface jump operation after a preset time period has been reached.

[0111] In some embodiments, a plurality of operation behavior types are preset, and the edge information of the behavior edge includes the operation behavior type to which the operation behavior belongs.

[0112] Regarding the recording method of the node jump situation before and after the operation behavior, in a possible implementation, the side information includes the node information of the corresponding node of the interface before and after the operation behavior is performed. For example, the side information includes the node identifier of the corresponding node of the interface before the operation behavior is performed, and the node identifier of the corresponding node of the interface after the operation behavior is performed. Among them, the interface identifiers corresponding to the interfaces before and after the operation behavior can be different interface identifiers (i.e., the interface jump occurs after the operation), or they can be the same interface identifier (i.e., the operation does not cause an interface jump).

[0113] The behavior edges stored in the basic file may include at least one of the following edge information: operation behavior type, trigger position or trajectory of the operation behavior, execution time of the operation behavior, execution duration of the operation behavior, node element identifier of the interface control triggered by the operation behavior, node identifier of the starting node before the operation behavior, node identifier of the ending node after the operation behavior, operation behavior name, functional description of the operation behavior, user identifier of the user who executes the operation behavior, etc.

[0114] 5) Operation path: The operation path is used to indicate the timing between operation behaviors.

[0115] Since the operation trajectory may require a series of operations to be performed and jumps between multiple interfaces, the computer device needs to sort the behavior edges corresponding to the operation behaviors according to the execution sequence between the operation behaviors to obtain the operation path (Path). Among them, the operation path is a directed path composed of multiple nodes and behavior edges between nodes.

[0116] Correspondingly, the path information of the operation path is composed of edge information, and can indicate the order relationship between the edge information and the corresponding behavior edge.

[0117] Based on the node information of the nodes involved in the operation trajectory, the edge information of the behavior edges involved, and the path information of the operation path involved, the computer device generates an operation trajectory model corresponding to the operation trajectory. The computer device stores the generated operation trajectory model so as to perform interface retrieval, positioning, and operation reproduction based on the operation trajectory model later.

[0118] 6) Factors: Factors include features used to identify nodes or node elements.

[0119] Factors include node factors and node element factors. Node factors include node factors that characterize interface features of interfaces from different dimensions. Node element factors include node element sub-factors that characterize control features of interface controls from different dimensions.

[0120] Optionally, a node factor is composed of several fields that characterize interface features from different dimensions, wherein the fields that characterize interface features can be referred to as node sub-factors. When determining whether two nodes are the same node, the computer device matches the field contents with the same field name in the two node factors, that is, matches the node sub-factors in different node factors, thereby determining whether the two nodes are the same node based on the matching results of multiple dimensions.

[0121] For example, the node factor may include interface features of at least one of the following dimensions: interface function, interface content, interface control, whether it contains images, whether it contains videos, whether it contains audio, the application to which the interface belongs, and the interface level.

[0122] Optionally, the node element factor (NodeElement Factor) is composed of several fields that characterize the control features of the interface control from different dimensions, wherein the fields that characterize the control features can be called node element sub-factors. Similar to the role of the node factor, the role of the node element factor is to retrieve and locate the interface control of the feature in the interface.

[0123] When determining whether two node elements are the same node element (i.e., determining whether two interface controls are the same interface control), the computer device matches the field contents with the same field name in the two node element factors, that is, matches the node element sub-factors in different node element factors, thereby determining whether the two node elements are the same node element based on the matching results of multiple dimensions.

[0124] 7) Operation behavior type.

[0125] Exemplarily, multiple operation behavior types may be predefined, for example, the operation behavior type may include at least one of the following: click, double click, slide, scroll, drag, shortcut key, etc. The operation behavior type of the behavior edge may be indicated from the predefined operation behavior types.

[0126] Based on the information in the above basic files, the basic architecture of the operation trajectory can be simply obtained to determine the interfaces included in the operation trajectory, the jump relationship between interfaces, the operation behaviors required for interface jumps, the association relationship between interfaces, the timing of at least two operation behaviors in the operation trajectory, etc.

[0127] Step 220: read at least one feature file of the operation trajectory; the feature file is used to store feature information of at least one object instance in different dimensions.

[0128] Exemplarily, the feature file includes at least one of the following:

[0129] 1) Root node file: The root node file is used to store the complete information of each interface control in the node.

[0130] Rootview (root node file) is used to fully record the information of each interface control in the interface, which can effectively restore the operation track. Exemplarily, a node corresponds to a root node file, and the root node file is used to record the interface control information of all interface controls contained in the interface of the node.

[0131] For example, the root node file records at least one of the following interface control information of the interface controls in the interface: node element identifier (or "interface control identifier"), interface control type, interface control name, position and / or size of the interface control in the interface, display style of the interface control, text on the interface control, type of operation behavior that the interface control can respond to, purpose of the interface control, at least one display state of the interface control, data of the interface control (for example, lists, drop-down boxes, etc. need to store the data items they contain and the display method of the data items), hierarchy of interface controls, dynamic effects of interface controls, etc.

[0132] 2) Screenshot file: The screenshot file is used to store the node interface screenshot.

[0133] Screenshot (screenshot file) is used to store the interface screenshot of the interface corresponding to the node. The screenshot file can be stored in a picture format. Exemplarily, one node corresponds to a screenshot file. The screenshot file may include an interface screenshot of the node, or the screenshot file may include at least one interface screenshot of the node. When a node corresponds to multiple interface screenshots, an interface screenshot with the most appearances can be selected as the screenshot file of the interface; or, multiple interface screenshots can be stored as the screenshot file of the node; or, multiple interface screenshots can be spliced ​​(when splicing, they can be aligned and spliced ​​according to the repeated images in the multiple interface screenshots, and the non-repeated part of the latter interface screenshot can be spliced ​​behind the repeated part of the previous interface screenshot to obtain a complete interface image of an interface) to obtain a new interface screenshot, and the new interface screenshot is stored as a screenshot file.

[0134] For example, screenshot files are not necessary files. For some interfaces with a short validity period, the interface screenshots may not be recorded and the screenshot files may not be stored.

[0135] For example, Figure 4 As shown, the screenshot file of the setting interface may include interface screenshot 101 and interface screenshot 102 .

[0136] 3) View tree file: The view tree file is used to store the summary information of each interface control in the node and the association relationship between the interface controls.

[0137] ViewTree (view tree file) is a file obtained by compressing the spatial information of each interface control in the interface, which is convenient for calling and retrieving interface controls (i.e., node elements). View tree files can be stored in text format.

[0138] Exemplarily, the view tree file may store at least one of the following interface control information of the interface control: interface control identification, location, interface control name, interface control type, node element factor, and hierarchical relationship of the interface control. Compared with the root node file, the interface control information stored in the view tree file has a smaller data volume and is more concise, which is convenient for quickly browsing or searching the interface controls contained in the node.

[0139] 4) Function file: Function file is used to store the function description text of a node or node group.

[0140] Function files are used to store the function description text of a node or node group. Function files can be stored in text format. Storing the function description text separately can reduce the content stored in the basic file and avoid the basic file from being too large.

[0141] 5) Skeleton graph file: The skeleton graph file is used to store the skeleton graph corresponding to the node.

[0142] Skeleton (skeleton file) is used to store the skeleton diagram of the node corresponding interface, which can be used to search and locate nodes. The skeleton diagram can be stored in image format.

[0143] Skeleton diagram is a visual framework used to illustrate the distribution of interface elements and the overall structure of the page. Skeleton diagram uses lines, rectangles or other basic graphics to depict the layout of the page before loading data or content in a concise and clear way. Skeleton diagram does not show specific content or details, but focuses on presenting the arrangement of interface elements, size ratios, and their relative positional relationships. For example, Figure 5 As shown, a skeleton diagram 502 of an interface is shown. The skeleton diagram 502 uses a shaded rectangular frame to mark the position and size of the text display area in the interface, and uses a white rectangular frame to mark the position and size of the image in the interface.

[0144] Step 230: deserialize the basic file and at least one feature file into at least one object instance of the operation track.

[0145] Exemplarily, the embodiment of the present application serializes each object instance in the operation track into a base file and a feature file for storage. When reading and rendering the operation track, the base file and the feature file are read from the file system, and each object instance in the operation track is obtained by deserialization based on the base file and the feature file.

[0146] Exemplarily, serialization is the process of converting primitive data types into storable or transmittable formats (such as binary format, JSON, XML, etc.) so that data can be transmitted between different environments or systems. Deserialization is the reverse process of serialization, which is to convert the serialized data back to the original data structure or object for use and operation in the program. Exemplarily, in order to persist each object instance in the operation trajectory, the object instance is serialized into storable basic files and feature files. When reading or previewing the operation trajectory, the basic files and feature files can be read and deserialized into the original object instances.

[0147] In summary, the method provided by the embodiment of the present application stores the operation track by category, uses the basic file to store the object instances contained in the operation track, and uses the feature file to store the feature information of the object instance in different dimensions, thereby reducing the data volume of a single file and improving the file reading speed. In addition, the feature information of the object instance can be stored in different feature files in the storage format that is most suitable for the feature information, ensuring the storage integrity of the feature information and facilitating the deserialization and restoration of the object instance. Using this method to access the operation track, the basic file simply and clearly records the various object instances contained in the operation track, and the feature file stores the feature information of each object instance in different dimensions in detail, making the storage structure of the operation track clearer; different types of operation tracks are stored in different files, respectively, so that the data volume of a single file is smaller, the reading speed is faster, the execution efficiency of the deserialization process is higher, and the reading of the operation track is faster.

[0148] The access system can restore the object instance based on the stored file deserialization. After obtaining the object instance, the access system can also perform functions related to the operation track based on the object instance. For example, reproduce the user's historical operations, search for the user's historical browsing information, etc.

[0149] In an optional embodiment, the access system may obtain a preview image of the operation trajectory based on object instance rendering.

[0150] Please refer to Figure 6 , which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, step 230 may include step 231 , and the method further includes step 240 .

[0151] Step 210, read the basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track.

[0152] Step 220: read at least one feature file of the operation trajectory; the feature file is used to store feature information of at least one object instance in different dimensions.

[0153] Exemplarily, the object instance includes a node, a node group, a behavior edge, an operation path, a factor, a node element, and an operation behavior type. The feature file includes a root node file.

[0154] Step 231, based on the basic file, deserialize to obtain a first type of object instance, the first type of object instance includes at least one of the following: node, node group, behavior edge, operation path, factor, operation behavior type; based on the root node file, deserialize to obtain a second type of object instance, the second type of object instance includes node elements corresponding to the interface control.

[0155] In an optional embodiment, the first class object instance includes a first node; the feature file also includes at least one of the following: a screenshot file, a view tree file, a function file, and a skeleton diagram file. The access system can perform at least one of the following to deserialize and obtain the first node:

[0156] In the case where the first node corresponds to a screenshot file, deserialization is performed based on the basic file and the screenshot file to obtain the first node, where the first node includes the interface screenshot stored in the screenshot file;

[0157] In the case where the first node corresponds to a view tree file, deserializing the basic file and the view tree file to obtain the first node, where the first node includes summary information of the interface control stored in the view tree file;

[0158] In the case where the first node corresponds to a function file, the first node is obtained by deserializing the basic file and the function file, and the first node includes a function description text stored in the function file;

[0159] In the case where the first node corresponds to a skeleton graph file, the first node is obtained by deserializing the basic file and the skeleton graph file, and the first node includes the skeleton graph stored in the skeleton graph file.

[0160] The basic file stores the node identifier of the first node, and the access system can traverse the feature file based on the node identifier of the first node to determine whether the first node corresponds to a feature file. Alternatively, the basic file stores the feature file corresponding to the first node.

[0161] It should be noted that when there is a feature file corresponding to the first node, the access system can synchronously read the basic file and the feature file, and synchronously perform the deserialization process on the basic file and the feature file (for example, deserialization of the basic file is performed through the first process, and deserialization of the feature file is performed through the second process, and the first process and the second process run at the same time) to restore the first node to improve the deserialization efficiency.

[0162] Step 240, based on at least one object instance, a preview image of the operation trajectory is rendered; the preview image includes a node corresponding to at least one interface in the operation trajectory, and a behavior edge connecting at least one node according to the interface jump situation corresponding to the operation behavior.

[0163] Exemplarily, the visual elements in the preview image mainly include: a node corresponding to the interface, a behavior edge consisting of a start node and an end node, and an operation track consisting of multiple nodes and behavior edges.

[0164] Optionally, the preview image includes at least nodes and behavior edges connecting the nodes. The preview image may also include an operation path formed by connecting nodes and behavior edges. Among them, the node can be displayed as a screenshot of the corresponding interface, or as a skeleton diagram of the corresponding interface, or as the interface name of the corresponding interface. The behavior edge is used to indicate the operation behavior and connect the nodes before and after the operation behavior. The behavior edge points from the starting node to the ending node.

[0165] For example, Figure 7 A preview diagram is shown, which includes a node 11 and a behavior edge 12. The preview diagram shows that the operation track jumps from interface 1 to interface 2, continues to jump to interface 3, interface 4, interface 5, and returns from interface 5 to interface 2 to form an operation path.

[0166] In an optional embodiment, when collecting the operation track, if the collection speed is fast, it may not be possible to capture the interface screenshot. In this case, the preview image will automatically use the skeleton image of the interface to replace the interface screenshot, so as to reproduce the operation track in extreme scenarios. Figure 8 As shown, the skeleton is used in the preview image Figure 1 To replace Figure 7 Interface 1 in the code, using the skeleton Figure 3 To replace Figure 7 Interface 2 in , using the skeleton Figure 3 To replace Figure 7 Interface 3 in , using the skeleton Figure 4 To replace Figure 7 Interface 4 in , using the skeleton Figure 5 To replace Figure 7 Interface 5.

[0167] Exemplarily, the rendering range in the preview image may include: the interface of the current application, the interface for entering the current application (the step length with the interface of the current application is 1, that is, connected to the interface of the current application through a behavior edge), and the interface for leaving the current application (the step length with the interface of the current application is 1, that is, connected to the interface of the current application through a behavior edge).

[0168] In an optional embodiment, when the preview image is used to render the operation track of the first application, the nodes belonging to the first application and the nodes not belonging to the first application can be distinguished and displayed in the preview image. Figure 7 As shown, nodes of different applications are distinguished by different shapes, nodes belonging to the first application are displayed as rectangles, and nodes not belonging to the first application are displayed as circles 15.

[0169] In another optional embodiment, the user can click on a node in the preview image to view detailed information of the node (e.g., any type of node information). The preview image can also display the nodes selected by the user separately. For example, the unselected nodes in the preview image are displayed as a first size, and the selected nodes are displayed as a second size, where the second size is larger than the first size. In response to the operation of selecting the first node in the preview image, the first node is converted from the first size to the second size to indicate that the first node is currently selected.

[0170] Exemplarily, when there are at least two behavior edges with the same start node and end node in the operation track, the at least two behavior edges can be merged and rendered into one behavior edge in the preview image, and the edge information of the two behavior edges can be merged and displayed in the preview image.

[0171] For example, in the default state, the preview image can simply display nodes and behavior edges, and does not display nodes, behavior edges, node groups, operation paths, etc. The user can view the node information corresponding to the node by selecting a node, or view the edge information corresponding to the behavior edge by selecting a behavior edge. The node information can include the node group to which the node belongs, and related information of the node group.

[0172] In an optional embodiment, the user can select a node or a behavior edge to view the operation path to which the node or behavior edge belongs, and select the operation path to view the nodes and behavior edges contained in the operation path, as well as related information.

[0173] In another optional embodiment, the user may view the operation path including the at least two nodes or behavior edges by selecting the at least two nodes or behavior edges.

[0174] In yet another optional embodiment, the computer device may quickly render a brief preview image of the operation behavior based on the basic file, where the brief preview image includes each object instance in the basic file.

[0175] In an optional embodiment, the access system renders the interface based on the node; renders the directed edge from the start node to the end node based on the behavior edge, the start node includes the interface for executing the operation behavior corresponding to the behavior edge, and the end node includes the interface to jump to after executing the operation behavior corresponding to the behavior edge; and renders the trajectory obtained by connecting the nodes and the behavior edges based on the operation path.

[0176] The preview system renders the node object instance as the interface in the preview image, renders the behavior edge object instance as the directed edge connecting the interface in the preview image, and connects the nodes and behavior edges according to the operation path rendering order to obtain the final preview image.

[0177] In summary, the embodiment of the present application extracts each object instance in the operation trajectory and stores it in a basic file; and stores the feature information of each object instance in multiple feature files respectively. Based on the basic file, each object instance in the operation trajectory can be clarified, and the association relationship between each object instance in the operation trajectory can be constructed. Based on the feature file, the feature information of each object instance is supplemented. The complete operation trajectory of the basic file is completed with the feature file, and then a preview of the operation trajectory is rendered. The operation trajectory has a logically clear storage structure, which is convenient for access, display and modification of the operation trajectory, and improves the access efficiency of the operation trajectory.

[0178] The method provided in the embodiment of the present application stores the interfaces, operation behaviors, operation paths, interface controls, interface groups, etc. in the operation track as independent object instances, identifies the interfaces that meet the similarity conditions as the same interface based on feature information, and restores the actual interface jump logic of the operation track.

[0179] The method provided in the embodiment of the present application stores the feature information of the above-mentioned object instance in different dimensions as multiple feature files, reduces the data volume of the basic file, improves the access structure of the operation trajectory, makes the storage logic of the operation trajectory clear, and facilitates the storage and management of the operation trajectory.

[0180] In the method provided by the embodiment of the present application, since the same interface in the operation track is stored as a node, when rendering the preview image of the operation track based on the stored data, each node displayed in the preview image is a different interface, and there is no situation where the same interface corresponds to multiple nodes, so that the structure of the preview image is clearer, the interface jump relationship is more concise, and it is convenient for users to view the interface jump relationship of the application from the preview image, thereby improving the efficiency of browsing and editing the operation track.

[0181] In an optional embodiment, if Fig. 9 As shown, access to the operation track is implemented by the file system 301 , the object system 302 and the preview system 303 .

[0182] In the file system 301, the basic file is used as a list directory to split the object instances in the operation trajectory for storage, ensuring data integrity while avoiding the problem of excessive single file size; at the same time, the feature file uses content classification to store the corresponding data content in the most suitable format (such as pictures to store interface screenshots and skeleton diagrams, json format to store interface control information, and text format to store compressed interface tree information).

[0183] The object system 302 can deserialize the data in the basic files and feature files in the file system into complete object instances for direct use by the object system. The basic files are restored to generate object instances such as node groups, nodes, behavior edges, operation paths, factors, and operation behavior types; the root node files are restored to generate node element object instances.

[0184] Exemplarily, through a customized serialization / deserialization process, one-click conversion between the file system and the object system is achieved, which greatly reduces the information link cost between the two systems.

[0185] The preview system 303 renders a preview image of the operation track based on the predefined operation track rendering rules by using the object instances generated in the object system and the static data stored in the file system. For example, the operation track is rendered in the application dimension by default, and a preview image is rendered for the operation track of an application.

[0186] For example, the use of operation traces generally includes two processes: reading from the file system, deserializing into an object instance, and reading the operation trace field; and modifying the operation trace field, serializing into a string, and writing to the file system. In the operation trace application scenario, the above two processes will be frequently triggered to ensure data consistency and operation simplicity. Fig.10 As shown, an operation track virtual environment 304 is provided, and the user of the operation track can directly obtain the object instance from the object system 302 through a predefined operation track acquisition method (such as the "getOUTS" method), without having to pay attention to how the object instance is stored and used in the file system 301; the entire process is implemented by the "operation track virtual environment 304". For example:

[0187] val outs=OUTSRepo.getOUTS(targetPackage!!)

[0188] println(">>>OUTS contains ${outs.edges.size} edges")

[0189] outs.edges.removeLast()

[0190] println(">>>Delete the last edge, leaving ${outs.edges.size} edges")

[0191] After the code is executed, the operation track virtual environment 304 will automatically synchronize the operation track field changes to the file system. The operation track virtual environment 304 can also call "notifyChanged (change notification)" to immediately update the content of the operation track field changes to the file system. Optionally, without calling the change notification, the operation track virtual environment 304 can also wait for a heartbeat duration (for example, 3s) and then synchronize all operation track field changes within the heartbeat time period to the file system.

[0192] In an optional embodiment, the computer device receives an operation of modifying a target field of an object instance; serializes the modified target field into a target string; and updates a base file and / or a feature file based on the target string. Optionally, the computer device may periodically obtain target strings that have changed during this period, and synchronize the target strings to the base file and / or the feature file. The computer device may also call a change notification, carrying the changed target string in the change notification, so that the file system synchronizes the target string to the base file and / or the feature file based on the change notification.

[0193] In another optional embodiment, the operation track access system should have the ability to synchronize the end and the cloud. Fig.11 As shown, the file system 301 includes an end-side file system, that is, a file system stored locally on a computer device; the file system 301 also includes a cloud file system, that is, a file system stored on a cloud server. The end-side file system and the cloud file system can transmit data to each other, realizing data transmission of operation tracks between multiple terminals. The operation track virtual environment 304 supports reading files of operation tracks from the end-side file system and the cloud file system, and generating object instances of operation tracks, so that the operation tracks can be previewed directly in the cloud. The operation track virtual environment 304 is used to realize complete storage virtualization, ensuring that the operation track can be obtained anytime, anywhere and at any time.

[0194] Exemplarily, the storage and preview of the operation trace is not limited to the above-mentioned information, and the system can also store other information. When expanding the system to store new information, it is only necessary to rewrite the operation trace virtual environment 304 and adjust the logic of reading and writing the file system, that is, to quickly expand the required capabilities. For example:

[0195]

[0196] In an optional embodiment, if Fig.12As shown, in addition to providing intuitive preview capabilities of nodes and behavior edges, the preview system also provides a variety of viewing and editing functions: the function of viewing interface controls 401, the function of viewing similarities 402, the function of deleting nodes or behavior edges 403, the function of viewing skeleton diagrams 403, the function of interface flashback 404, etc.

[0197] 1. Delete a node or behavior edge (Delete).

[0198] Please refer to Fig.13 , which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, the method further comprises the following steps.

[0199] Step 241: receiving an operation of deleting a second node in the preview graph, and deleting the second node; and / or receiving an operation of deleting a first row as an edge in the preview graph, and deleting the first row as an edge.

[0200] Optionally, the preview system allows deleting some nodes or behavior edges when previewing the operation trajectory. For example, a delete control is displayed in the display interface of the preview image, and the user can select the node or behavior edge to be deleted in the preview image, and then click the delete control to delete the selected node or behavior edge.

[0201] Exemplarily, when the second node is deleted, the behavior edge connected to the second node can be automatically deleted. When the first behavior edge is deleted, if the first behavior edge is used to connect the first node and the second node, and the first node is not connected to other behavior edges other than the first behavior edge, the first node is deleted in the preview graph. If the second node is connected to the second behavior edge, the second node is retained in the preview graph.

[0202] Optionally, the deletion operation of nodes or behavior edges in the preview image can be synchronized to the base file and feature file for storage. In the case of deleting the second node, the operation track virtual environment synchronously deletes the second node stored in the base file, or sets the second node to an inactive state. The node in the inactive state will not be deserialized into an object instance and will not be displayed in the preview image. Similarly, in the case of deleting the first behavior edge, the operation track virtual environment synchronously deletes the first behavior edge stored in the base file, or sets the first behavior edge to an inactive state. The behavior edge in the inactive state will not be deserialized into an instance object.

[0203] Optionally, the user can also edit and modify other information of the object instance in the preview. For example, the node element information of the node element can be adjusted in the preview, the interface screenshot of the node can be changed, the operation behavior type of the edge can be changed, etc. The user's modification content of the operation track in the preview will be serialized into a string and stored in the file system synchronously.

[0204] 2. View the interface controls (Viewtree-preview).

[0205] Please refer to Fig.14 , which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, the method further comprises the following steps.

[0206] Step 251: receiving an operation of selecting a third node in the preview image and a preview interface control.

[0207] Step 252: Display the compressed interface tree corresponding to the third node, where the compressed interface tree includes summary information of each interface control in the third node and the association relationship between the interface controls.

[0208] For example, a control view control is displayed in the display interface of the preview image, and the user can select the node to be viewed in the preview image and then click the control view control. Fig.15 As shown, the computer device will display the compressed interface tree 501 corresponding to the node. The compressed interface tree lists the brief information of the settlement elements contained in the node. The user can quickly understand the interface controls in the interface by browsing the compressed interface tree, and can also quickly search for interface controls.

[0209] 3. View the skeleton diagram (skeleton-preview).

[0210] Please refer to Fig.16 , which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, the method further comprises the following steps.

[0211] Step 261, receiving an operation of selecting the fourth node in the preview image and viewing the skeleton image.

[0212] Step 262, displaying the skeleton graph corresponding to the fourth node.

[0213] For example, a skeleton diagram control is displayed in the display interface of the preview image, and the user can select the node to be viewed in the preview image and then click the skeleton diagram control. Figure 5As shown, the computer device will display the skeleton graph 502 corresponding to the node.

[0214] 4. Check the interface similarity (Similar).

[0215] Please refer to Fig.17 , which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, the method further comprises the following steps.

[0216] Step 271 , receiving an operation of selecting the fifth node and the sixth node in the preview graph and checking the similarity.

[0217] Step 272, displaying the similarity between the interface corresponding to the fifth node and the interface corresponding to the sixth node.

[0218] For example, the computer device can calculate the similarity of the two nodes based on the node information of the two nodes, and the user can determine whether the two nodes belong to the same interface by checking the similarity of the two nodes. Alternatively, the computer device can also calculate the similarity of the two nodes based on the node factors of the two nodes.

[0219] Exemplarily, a similarity control is displayed in the display interface of the preview image, and the user can select two nodes to be viewed in the preview image, and then click the similarity control, and the system will calculate the similarity of the two nodes and display the similarity.

[0220] In an optional embodiment, when the similarity is higher than a threshold, the preview system may further display a node information comparison diagram of the fourth node and the fifth node. For example, the node information comparison diagram may display the node information of the fourth node on the left, and display the same type of node information of the fifth node at the corresponding position on the right, and calculate the similarity of each type of node information respectively, and highlight the two node information corresponding to the types with similarity higher than the threshold, so that the user can further determine whether the two nodes belong to the same node.

[0221] In another optional embodiment, when the similarity is higher than a threshold, a merge control may be displayed on the interface, and the user clicks the merge control to automatically merge the fifth node and the sixth node into the same node, and the node information of the fifth node and the sixth node is deduplicated and merged and stored in a file. Exemplarily, the node with more node information of the two nodes may be automatically retained, or the node with more connecting behavior edges of the two nodes may be retained, and the other node may be deleted.

[0222] 5. The interface corresponding to the flashback node (Navigation).

[0223] Please refer to Fig.18, which shows a flowchart of an operation track access method provided by an exemplary embodiment of the present application. This embodiment takes the method used in a computer device as an example for explanation. Figure 3 In the illustrated embodiment, the method further comprises the following steps.

[0224] Step 281, receiving the seventh node selected in the preview image and the flashback operation.

[0225] Step 282, navigating to the interface corresponding to the seventh node by simulating a click.

[0226] Exemplarily, the interface flashback function means that the computer device can automatically control the operating system or application navigation to jump to the interface selected by the user based on the operation track. For example, if the user selects the settings interface and triggers the flashback control, the computer device can automatically open the settings interface through simulated operation.

[0227] like Fig.19 As shown, the user can select interface 4 in the preview image and then click the flashback control 503. The computer device will automatically navigate to open the operation method of interface 4 based on the operation track related to interface 4, open interface x and interface y in sequence based on the operation method, and finally jump from interface y to interface 4.

[0228] In summary, the method provided by the embodiment of the present application provides a variety of methods for viewing and modifying operation trajectories in the preview image interface, allowing users to quickly edit operation trajectories in the preview image, quickly view the skeleton diagram of the interface, interface controls, similarity and other related information, and quickly execute interface flashback, thereby improving the efficiency of users browsing and editing operation trajectories.

[0229] The method provided in the embodiment of the present application can form a complete link from the collection to the use of operation trajectory data through the operation trajectory access preview solution. The relevant terms are aligned, which is conducive to the implementation and realization of subsequent application scenarios related to the operation trajectory. A concise and effective preview editing and additional function system is provided, making the operation trajectory editing and preview more convenient and intuitive; it is helpful to guide the promotion and publicity of related technologies through this concept in the industry.

[0230] Virtual Device Test Fig. 20 , which shows a structural block diagram of an operation trace access device provided by an exemplary embodiment of the present application. The device includes:

[0231] File module 1001, used to read the basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track;

[0232] The file module 1001 is used to read at least one feature file of the operation trajectory; the feature file is used to store feature information of the at least one object instance in different dimensions;

[0233] The object module 1002 is used to deserialize the basic file and the at least one feature file into the at least one object instance of the operation track.

[0234] In an optional embodiment, the feature file includes a root node file, and the root node file is used to store information of each interface control in the at least one interface;

[0235] The object module 1002 is used to deserialize the basic file to obtain a first type of object instance, wherein the first type of object instance includes at least one of the following: a node, a node group, a behavior edge, an operation path, a factor, and an operation behavior type;

[0236] The object module 1002 is used to deserialize and obtain a second type of object instance based on the root node file, where the second type of object instance includes a node element corresponding to the interface control.

[0237] In an optional embodiment, the first type of object instance includes a first node; the feature file also includes at least one of the following: a screenshot file, a view tree file, a function file, and a skeleton diagram file;

[0238] The object module 1002 is configured to perform at least one of the following:

[0239] In the case where the first node corresponds to the screenshot file, deserializing based on the basic file and the screenshot file to obtain the first node, where the first node includes the interface screenshot stored in the screenshot file;

[0240] In the case where the first node corresponds to the view tree file, deserializing the basic file and the view tree file to obtain the first node, wherein the first node includes summary information of the interface control stored in the view tree file;

[0241] In the case where the first node corresponds to the function file, the first node is obtained by deserializing the basic file and the function file, and the first node includes a function description text stored in the function file;

[0242] In the case that the first node corresponds to the skeleton graph file, the first node is obtained by deserializing the basic file and the skeleton graph file, and the first node includes the skeleton graph stored in the skeleton graph file.

[0243] In an optional embodiment, the object instance includes at least one of the following:

[0244] A node corresponding to the at least one interface;

[0245] A node group consisting of at least one node;

[0246] A node element corresponding to an interface control in the at least one interface;

[0247] A behavior edge corresponding to the operation behavior, the behavior edge is used to indicate the interface jump situation before and after the operation behavior;

[0248] an operation path, the operation path comprising the behavior edges and the nodes connected in sequence;

[0249] A factor, wherein the factor includes a feature for identifying the node or the node element;

[0250] The operation behavior type of the operation behavior.

[0251] In an optional embodiment, the feature file includes a root node file, and the root node file is used to store information of each interface control in the at least one interface; the feature file also includes zero or more of the following files:

[0252] A screenshot file, the screenshot file is used to store the interface screenshot of the at least one interface;

[0253] A view tree file, wherein the view tree file is used to store summary information of each interface control in the at least one interface and association relationships between the each interface control;

[0254] A function file, wherein the function file is used to store a function description text of the node or node group;

[0255] A skeleton graph file, wherein the skeleton graph file is used to store a skeleton graph of the at least one interface.

[0256] In an optional embodiment, the device further includes:

[0257] The preview module 1003 is used to render a preview image of the operation trajectory based on the at least one object instance; the preview image includes a node corresponding to the at least one interface in the operation trajectory, and a behavior edge connecting at least one node according to the interface jump situation corresponding to the operation behavior.

[0258] In an optional embodiment, the object instance includes at least one of at least one node, a behavior edge, and an operation path;

[0259] The preview module 1003 is used to perform at least one of the following:

[0260] Rendering the at least one interface based on the at least one node;

[0261] Rendering a directed edge from a start node to an end node based on the behavior edge, wherein the start node includes a node before the operation behavior corresponding to the behavior edge is executed in the at least one node, and the end node includes a node to which the at least one node jumps after the operation behavior corresponding to the behavior edge is executed in the at least one node;

[0262] The nodes and the behavior edges are connected in a rendering order based on the operation path.

[0263] In an optional embodiment, the preview module 1003 is used to perform at least one of the following:

[0264] Receive an operation of deleting a second node in the preview graph, and delete the second node;

[0265] An operation of deleting a first line in the preview image as an edge is received, and the first line is deleted.

[0266] In an optional embodiment, the preview module 1003 is used to receive an operation of selecting a third node in the preview image and a preview interface control;

[0267] The preview module 1003 is used to display the compressed interface tree corresponding to the third node, and the compressed interface tree includes summary information of each interface control in the third node and the association relationship between the interface controls.

[0268] In an optional embodiment, the preview module 1003 is used to receive an operation of selecting the fourth node in the preview image and viewing the skeleton image;

[0269] The preview module 1003 is used to display the skeleton diagram corresponding to the fourth node.

[0270] In an optional embodiment, the preview module 1003 is used to receive an operation of selecting the fifth node and the sixth node in the preview graph and checking the similarity;

[0271] The preview module 1003 is used to display the similarity between the interface corresponding to the fifth node and the interface corresponding to the sixth node.

[0272] In an optional embodiment, the preview module 1003 is used to receive an operation of selecting the seventh node in the preview image and flashing back;

[0273] The preview module 1003 is used to navigate to the interface corresponding to the seventh node by simulating a click.

[0274] In an optional embodiment, the preview module 1003 is used to receive an operation of modifying a target field of a first object instance, where the first object instance is one of the at least one object instance;

[0275] The object module 1002 is used to serialize the modified target field into a target string;

[0276] The file module 1001 is used to update the basic file and / or the feature file based on the target character string.

[0277] See also Fig.21 , Fig.21 15 is a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device can be implemented as a terminal device or a server in the above embodiment. The computer device can also include one or more of the following components: a processor 1510 and a memory 1520.

[0278] Optionally, the processor 1510 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1520, and calling data stored in the memory 1520. Optionally, the processor 1510 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1510 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1510, but may be implemented by a separate chip.

[0279] The memory 1520 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1520 includes a non-transitory computer-readable storage medium. The memory 1520 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0280] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device, and the computer device may include more (such as communication components, display components) or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0281] An embodiment of the present application provides a computer-readable storage medium storing at least one computer instruction, wherein the at least one computer instruction is used to be executed by a processor to implement a method for accessing operation traces on a terminal device or an access server side of the operation trace as described in the above embodiment.

[0282] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for accessing operation traces on the terminal device or the access server side of the operation trace as described in the above embodiments.

[0283] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0284] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for accessing an operation track, characterized in that: The method comprises: Reading a basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track; Reading at least one feature file of the operation trajectory; the feature file is used to store feature information of the at least one object instance in different dimensions; The basic file and the at least one feature file are deserialized into the at least one object instance of the operation track.

2. The method according to claim 1, characterized in that The feature file includes a root node file, and the root node file is used to store information of each interface control in the at least one interface; The deserializing the basic file and the at least one feature file into the at least one object instance of the operation track includes: Based on the basic file, deserialize to obtain a first class object instance, wherein the first class object instance includes at least one of the following: a node, a node group, a behavior edge, an operation path, a factor, and an operation behavior type; Based on the root node file, deserialization is performed to obtain a second type of object instance, where the second type of object instance includes a node element corresponding to the interface control.

3. The method according to claim 2, characterized in that The first type of object instance includes a first node; the feature file also includes at least one of the following: a screenshot file, a view tree file, a function file, and a skeleton diagram file; Deserializing based on the basic file to obtain a first class object instance includes at least one of the following: In the case where the first node corresponds to the screenshot file, deserializing based on the basic file and the screenshot file to obtain the first node, where the first node includes the interface screenshot stored in the screenshot file; In the case where the first node corresponds to the view tree file, deserializing the basic file and the view tree file to obtain the first node, wherein the first node includes summary information of the interface control stored in the view tree file; In the case where the first node corresponds to the function file, the first node is obtained by deserializing the basic file and the function file, and the first node includes a function description text stored in the function file; In the case that the first node corresponds to the skeleton graph file, the first node is obtained by deserializing the basic file and the skeleton graph file, and the first node includes the skeleton graph stored in the skeleton graph file.

4. The method according to any one of claims 1 to 3, characterized in that: The object instance includes at least one of the following: A node corresponding to the at least one interface; A node group consisting of at least one node; A node element corresponding to an interface control in the at least one interface; A behavior edge corresponding to the operation behavior, the behavior edge is used to indicate the interface jump situation before and after the operation behavior; an operation path, the operation path comprising the behavior edges and the nodes connected in sequence; A factor, wherein the factor includes a feature for identifying the node or the node element; The operation behavior type of the operation behavior.

5. The method according to any one of claims 1 to 3, characterized in that: The feature file includes a root node file, and the root node file is used to store information of each interface control in the at least one interface; the feature file also includes zero or more of the following files: A screenshot file, the screenshot file is used to store an interface screenshot of the at least one interface; A view tree file, wherein the view tree file is used to store summary information of each interface control in the at least one interface and association relationships between the each interface control; A function file, wherein the function file is used to store a function description text of the node or node group; A skeleton graph file, wherein the skeleton graph file is used to store a skeleton graph of the at least one interface.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Based on the at least one object instance, a preview image of the operation trajectory is rendered; the preview image includes a node corresponding to the at least one interface in the operation trajectory, and a behavior edge connecting at least one node according to the interface jump situation corresponding to the operation behavior.

7. The method according to claim 6, characterized in that The object instance includes at least one of at least one node, a behavior edge, and an operation path; The rendering, based on the at least one object instance, to obtain a preview image of the operation trajectory includes at least one of the following: Rendering the at least one interface based on the at least one node; Rendering a directed edge from a start node to an end node based on the behavior edge, wherein the start node includes a node before the operation behavior corresponding to the behavior edge is executed in the at least one node, and the end node includes a node to which the at least one node jumps after the operation behavior corresponding to the behavior edge is executed in the at least one node; The nodes and the behavior edges are connected in sequence based on the operation path rendering order.

8. The method according to claim 6, characterized in that The method further comprises at least one of the following: Receiving an operation of deleting a second node in the preview graph, and deleting the second node; An operation of deleting a first line in the preview image as an edge is received, and the first line is deleted.

9. The method according to claim 6, characterized in that The method further comprises: Receive an operation of selecting a third node in the preview image and a preview interface control; The compressed interface tree corresponding to the third node is displayed, wherein the compressed interface tree includes summary information of each interface control in the third node and association relationships between the interface controls.

10. The method according to claim 6, characterized in that The method further comprises: Receive an operation of selecting the fourth node in the preview image and viewing the skeleton image; The skeleton diagram corresponding to the fourth node is displayed.

11. The method according to claim 6, characterized in that The method further comprises: Receive an operation of selecting the fifth node and the sixth node in the preview graph and checking the similarity; The similarity between the interface corresponding to the fifth node and the interface corresponding to the sixth node is displayed.

12. The method according to claim 6, characterized in that The method further comprises: Receive an operation of selecting the seventh node in the preview image and flashing back; Navigate to the interface corresponding to the seventh node by simulating a click.

13. The method according to claim 6, characterized in that The method further comprises: receiving an operation to modify a target field of a first object instance, the first object instance being one of the at least one object instance; Serializing the modified target field into a target string; The base file and / or the feature file is updated based on the target character string.

14. An access device for operation track, characterized in that: The device comprises: A file module, used to read a basic file of the operation track; the operation track is used to record the operation behavior in at least one interface and the interface jump situation before and after the operation behavior; the basic file is used to indicate at least one object instance contained in the operation track; The file module is used to read at least one feature file of the operation trajectory; the feature file is used to store feature information of the at least one object instance in different dimensions; The object module is used to deserialize the basic file and the at least one feature file into the at least one object instance of the operation track.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the method for accessing the operation trajectory as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for accessing the operation trajectory as described in any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for accessing the operation trajectory as described in any one of claims 1 to 13.

Images