Method, device, equipment and computer program product for displaying machine usage

Abstract

The embodiments of the present application disclose a method, device, equipment and computer program product for displaying machine usage, which belongs to the field of human-computer interaction. The method includes: generating operation path information during machine usage, wherein the operation path information is used to characterize at least one of the operation paths between applications and the operation paths within an application; upon receiving a machine usage display instruction, displaying a machine usage display interface, wherein the machine usage display interface includes a usage display control; in response to a trigger operation on the usage display control, displaying an operation path diagram based on the operation path information, wherein the operation path diagram includes at least one of the jump relationships between different applications and the jump relationships between different interfaces in the same application. The solution provided by the embodiments of the present application can facilitate users to review historical operations within and between applications, and realize a more fine-grained machine usage display.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction, and in particular to a method, device, equipment and computer program product for displaying computer usage status. Background Art

[0002] The machine usage display function is a function that centrally displays the terminal usage within a unit time (such as one day).

[0003] In the related art, the information provided by the machine usage display function includes the usage time of each application and the power consumption in units of time. Summary of the invention

[0004] The present application provides a method, device, equipment and computer program product for displaying the usage of a device. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides a method for displaying a usage status of a device, the method comprising:

[0006] During the use of the machine, operation path information is generated, where the operation path information is used to represent at least one of an operation path between applications and an operation path within an application;

[0007] When receiving a usage status display instruction, displaying a usage status display interface, wherein the usage status display interface includes a usage status display control;

[0008] In response to a triggering operation on the usage display control, an operation path graph is displayed based on the operation path information, and the operation path graph includes at least one of a jump relationship between different applications and a jump relationship between different interfaces in the same application.

[0009] On the other hand, an embodiment of the present application provides a device for displaying a usage status of a mobile phone, the device comprising:

[0010] A recording module, used to generate operation path information during the use of the machine, wherein the operation path information is used to represent at least one of an operation path between applications and an operation path within an application;

[0011] A display module, configured to display a usage status display interface upon receiving a usage status display instruction, wherein the usage status display interface includes a usage status display control;

[0012] The display module is also used to respond to the triggering operation of the usage display control and display an operation path diagram based on the operation path information. The operation path diagram includes at least one of the jump relationships between different applications and the jump relationships between different interfaces in the same application.

[0013] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the method for displaying machine usage as described in the above aspects.

[0014] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for displaying machine usage as described in the above aspects.

[0015] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for displaying machine usage as described in the above aspects.

[0016] In an embodiment of the present application, by recording the operation paths between applications and the operation paths within an application during the use of the machine and generating operation path information, when the machine usage status is subsequently displayed, an operation path diagram representing the jump relationship between different applications and the jump relationship between different interfaces in the same application can be restored based on the operation path information, thereby facilitating users to review historical operations within and between applications and achieving a more fine-grained display of the machine usage status. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of an operation trajectory model corresponding to an operation process shown in an exemplary embodiment of the present application;

[0018] Figure 2 A flowchart of a method for displaying a machine usage situation provided by an exemplary embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of an interface for displaying a machine usage status provided by an exemplary embodiment of the present application is shown;

[0020] Figure 4 is a flowchart of a global and application operation path information generation process shown in an exemplary embodiment of the present application;

[0021] Figure 5 is a schematic diagram of an interface skeletonization process shown in an exemplary embodiment of the present application;

[0022] Figure 6 is a schematic diagram of interface skeleton similarity calculation shown in an exemplary embodiment of the present application;

[0023] Figure 7 A schematic diagram of an interface for displaying a machine usage status provided by another exemplary embodiment of the present application is shown;

[0024] Figure 8 is a flowchart of a content backtracking process shown in an exemplary embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of an implementation of a content backtracking process shown in an exemplary embodiment of the present application;

[0026] Fig.10 is a schematic diagram of a machine usage report generation process shown in an exemplary embodiment of the present application;

[0027] Fig.11 A structural block diagram of a device for displaying machine usage provided by another exemplary embodiment of the present application is shown;

[0028] Fig.12 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0030] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0031] For ease of understanding, the operation trajectory model involved in the embodiments of the present application is first described below.

[0032] The operation trajectory model (Trajectory Structure) is a model (or data structure) used to record the operation process. Figure 1 As shown, the operation trajectory model 13 corresponding to the operation process is composed of nodes 131, behavior edges 132 and operation paths 133. Among them, the node 131 corresponds to the interface 11 passed during the operation process; the behavior edge 132 corresponds to the operation behavior in the interface, which is used to connect different nodes (interface switching occurs after the operation) or connect the same node (no interface switching occurs after the operation) 12; the operation path 133 corresponds to the operation behavior sequence.

[0033] In addition, in order to ensure the accuracy of the operation process record, the node information 141 of the node 131 represents the interface features of the interface from different dimensions to ensure the accuracy of the subsequent interface retrieval and positioning; the edge information 142 of the behavior edge 132 indicates the operation behavior and the node jumps before and after the operation behavior to ensure the accurate restoration of the operation in the subsequent interface; the path information 143 of the operation path 133 represents the execution sequence between different operation behaviors to ensure that complex operations are executed in sequence.

[0034] Compared with screenshots and videos, the operation trajectory model has a lower data volume and can be automatically generated without manual triggering by the user. In addition, it can automatically backtrack and reproduce based on node information, edge information, and path information, improving the efficiency of operation backtracking and reproduction.

[0035] Regarding the method of generating the operation trajectory model, in one possible implementation, a computer device traverses the entire interface of the application and all interface elements in the interface, and performs operations on the interface elements, thereby generating an operation trajectory model that includes node information representing interface features, edge information representing operation behaviors, and path information representing operation paths.

[0036] node

[0037] The interface is a visual user interface, such as a desktop, an application program interface, a system interface, etc. In the embodiment of the present application, different interfaces in the operation process correspond to respective nodes, and different nodes represent the interface features of their respective corresponding interfaces through their respective node information, that is, different nodes can be distinguished based on the node information.

[0038] Optionally, dimensions characterizing interface features may include interface content dimension, interface layout dimension, interface function dimension, interface description dimension, the window to which the interface belongs, etc. The following embodiments will describe in detail the specific content contained in the node information.

[0039] The interface in this embodiment is not limited to the visible range of the screen, but includes the completed interface within and outside the visible range of the screen. Accordingly, before and after the interface content is scrolled, even if the displayed content within the visible range of the screen changes significantly, the interface before and after the scrolling will not be judged as a different interface.

[0040] In the embodiments of the present application, different interfaces refer to interfaces with significant differences in interface layout, content, and style. Usually, when an interface switch occurs, the interfaces before and after the switch are determined to be different interfaces. For example, when text or images in an interface change dynamically, or when list elements in an interface change, they are not determined to be different interfaces; when clicking on a control in an interface to trigger an interface jump, the interfaces before and after the jump are determined to be different interfaces.

[0041] Behavior Edge

[0042] The behavior edge (Edge) corresponding to the operation behavior is a directed edge, and the direction of the behavior edge is the interface jump direction, that is, the behavior edge points from the interface before the jump to the interface after the jump.

[0043] Optionally, the operation behavior refers to a manual operation behavior of the user, such as a trigger operation on a control in the interface; or, the operation behavior refers to a system behavior, such as an automatic interface jump operation after a preset time period has been reached.

[0044] In some embodiments, a plurality of operation behavior types are preset, and the edge information of the behavior edge includes the operation behavior type to which the operation behavior belongs.

[0045] Regarding the recording method of the node jump situation before and after the operation behavior, in a possible implementation, the side information includes information about the corresponding nodes of the interface before and after the operation behavior is performed. For example, the side information includes the node identifier of the node corresponding to the interface before the operation behavior is performed, and the node identifier of the node corresponding to the interface after the operation behavior is performed. Among them, the interface identifiers corresponding to the interfaces before and after the operation behavior can be different interface identifiers (i.e., the interface jump occurs after the operation), or they can be the same interface identifier (i.e., the operation does not cause an interface jump).

[0046] Operation Path

[0047] Since a series of operations may need to be performed during the operation process and multiple interfaces may need to be jumped, the computer device needs to sort the behavior edges corresponding to the operation behaviors according to the execution sequence between the operation behaviors to obtain the operation path (Path). Among them, the operation path is a directed path composed of multiple nodes and behavior edges between nodes.

[0048] Correspondingly, the path information of the operation path is composed of edge information, and can indicate the order relationship between the edge information and the corresponding behavior edge.

[0049] Please refer to Figure 2 , which shows a flow chart of a method for displaying a usage status of a device provided by an exemplary embodiment of the present application. The method is executed by a computer device (smartphone, tablet computer, personal computer, etc.), and the method may include the following steps.

[0050] Step 201 : generating operation path information during the use of the machine, where the operation path information is used to represent at least one of an operation path between applications and an operation path within an application.

[0051] In a possible implementation, the computer device generates operation path information based on an operation trajectory model, wherein the operation trajectory model includes node information corresponding to a node in an interface in an application and edge information corresponding to a behavior edge of an operation behavior in the interface.

[0052] Accordingly, the computer device determines the interfaces passed through on the operation path and the operation behaviors performed on the operation path, thereby generating target path information of the target operation path based on the node information corresponding to the nodes passed through the interfaces and the edge information corresponding to the behavior edges of the performed operation behaviors.

[0053] In some embodiments, the edge information in the operation trajectory model includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of a starting node corresponding to a starting interface before the operation behavior, an ending node identifier (toNodeID) of a terminating node corresponding to a terminating interface after the operation behavior, and an edge identifier (a unique identifier of the edge).

[0054] Since the interface before and after the operation is performed can be determined based on the edge information, in some embodiments, the path information includes a behavior edge list. The behavior edge list contains a series of behavior edges corresponding to the operation behaviors, and the behavior edges are arranged in the time sequence of the operation behaviors. For example, the behavior edge at the head of the behavior edge list corresponds to the starting operation behavior in the operation path, and the behavior edge at the end of the list corresponds to the ending operation behavior in the operation path.

[0055] In one possible implementation, the computer device generates application-level operation path information based on applications, and the operation path information is used to characterize the operation path within the application; and / or the computer device generates device-level operation path information based on devices, and the operation path information is used to characterize the operation path between different applications in the device.

[0056] In some embodiments, the computer device periodically generates the operation path information, thereby obtaining the operation path information in different time units. For example, the computer device periodically generates the operation path information for each day in units of days.

[0057] Step 202: upon receiving a usage status display instruction, display a usage status display interface, wherein the usage status display interface includes a usage status display control.

[0058] In a possible implementation, the computer device provides a usage status display portal, and when receiving a trigger operation on the portal, the computer device determines that a usage status display instruction has been received, thereby displaying a usage status display interface. The usage status display portal may provide a portal for a system of the computer device.

[0059] Of course, the machine usage display instruction can also be triggered by voice, specified time or gesture, etc., which is not limited in this embodiment.

[0060] In the embodiment of the present application, the usage status display interface is an interface for displaying the usage status of computer devices. In some embodiments, the user status display interface provides the function of displaying the usage status according to different dimensions. For example, it provides the function of displaying the usage status according to the day or week, or it provides the function of displaying the usage status according to the application or device dimension.

[0061] In the embodiment of the present application, the machine usage display interface includes a usage display control, and the usage display control is used to trigger the display of the machine usage in the form of an operation path diagram.

[0062] In some embodiments, the usage display control is used to trigger the display of a device-level or application-level operation path map.

[0063] Indicatively, Figure 3 As shown, the computer device displays a global usage display control 311 and application usage display controls 312 corresponding to each application in the usage display interface 31.

[0064] Step 203, in response to a trigger operation on the usage display control, an operation path map is displayed based on the operation path information, and the operation path map includes at least one of a jump relationship between different applications and a jump relationship between different interfaces in the same application.

[0065] In a possible implementation, the computer device restores the jump relationship between different applications and / or between different interfaces in the same application based on the recorded operation path information to obtain an operation path map. The operation path map may be a global operation path map or an application operation path map. The specific generation method of the operation path map will be described in detail in the following embodiments.

[0066] Optionally, the operation path diagram includes interface icons and edge icons between interfaces. The interface icons may be interface screenshots, interface names, etc., and the edge icons are directional and used to indicate the order of jumps between interfaces, and the edge icons have the function of indicating the operation that triggers the interface jump.

[0067] Indicatively, Figure 3 As shown, when a trigger operation is received for the application usage display control 312 corresponding to application A, the computer device displays a usage display interface 32 in which an operation path diagram 321 of application A is displayed.

[0068] To summarize, in an embodiment of the present application, by recording the operation paths between applications and the operation paths within an application during the use of the machine, and generating operation path information, when the machine usage status is subsequently displayed, an operation path diagram representing the jump relationship between different applications and the jump relationship between different interfaces in the same application can be restored based on the operation path information, thereby facilitating users to review historical operations within and between applications and achieving a more fine-grained display of machine usage status.

[0069] Multi-granularity operation path information generation

[0070] In order to realize the global and application-dimensional display of the usage of the computer, the computer device needs to generate global operation path information and application operation path information, wherein the global operation path information is used to represent the operation path between different applications, and the application operation path information is used to represent the operation path within the application. Figure 4 As shown, the process of generating operation path information of different dimensions may include the following steps:

[0071] Step 201A: during the use of the machine, global operation path information is generated, and the global operation path information is used to represent the operation path between different applications.

[0072] In order to reflect the historical switching process between different applications during the use of the machine, the computer device is not limited to recording the operation path within the application in units of applications, but also needs to record the operation path between applications to obtain global operation path information. The global operation path information is used to characterize the operation path within the application and the operation path between different applications.

[0073] In a possible implementation, when the global usage display function is enabled, the computer device generates global operation path information during the use of the computer. When the global usage display function is not enabled, the computer device generates application operation path information at the application level during the use of the computer.

[0074] Optionally, generating global operation path information may include the following sub-steps.

[0075] Step 201A1, creating a global application, wherein all interfaces displayed during the use of the machine belong to the global application.

[0076] In order to realize the global jump relationship record, the computer device creates a global application. The global application is a virtual application, and its interface is all the interfaces displayed by the device during the use of the machine.

[0077] Indicatively, a special application type Global is added to the computer device. In the process of generating the operation path information, if the application package name of the current application is Global, it is determined that all interfaces displayed by the device belong to the current application (global application).

[0078] Step 201A2, during the use of the machine, determine the target node corresponding to the target interface involved in the operation path and the target behavior edge corresponding to the target operation behavior involved in the operation path from the operation trajectory model. The operation trajectory model includes the node information of the corresponding node of the interface in the application and the edge information of the behavior edge corresponding to the operation behavior in the interface.

[0079] The target interface is any interface on the operation path, that is, for each interface on the operation path, the computer device needs to determine the recorded node corresponding to the interface from the operation trajectory model.

[0080] Since node information is used in the operation trajectory model to represent the interface features of the interface, and interface feature matching is usually used to identify whether two interfaces are the same interface, the computer device determines the target node (recorded node) corresponding to the target interface based on the node information of the recorded node corresponding to the recorded interface in the operation trajectory model.

[0081] In some embodiments, the computer device generates node information corresponding to the target interface in the same manner as the recorded interface, thereby determining the target node corresponding to the target interface by comparing the node information of the recorded node corresponding to the recorded interface and the node information corresponding to the target interface.

[0082] The following first describes the node information:

[0083] Node Information

[0084] In some embodiments, the node information of a node includes at least a node identifier and a node factor of the node.

[0085] The node identifier is a unique identifier assigned to a node (with no actual meaning), which can be represented in the form of a string. Different nodes have different node identifiers, while the same node has the same node identifier.

[0086] Optionally, a node factor is composed of several fields that characterize interface features from different dimensions, wherein the fields that characterize interface features can be referred to as node sub-factors. When determining whether two nodes are the same node, the computer device matches the field contents with the same field name in the two node factors, that is, matches the node sub-factors in different node factors, thereby determining whether the two nodes are the same node based on the matching results of multiple dimensions.

[0087] In some embodiments, the node factor may include at least one of the following node sub-factors:

[0088] First node subfactor (windowName)

[0089] Since the same interface is definitely located in the same window, and interfaces located in different windows are definitely not the same interface, the computer device can determine whether two nodes are the same node based on the dimension of the window to which the interface belongs.

[0090] In a possible implementation, the computer device determines the first node sub-factor based on the window name of the window to which the interface belongs.

[0091] Among them, the window name of the window to which the interface belongs can be extracted from the interface code.

[0092] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the first node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, since different interfaces may be located in the same window, the computer device cannot directly determine that the two nodes with the same window name belong to the same node, but needs to make further determinations based on other node sub-factors.

[0093] Second node sub-factor (operableElements)

[0094] The interface usually includes a large number of operable elements, such as button controls, expandable list controls, text input box controls, upload controls, etc., and these operable elements can reflect the interface characteristics of the interface to a certain extent.

[0095] Therefore, the computer device can determine whether two nodes are the same node based on the dimension of the operable elements contained in the interface.

[0096] Affected by factors such as version updates or dynamic changes in content, the element content of the operable elements in the same interface may change (for example, the button display text of a button control changes, or the list item content in an expandable list control changes), but the element type of the operable elements in the same interface usually does not change. Therefore, in a possible implementation, the computer device determines the element type of the operable element, and performs information summary processing on the string obtained by splicing the element type, thereby determining the information summary result as the second node sub-factor. Among them, the element type of the operable element can be extracted from the interface code.

[0097] Among them, since the character string after the information digest is of fixed length, it is convenient to directly compare the information digest results later.

[0098] Optionally, the computer device performs MD5 processing on the string obtained by concatenating the element types, and determines the obtained string as the second node sub-factor.

[0099] Illustratively, when the interface includes 2 button controls, 1 text box control, and 3 radio selection controls, the computer device performs MD5 processing on the concatenated string Button_TextBox_RadioButton, and determines the obtained 64-bit string as the second node sub-factor.

[0100] Of course, in addition to being represented by a character string obtained from the information digest, the computer device may also determine an element type list (list) composed of element types as the second node sub-factor, which is not limited in this embodiment.

[0101] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the second node sub-factors in the node factors of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, in order to avoid judging two different simple interfaces (i.e., interfaces containing a small number of operable elements) as the same interface, the computer device needs to make further judgments based on other node sub-factors.

[0102] The third node subfactor (layoutVector)

[0103] Affected by factors such as version updates, the interface content of the same interface may change significantly, but the overall layout of the interface usually changes less. Therefore, the computer device can determine whether two nodes are the same node based on the dimension of interface layout.

[0104] In a possible implementation, the computer device determines the third node sub-factor based on the interface layout of the interface.

[0105] Optionally, in order to facilitate the subsequent comparison of interface layouts, the third node sub-factor is represented by a vector, wherein vector elements of different dimensions in the vector are used to represent different interface elements, and the vectors obtained by vectorizing the interface layouts of different interfaces have the same number of dimensions (for example, all are 512-dimensional vectors).

[0106] Regarding the vectorization method of the interface layout, in a possible implementation, the computer device extracts the interface layout information of the interface, performs vectorization processing on the interface layout information, and determines the obtained interface layout vector as the third node sub-factor. The interface layout information of the interface can be extracted from the interface code.

[0107] Optionally, the computer device generates an interface element skeleton based on the interface layout information, and the interface element skeleton is only used to characterize the position and size of different interface elements in the interface, and does not include the specific element content of the interface elements. Further, the computer device vectorizes the interface element skeleton to obtain an interface layout vector.

[0108] Indicatively, Figure 5 As shown, the computer device performs skeleton processing on the interface layout information of the interface 51 to obtain an interface element skeleton 52.

[0109] Fourth node sub-factor (screenVector)

[0110] In addition to determining different interfaces (nodes) from the dimension of interface layout, different interfaces can also be determined from a more intuitive visual level, that is, a computer device can determine whether two nodes are the same node from the dimension of the interface screen.

[0111] In a possible implementation, the computer device determines the fourth node sub-factor based on an interface screen of the interface.

[0112] In some embodiments, the computer device directly determines the interface screen as a fourth node sub-factor, and the fourth node sub-factor includes a pixel value of each pixel in the interface screen.

[0113] In order to reduce the amount of storage and simplify the amount of calculation in subsequent comparisons, in other embodiments, the computer device performs vector processing on the interface screen of the interface, and determines the obtained interface screen vector as the fourth node sub-factor. The number of dimensions of the interface screen vectors obtained by vector processing of different interface screens is consistent.

[0114] Regarding the method of vectorizing the interface screen, optionally, the computer device extracts image features of the interface screen through a neural network (such as a CNN network, a DNN network, etc.), and uses the extracted image feature vector as the fourth node sub-factor. Alternatively, the computer device can extract color features (such as obtained through a color histogram), texture features (such as obtained through texture analysis), and shape features (such as obtained through edge detection) of the interface screen, and obtain the fourth node sub-factor by vectorizing the color features, texture features, and shape features. The embodiments of the present application do not limit the specific method of screen vectorization.

[0115] Fifth node sub-factor (summaryFunction)

[0116] During the interface development phase, developers usually add interface function description information to the interface code. Even if the interface effect is subsequently updated, the interface function description information usually does not change. Therefore, the computer device can determine whether two nodes are the same node from the dimension of interface function.

[0117] In a possible implementation, the computer device determines the fifth node sub-factor based on interface function description information of the interface, wherein the interface function description information can be extracted from the interface code.

[0118] It should be noted that, since not all interfaces will add interface function description information, if the interface function description information cannot be obtained, the fifth node sub-factor may be empty.

[0119] Furthermore, when subsequently identifying whether two nodes are the same node, the computer device can compare whether the fifth node sub-factors in the node factors of the two nodes are the same. If they are the same, it is determined that the two nodes are the same node; if they are not the same, the computer device needs to make further determinations based on other node sub-factors.

[0120] In a possible implementation, the computer device may generate a node factor corresponding to the target interface, and compare the node factor with the node factor of the recorded node in the operation trajectory model, so as to determine the target node corresponding to the target interface. This step may include the following sub-steps:

[0121] Sub-step 1: Generate node factors corresponding to the target interface based on interface features of different dimensions of the target interface.

[0122] In a possible implementation, the computer device generates corresponding node sub-factors based on interface features of different dimensions of the target interface, and then determines the node factor corresponding to the target interface.

[0123] In some embodiments, the node factor corresponding to the target interface may include all or part of the above node sub-factors. For example, the computer device determines the first to fifth node sub-factors corresponding to the target interface.

[0124] Sub-step 2: determining a recorded interface that matches the target interface based on the node factors of the recorded nodes corresponding to the recorded interfaces in the operation trajectory model and the node factors corresponding to the target interface.

[0125] In a possible implementation, the computer device matches the node factors of the target interface with the node factors corresponding to the recorded interfaces in the operation trajectory model one by one, thereby determining the recorded interface that matches the target interface.

[0126] Optionally, the computer device may determine a recorded interface that matches the target interface based on a single node sub-factor in the node factor, or in combination with multiple node sub-factors in the node factor.

[0127] In some embodiments, when the node factor includes the first to fifth node sub-factors described above, the computer device may determine whether the target interface matches the recorded interface in the following cases.

[0128] Case 1: When the first node sub-factors corresponding to the target interface and the recorded interface are different, it is determined that the target interface does not match the recorded interface.

[0129] Since the same interface belongs to the same window, by comparing the first node sub-factors of the target interface and the recorded interface, and when the first node sub-factor of the target interface is different from the first node sub-factor of the recorded interface, the computer device determines that the target interface does not match the recorded interface.

[0130] When the first node sub-factor of the target interface is the same as the first node sub-factor of the recorded interface, the computer device needs to further determine whether the target interface matches the recorded interface in combination with other node sub-factors.

[0131] Optionally, when making subsequent determinations in combination with other node sub-factors, the computer device only needs to consider the recorded interfaces having the same first node sub-factor as the target interface, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0132] Case 2: When the second node sub-factors corresponding to the target interface and the recorded interface are different, it is determined that the target interface does not match the recorded interface.

[0133] Since the types of operable elements included in different interfaces are generally different, the computer device determines that the target interface does not match the recorded interface by comparing the second node sub-factors of the target interface and the recorded interface, and when the second node sub-factor of the target interface is different from the second node sub-factor of the recorded interface.

[0134] When the second node sub-factor of the target interface is the same as the second node sub-factor of the recorded interface (they may be two different interfaces containing very few operable elements), the computer device needs to combine other node sub-factors to further determine whether the target interface matches the recorded interface.

[0135] Optionally, when making subsequent determinations in combination with other node factors, the computer device only needs to consider the recorded interfaces that have the same second node sub-factor as the target interface, thereby narrowing the determination scope, reducing the amount of calculation in the determination process, and improving determination efficiency.

[0136] Case 3: When the fifth node sub-factors corresponding to the target interface and the recorded interface are the same, it is determined that the target interface matches the recorded interface.

[0137] If the functions of the two interfaces are completely the same, then the two interfaces should belong to the same interface. Therefore, in some embodiments, when the fifth node sub-factors of the target interface and the recorded interface are not empty, the computer device compares whether the fifth node sub-factors of the two are the same, and when the fifth node sub-factors of the two are the same, it is determined that the target interface matches the recorded interface.

[0138] Optionally, if at least one of the fifth node sub-factors of the target interface and the recorded interface is empty, or the fifth node sub-factors of the two are different, the computer device needs to combine other node sub-factors to further determine whether the target interface matches the recorded interface.

[0139] Case 4: When the first node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the second node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the fifth node sub-factor corresponding to the target interface is different from that of the recorded interface, and the similarity between the third node sub-factor corresponding to the target interface and the recorded interface is greater than the first similarity threshold, and / or the similarity between the fourth node sub-factor corresponding to the target interface and the recorded interface is greater than the second similarity threshold, it is determined that the target interface matches the recorded interface.

[0140] When it is impossible to determine whether the target interface matches the recorded interface based on the first node sub-factor, the second node sub-factor, and the fifth node sub-factor, since the same interface usually has a very similar interface layout and / or a very similar interface visual representation, the computer device can further combine at least one of the interface layout and the interface visual representation to determine whether the target interface matches the recorded interface.

[0141] Optionally, when the similarity between the interface layout of the target interface and the recorded interface, and / or the similarity between the interface visual presentation of the target interface and the recorded interface is greater than a threshold, the computer device determines that the target interface matches the recorded interface.

[0142] In some embodiments, when the third node sub-factor is represented by an interface layout vector, the computer device determines the vector distance between the target interface and the interface layout vector of the recorded interface as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiments of the present application.

[0143] In some embodiments, when the fourth node sub-factor is represented by an interface screen vector, the computer device determines the vector distance between the target interface and the interface screen vector of the recorded interface as the similarity. The vector distance can be a cosine distance, a Euclidean distance, a Manhattan distance, etc., which is not limited in the embodiment of the present application.

[0144] Indicatively, Figure 6 As shown, the computer device vectorizes the first interface element skeleton 61 corresponding to the target interface to obtain the first interface layout vector, and vectorizes the second interface element skeleton 62 corresponding to the recorded interface to obtain the second interface layout vector. By calculating the cosine distance between the first interface layout vector and the second interface layout vector, the interface layout similarity between the target interface and the recorded interface is 0.42490305.

[0145] Case 5: When the first node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the second node sub-factor corresponding to the target interface is the same as that of the recorded interface, and the fifth node sub-factor corresponding to the target interface is different from that of the recorded interface, and the similarity between the third node sub-factor corresponding to the target interface and the recorded interface is less than the first similarity threshold, and the similarity between the fourth node sub-factor corresponding to the target interface and the recorded interface is less than the second similarity threshold, it is determined that the target interface does not match the recorded interface.

[0146] Optionally, when the similarity between the interface layout of the target interface and the recorded interface, and the similarity between the interface visual presentation of the target interface and the recorded interface are both less than a threshold, the computer device determines that the target interface does not match the recorded interface.

[0147] Sub-step 3: determining the recorded node corresponding to the matched recorded interface as the target node corresponding to the target interface.

[0148] In some embodiments, when a single recorded interface is matched, the computer device determines the recorded node corresponding to the recorded interface as the target node corresponding to the target interface; when multiple recorded interfaces are matched, the computer device determines the recorded node corresponding to the recorded interface with the highest matching degree as the target node corresponding to the target interface.

[0149] The target operation behavior is any operation behavior on the operation path. That is, for each operation behavior on the operation path, the computer device needs to determine the behavior edge corresponding to the operation behavior from the operation trajectory model.

[0150] In some embodiments, the operation trajectory model records the edge information of the behavior edge corresponding to the full amount of operation behaviors. In the process of generating the operation path information, the computer device determines the target behavior edge corresponding to the target operation behavior according to the edge information corresponding to the target operation behavior and the edge information of the recorded operation behavior corresponding to the recorded behavior edge.

[0151] The side information in the operation trajectory model is explained below.

[0152] Side Information

[0153] Since it accurately describes what operation is performed in the interface and the changes in the interface before and after the operation is performed, the edge information corresponding to the operation behavior at least has the ability to indicate the operation behavior and the interface before and after the operation behavior is performed.

[0154] In some embodiments, the edge information includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of the starting node corresponding to the starting interface before the operation behavior, a terminating node identifier (toNodeID) of the terminating node corresponding to the terminating interface after the operation behavior, and an edge identifier (a unique identifier similar to a node identifier).

[0155] Among them, when the interface before and after the operation behavior does not switch, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are the same; when the interface before and after the operation behavior switches, the starting node identifier and the ending node identifier in the side information corresponding to the operation behavior are different.

[0156] As an important content in the side information, the behavior information needs to be able to clearly indicate the operation behavior. In a possible implementation, the behavior information includes the action type (ActionType) of the operation behavior.

[0157] In some embodiments, a common action class (CommonAction) is defined, which includes all operation actions of users and systems, so as to ensure the accuracy of recorded operation actions. In addition, the behavior type of each operation behavior belongs to the behavior type class. Optionally, the behavior type class is an enumeration class.

[0158] In an illustrative example, taking the commonly used operations in a mobile phone as an example, the behavior type class may include at least one of the following behavior types: GoHome (return to the home page), RecentApps (recent applications), GoBack (return), StartApp (start application), Click (click), DoubleClick (double-click), LongClick (long press), InputText (input text), Scroll (scroll), Concurrent (collaborative operation, that is, operation behaviors that occur at the same time), Continuous (continuous operation, referring to operation behaviors performed in sequence), Stop (terminate), Wait (wait, a scenario in which the interface changes automatically over time), and Unknown (unknown behavior, as a fallback behavior).

[0159] Since some operation behaviors are directed to interface controls within the interface, in order to ensure that the subsequent operation can determine which interface control to operate on based on the behavior information, in some embodiments, when the behavior type of the operation behavior is determined as behavior information, if the operation behavior is related to the interface control within the interface, the computer device also determines the node element identifier of the node element corresponding to the interface control as the behavior information.

[0160] For example, when the operation behavior is a click operation, the behavior information corresponding to the operation behavior includes the operation type "Click" and the node element identifier of the node element corresponding to the clicked button; for another example, when the operation behavior is a text input operation, the behavior information corresponding to the operation behavior includes the operation type "InputText" and the node element identifier of the node element corresponding to the text input box where the text is entered.

[0161] In a possible implementation, determining the target behavior edge may include the following sub-steps:

[0162] Sub-step 1: Determine, from the operation trajectory model, a starting node corresponding to the starting interface before the target operation behavior and an ending node corresponding to the ending interface after the target operation behavior.

[0163] In one possible implementation, the computer device determines the starting node corresponding to the starting interface based on the node information corresponding to the starting interface, and obtains the starting node identifier of the starting node; determines the terminating node corresponding to the terminating interface based on the node information corresponding to the terminating interface, and obtains the terminating node identifier of the terminating node.

[0164] In some embodiments, in order to ensure the accuracy of the determined termination node, when the duration of the target operation behavior is received and reaches a preset duration (because there may be a response delay in the interface jump), the computer device determines the current interface as the termination interface, and then determines the termination node corresponding to the termination interface.

[0165] Sub-step 2: when the target operation behavior is related to the target interface control, determine the node element corresponding to the target interface control from the operation trajectory model.

[0166] In some embodiments, the computer device determines whether the target operation behavior is related to the target interface control. If the target operation behavior is related to the target interface control (i.e., the target operation behavior acts on the target interface control), the computer device determines the node element corresponding to the target interface control. If the target operation behavior is not related to the target interface control (for example, the target operation behavior is the system automatic interface jump behavior after reaching a specified time length), the computer device does not need to determine the node element.

[0167] In a possible implementation, the operation trajectory model includes node element information of node elements corresponding to each recorded interface control in the recorded interface. Accordingly, the computer device determines the node element information corresponding to the target interface control, and matches the node element information with the node element information of the node element corresponding to the recorded interface control, thereby determining the matched recorded node element as the node element corresponding to the target interface control.

[0168] In some embodiments, the computer device determines a node element factor corresponding to the target interface control (which is part of the node element information and is used to characterize the characteristics of the node element from multiple dimensions), and thereby determines the node element corresponding to the target interface control based on the degree of matching between the node element factor and the node element factor corresponding to the recorded interface control.

[0169] Sub-step 3: determining a target behavior edge corresponding to the target operation behavior based on the start node, the end node, the node element, and the target behavior type of the target operation behavior.

[0170] In one possible implementation, the computer device matches the starting node identifier of the starting node, the ending node identifier of the ending node, the node element identifier of the node element, and the target behavior type of the target operation behavior with the edge information of the recorded behavior edge corresponding to the recorded operation behavior in the operation trajectory model, thereby determining the matched recorded behavior edge as the target behavior edge corresponding to the target operation behavior.

[0171] Step 201A3: Generate global operation path information corresponding to the global application based on the target node information of the target node and the target edge information of the target behavior edge.

[0172] After determining each target node passed through on the operation path and the target behavior edges between each target node from the operation trajectory model, the computer device can generate the path information of the operation path. In addition, since the operation path is for global applications, including each interface during the use of the device, the path information is global operation path information. Based on the global operation path information, the computer device can determine each node passed through on the operation path and the behavior edges between the nodes.

[0173] Step 201B, split the global operation path information according to the application dimension to obtain application operation path information of different applications. The application operation path is used to represent the operation path within the application.

[0174] In order to facilitate the subsequent preview of the operation path of each application, that is, to realize the usage viewing at the application dimension level, the computer device also needs to generate the application operation path information of each application.

[0175] Since the global operation path information indicating the operation paths within each application and the operation paths between different applications has been obtained through the above step 201A, the computer device can obtain the application operation path information corresponding to different applications by splitting the global operation path information.

[0176] Since the interfaces before and after the jump within the same application belong to the same application, while the interfaces before and after the jump between different applications belong to different applications, and the target behavior edge in the global operation path information has the function of indicating the starting node and the ending node, the computer device can split the global operation path information by traversing the edge information of the target behavior edge in the global operation path information to obtain the application operation path information of different applications.

[0177] In a possible implementation, this step may include the following sub-steps:

[0178] Step 201B1: for each target behavior edge in the global operation path information, determine the start node and the end node corresponding to the target behavior edge.

[0179] In a possible implementation, the computer device traverses the behavior edge information corresponding to each target behavior edge in the global operation path information to determine the start node and the end node corresponding to each target behavior edge. The computer device may obtain the start node identifier (fromNodeID) and the end node identifier (toNodeID) from the behavior edge information.

[0180] Furthermore, the computer device determines the applications to which the start node and the end node respectively belong, and determines whether the applications to which the start node and the end node belong are the same application.

[0181] In some embodiments, the computer device determines the application to which the start node belongs based on the start node identifier, and determines the application to which the end node belongs based on the end node identifier.

[0182] Since the interface of a computer device is usually represented by an activity, in actual scenarios, there are situations where a single activity corresponds to multiple fragments, or a single activity corresponds to multiple pages. In this case, it is difficult to distinguish the two interfaces based on the name of the activity, so it is impossible to distinguish the nodes directly based on the name of the activity.

[0183] In order to solve this problem, in the embodiments of the present application, based on the concept of node, the concept of node group (NodeGroup) is extended.

[0184] In some embodiments, the node group corresponds to the activity at the code level, that is, different activities correspond to different node groups. The node group is composed of at least one node, and the interface of the node at the visual level corresponds.

[0185] In a possible implementation, when generating the operation trajectory model, the computer device determines node group information of the node group based on the node information, and generates the operation trajectory model based on the node group information, the edge information, and the path information.

[0186] In order to enable the node group information to have the ability to characterize each node included in the node group and distinguish different node groups, the node group information at least includes a node list and a node group factor (NodeGroupFactor).

[0187] The node list contains nodes belonging to a node group, and the node group factor is used for the node group characteristics of the node group.

[0188] In a possible implementation, when determining the node group information, the computer device determines the node group factor of the node group, and based on the node information, determines the node list (childNode) of the node group, thereby determining the node group factor and the node list as the node group information of the node group.

[0189] Optionally, the computer device groups nodes having the same activity name into the same node group.

[0190] In a possible implementation, at the code level, the node group is uniquely indicated by a combination of an application package name + an activity name, that is, the node group factor includes the application package name (packageName) and the activity name (activityName) of the application.

[0191] In another possible implementation, at the code level, the node group is uniquely indicated by a combination of application package name+activity name+activity title, that is, the node group factor includes the application package name (packageName), activity name (activityName) and activity title (activityTitle) of the application.

[0192] When determining the application to which the starting node belongs based on the starting node identifier, the computer device determines the node group to which the starting node belongs based on the starting node identifier, and determines the application to which the starting node belongs based on the application package name in the node group factor corresponding to the node group; when determining the application to which the terminating node belongs based on the terminating node identifier, the computer device determines the node group to which the terminating node belongs based on the terminating node identifier, and determines the application to which the terminating node belongs based on the application package name in the node group factor corresponding to the node group.

[0193] Step 201B2: when the start interface corresponding to the start node belongs to the first application and the end interface corresponding to the end node belongs to the first application, add the target behavior edge to the first application operation path information corresponding to the first application.

[0194] When the starting interface corresponding to the starting node belongs to the first application, and the terminating interface corresponding to the terminating node belongs to the first application, it indicates that the interfaces corresponding to the starting node and the terminating node belong to the same application. The computer device determines that the target behavior edge is a behavior edge between interfaces within the first application, thereby adding the target behavior edge to the first application operation path information corresponding to the first application.

[0195] Step 201B3, when the starting interface corresponding to the starting node belongs to the first application and the terminating interface corresponding to the terminating node belongs to the second application, add the target behavior edge to the first application operation path information corresponding to the first application, and add the target behavior edge to the second application operation path information corresponding to the second application.

[0196] When the starting interface corresponding to the starting node belongs to the first application, and the terminating interface corresponding to the terminating node belongs to the second application, it indicates that the interfaces corresponding to the starting node and the terminating node belong to different applications. The computer device determines that the target behavior edge is a behavior edge between interfaces of different applications, thereby adding the target behavior edge to the application operation path information corresponding to the first application and the second application respectively.

[0197] Optionally, the computer device adds the target behavior edge to the first application operation path information corresponding to the first application, so that the first application operation path information can indicate a jump from the first application to the second application; and adds the target behavior edge to the second application operation path information corresponding to the second application, so that the second application operation path information can indicate that the second application is obtained by jumping from the first application.

[0198] In some embodiments, the computer device stores the generated global operation path information and the application operation path information of different applications in the directories corresponding to the respective application package names. For example, the global operation path information is stored in the directory corresponding to the global application package name, and the application operation path information is stored in the directory corresponding to the application package name.

[0199] Among them, when storing application-related files such as interface screenshots, the computer device will store the files in the directory corresponding to the application package name, while such files will not be stored in the directory corresponding to the global application package name to avoid duplicate storage of files.

[0200] In this embodiment, the computer device creates a global application and determines all interfaces displayed by the device as interfaces of the global application, thereby generating device-level global operation path information for subsequent device-level usage display; and the computer device splits the global operation path information into application operation path information corresponding to different applications to meet subsequent application-level usage display requirements.

[0201] Multi-granularity usage display

[0202] When global operation path information and application operation path information are generated, the computer device can display global usage display controls and application usage display controls corresponding to different applications in the usage display interface to meet usage display requirements of different granularities.

[0203] In some embodiments, in response to a trigger operation on a global usage display control, a computer device displays a global operation path map based on global operation path information, where the global operation path map includes interfaces of different applications and jump relationships between the interfaces.

[0204] Indicatively, in Figure 3 On the basis of Figure 7 As shown, when a trigger operation of a global application usage display control 311 corresponding to a global application is received, the computer device displays a usage display interface 32 in which a global operation path diagram 322 is displayed.

[0205] The generation process of the global operation path graph may include two stages: interface diagram and edge diagram. The interface diagram is used to represent the interface on the operation path, and the edge diagram is used to represent the jump direction between the interfaces.

[0206] In a possible implementation, the generation of the global operation path graph may include the following steps:

[0207] 1. Based on the target node information in the global operation path information, an interface diagram in the global operation path diagram is generated. The target node information is the node information of the target node corresponding to the target interface involved in the operation path.

[0208] Optionally, for each target node on the global operation path, the computer device generates an interface diagram of the target interface corresponding to the target node based on the target node information of the target node, wherein the interface diagram may be an interface screenshot or text information for indicating the interface.

[0209] In some embodiments, the computer device determines the interface screenshot obtained when generating the fourth node sub-factor in the target node information as the interface icon of the target interface corresponding to the target node. Alternatively, the computer device generates an interface icon based on the window name indicated by the first node sub-factor or the interface function description indicated by the fifth node sub-factor in the target node information, and the interface icon is composed of an interface frame and text information.

[0210] Of course, the interface diagram generated by the computer device based on the target node information can also adopt other forms of expression, such as preset interface diagrams set for different interfaces, or interface skeleton diagrams, etc., which are not limited in the embodiments of the present application.

[0211] 2. Based on the target edge information in the global operation path information, an edge diagram connecting the interface diagrams in the global operation path diagram is generated. The edge diagram is used to indicate the jump direction between interfaces. The target edge information is the edge information of the target behavior edge corresponding to the target operation behavior involved in the operation path.

[0212] Optionally, for each target behavior edge on the global operation track, the computer device generates an edge diagram corresponding to the target behavior edge based on the target edge information of the target behavior edge, wherein the edge diagram may be a directed connection line connecting interface diagrams (the same interface diagram or different interface diagrams).

[0213] The jump direction represented by the edge diagram is determined based on the start node and the end node in the target edge information, and the edge diagram points from the interface diagram corresponding to the start node to the interface diagram corresponding to the end node.

[0214] In some embodiments, in order to enable subsequent backtracking to know the operation performed when triggering the interface jump, the edge diagram may also include the behavior type of the operation behavior in the target edge information, and / or the control information of the triggered interface control.

[0215] In response to a trigger operation on an application usage display control corresponding to a target application, a computer device displays a target application operation path map of the target application based on the application operation path information of the target application. The target application operation path map includes interfaces in the target application and jump relationships between interfaces.

[0216] The generation process of the application operation path map may refer to the generation process of the global operation path map described above, the difference being that the application operation path information of the target application is used when generating the application operation path map, which will not be elaborated in this embodiment.

[0217] Optionally, the above-mentioned process of generating a global / application operation path map can be pre-executed after the global / application operation path information is generated, and when a trigger operation for a global / application usage display control is subsequently received, the computer device directly displays it, or the above-mentioned process of generating a global / application operation path map can be generated in real time upon receiving a trigger operation for a global / application usage display control.

[0218] In this embodiment, the computer device generates a global operation path map based on the global operation path information, and generates an application operation path map based on the application operation path information of different applications, and provides the user with the option to view it, thereby satisfying the user's viewing needs for machine usage at different granularities.

[0219] Backtracking function

[0220] The computer device can implement a backtracking function based on the operation path information, so that the user can backtrack specific content in the application. In a possible implementation, the machine usage display interface also includes a backtracking control, which is used to trigger content backtracking. Figure 8 As shown, the implementation process of the backtracking function includes the following steps:

[0221] Step 801, in response to a triggering operation on a backtracking control, obtaining content to be backtracked.

[0222] In a possible implementation, the backtracking control is a global backtracking control for triggering content backtracking of all interfaces displayed on the device; or, the backtracking control is an application backtracking control for triggering content backtracking of an interface corresponding to a specific application.

[0223] Indicatively, Fig. 9 As shown, the machine usage display interface 91 displays a global backtracking control 911 and application backtracking controls 912 corresponding to different applications.

[0224] Optionally, the content to be retraced may be at least one of text content, voice content, picture content or multimedia content. The text content may be the file name, natural language description, etc. of the content to be retraced; the picture content may be a sketch of the content to be retraced, etc., which will not be described in detail in this embodiment.

[0225] Indicatively, Fig. 9 As shown, when a click operation on the application backtracking control 912 corresponding to the A application is received, the computer device displays a content backtracking interface 92 and obtains the to-be-backtracked content 921 input by the user.

[0226] Step 802 : determining a backtracking interface including the content to be backtracked based on the content to be backtracked and the interface content information corresponding to the interface in the application operation path information, wherein the interface content information is used to represent the content included in the interface.

[0227] In a possible implementation, in order to facilitate backtracking, when generating the operation path information, the computer device associates and stores the interface with the interface content information representing the content contained in the interface. When performing content backtracking, the computer device matches the content to be backtracked with the interface content information, thereby determining the backtracking interface containing the content to be backtracked.

[0228] Optionally, when performing global content backtracking, the computer device matches the content to be backtracked with the interface content information corresponding to each application; when performing application content backtracking, the computer device matches the content to be backtracked with the interface content information corresponding to the specified application.

[0229] Optionally, when a trigger operation on a global backtracking control is received and the application to which the content to be backtracked belongs cannot be determined based on the content to be backtracked, the computer device performs global content backtracking; or, when a trigger operation on an application backtracking control for a specific application is received and the backtracking interface containing the content to be backtracked is not determined, the computer device re-performs global content backtracking.

[0230] When a trigger operation for a global backtracking control is received and the application to which the content to be backtracked belongs can be determined based on the content to be backtracked, the computer device backtracks the application content; or, when a trigger operation for an application backtracking control for a specific application is received, the computer device backtracks the application content.

[0231] Optionally, the interface content information includes at least one of the following:

[0232] 1. The text contained in the interface;

[0233] In some embodiments, the computer device extracts text from the interface and stores the text in association with the interface.

[0234] 2. Picture description information of pictures in the interface;

[0235] In order to find the interface containing the image according to the user's description of the image, the computer device generates image description information of the image in the interface. The image description information is generated with the help of a large language model, which is not described in detail in this embodiment.

[0236] 3. A summary of the content contained in the interface;

[0237] In some embodiments, when the interface includes multiple contents, the computer device can generate a content summary of the contents included in the interface through a large language model, wherein the content summary can be expressed in text form.

[0238] 4. Content embedding representation of the content contained in the interface;

[0239] In order to achieve more accurate tracing, the computer device extracts features of the content contained in the interface to obtain a content embedding representation of the content. Optionally, for different types of content, the computer device uses different feature extraction networks to extract features and uses the extracted content features as a content embedding representation.

[0240] 5. Named entities extracted from the interface.

[0241] Since users usually trace back named entities in an interface, in a possible implementation, a computer device performs named entity recognition (NER) on the interface to obtain named entities contained in the interface.

[0242] Of course, in addition to extracting the above-mentioned interface content information from the interface, the computer device may also use other methods to extract other forms or types of interface content information from the interface, and the embodiments of the present application do not constitute a limitation to this.

[0243] In an illustrative example, the correspondence between the interface and the interface content information stored in the computer device is shown in Table 1.

[0244] Table 1

[0245]

[0246] In some embodiments, a computer device may store interface content information for a period of history. Accordingly, in order to improve the accuracy of content backtracking, when the content to be backtracked contains time information, the computer device determines application operation path information that matches the time information, thereby determining a backtracking interface containing the content to be backtracked based on the content to be backtracked and the interface content information corresponding to the interface in the application operation path information, thereby ensuring the timeliness of the interface obtained by backtracking.

[0247] Combination Fig. 9 As shown in Table 1, the computer device obtains the application operation path information of application A generated yesterday based on the time information "yesterday" included in the content to be traced back 921. Based on the interface content information and the content to be traced back 921, the computer device determines the interface corresponding to the sports news node as the traceback interface.

[0248] Step 803, jump to the backtracking interface.

[0249] In order to reproduce the content to be backtracked, the computer device automatically jumps to the backtracking interface. In a possible implementation, automatically jumping to the backtracking interface may include the following steps:

[0250] Step 803A, based on the operation trajectory model, determine the operation path to jump to the backtracking interface. The operation trajectory model includes node information corresponding to the node of the interface in the application and edge information corresponding to the behavior edge of the operation behavior in the interface. The operation path is composed of nodes and behavior edges between nodes.

[0251] Since the edge information in the operation trajectory model is used to characterize the jump relationship between different nodes, the computer device can determine the operation path to jump to the backtracking interface by using a path search algorithm, with the backtracking node as the termination node, when determining the backtracking node corresponding to the backtracking interface. The path search algorithm can be a depth-first search, a breadth-first search algorithm, Dijkstra, an A* algorithm, a D* algorithm, etc., which is not limited in the embodiments of the present application.

[0252] Optionally, the operation path to jump to the backtracking interface may be an operation path from a specified interface to the backtracking interface, or an operation path from a current interface to the backtracking interface, that is, the starting interface of the operation path may be an interface corresponding to the specified interface, or an interface corresponding to the current interface. For example, the specified interface may be a launcher interface.

[0253] Step 803B, based on the path information corresponding to the operation path, jump to the backtracking interface by simulating the operation behavior, the path information includes edge information corresponding to the behavior edge between nodes on the operation path, and the edge information includes behavior information indicating the operation behavior.

[0254] In one possible implementation, when the path information includes a behavior edge list, the computer device sequentially obtains the edge information of the behavior edge from the behavior edge list, and then determines the starting interface before the operation behavior is executed, the behavior information of the operation behavior, and the termination interface after the operation behavior is executed from the operation trajectory model based on the edge information.

[0255] In some embodiments, when the computer device determines that the current interface is the starting interface, it simulates the operation behavior based on the behavior information of the operation behavior, and detects whether to jump to the termination interface (backtracking interface) after the operation behavior is executed.

[0256] In a possible implementation, the computer device determines the edge information corresponding to the behavior edge between nodes on the operation path based on the path information, and then simulates the operation behavior indicated by the behavior information in the interface based on the behavior information included in the edge information.

[0257] In one possible implementation, in order to ensure the timing accuracy of the operation behavior, the computer device determines the edge information corresponding to the behavior edges between the nodes on the operation path in sequence according to the execution timing of the operation behavior indicated by the edge information in the path information, thereby simulating the operation according to the target operation behavior indicated by the edge information.

[0258] The operation behavior in the interface can be a manual operation behavior of the user, such as a trigger operation on a control in the interface; or the operation behavior is a system behavior, such as an automatic interface jump operation after reaching a preset time. Accordingly, the target side information includes target behavior information, which is used to indicate the operation behavior in the interface.

[0259] In some embodiments, when the operation behavior indicated by the behavior information is related to the interface control, the behavior information includes the node element identifier of the node element corresponding to the interface control and the behavior type of the operation behavior.

[0260] The node element information is described below.

[0261] Node element information

[0262] In some embodiments, the node element information of the node element includes at least a node element identifier and a node element factor of the node element.

[0263] The node element identifier is a unique identifier assigned to the node element (with no actual meaning), which can be expressed in the form of a string. Different node elements have different node element identifiers.

[0264] Optionally, the node element factor (NodeElement Factor) is composed of several fields that characterize the control features of the interface control from different dimensions, wherein the fields that characterize the control features can be called node element sub-factors. Similar to the role of the node factor, the role of the node element factor is to retrieve and locate the interface control of the feature in the interface.

[0265] When determining whether two node elements are the same node element (i.e., determining whether two interface controls are the same interface control), the computer device matches the field contents with the same field name in the two node element factors, that is, matches the node element sub-factors in different node element factors, thereby determining whether the two node elements are the same node element based on the matching results of multiple dimensions.

[0266] In some embodiments, the node element factor may include at least one of the following node element sub-factors:

[0267] First node element subfactor (packageName)

[0268] In a possible implementation, the computer device determines the first node element sub-factor based on the package name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control package name dimension.

[0269] Optionally, the computer device obtains the package name of the control package referenced by the interface from the interface code, and thereby determines the package name of the control class to which the interface control belongs based on the package name of the control package.

[0270] In an illustrative example, the computer device determines the package name android.widget of the control class as the first node element sub-factor based on the package name of the widget package referenced by the interface.

[0271] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the first node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control package, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0272] Second node element sub-factor (className)

[0273] In a possible implementation, the computer device determines the second node element sub-factor based on the class name of the control class to which the interface control belongs, that is, describes the control feature of the interface control from the control class name dimension.

[0274] Optionally, the computer device obtains the control code of the interface control and the package name of the control package referenced by the interface from the interface code, thereby determining the package name of the control class to which the interface control belongs based on the control code and the control package name.

[0275] In an illustrative example, the computer device determines that the class name of the control class in which the text box control in the widget is located, android.widget.TextView, is determined as the second node element sub-factor.

[0276] Further, when subsequently identifying whether two interface controls are the same interface control, the computer device can compare whether the second node element sub-factors corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control class at the same time, the computer device cannot directly determine that the two interface controls are the same interface control, but needs to make further determinations based on other node element sub-factors.

[0277] Third node element sub-factor (ResourceId)

[0278] The resources of the interface control can be layout, string, color, image, etc., and each interface control is assigned a unique ID, namely, resource ID, when it is defined in the layout file. The interface control can be referenced or operated in the code through the resource ID. Since the resource ID of the interface control is unique and generally does not change (it may be updated during the version iteration process), in a possible implementation, the computer device determines the third node element sub-factor based on the resource identifier corresponding to the interface control, that is, describes the control characteristics of the interface control from the resource dimension.

[0279] Optionally, the computer device may start from the interface root node of the interface, traverse each control under each interface root node, and obtain the resource ID of each interface control by means of code printing.

[0280] In an illustrative example, the computer device determines the resource ID "com.phonebrand.someapp:id / tv_name" of the interface control as the third node element sub-factor.

[0281] Fourth node element sub-factor (contentDescription)

[0282] The interface code may include a control description of the interface control (used to describe the control content, usually used for barrier-free functions). For example, for a text box control, the interface code may contain the following control description: text box, text content xxx. Since the control description generally does not change (it may be manually adjusted during version iteration), in a possible implementation, the computer device determines the fourth node element sub-factor based on the control description corresponding to the interface control, that is, describes the control characteristics of the interface control from the control description dimension.

[0283] Optionally, the computer device may extract a control description corresponding to the interface control from the interface code, wherein the control description of the interface control may be empty.

[0284] Fifth node element sub-factor (Text)

[0285] Some interface controls in the interface usually display text so that users can know the purpose of the interface control, such as the word "Upload" displayed on the upload button. Since the text content corresponding to the interface control usually does not change, in a possible implementation, the computer device determines the fifth node element sub-factor based on the text content corresponding to the interface control, that is, describes the control feature of the interface control from the dimension of the control's explicit text content.

[0286] Optionally, the computer device may traverse each control under the root node of the interface, and when the text attribute field of the control is not empty, determine the text content of the text attribute field as the fifth node element sub-factor.

[0287] Optionally, the computer device may also recognize the text displayed by the interface control through OCR, and use the recognized text as the text content corresponding to the interface control, which is not limited in this embodiment.

[0288] Sixth node element subfactor (listOrder)

[0289] For an interface control located in a list, the text content of the same interface control may change dynamically (for example, it may change dynamically with the time period or scene), but its absolute position or relative position in the list will not change. Therefore, in order to improve the comprehensiveness of the node element information and improve the accuracy of the subsequent positioning of the interface control, in a possible implementation, when the interface control is located in a list control, the computer device determines the sixth node element sub-factor based on the position of the interface control in the list control.

[0290] Optionally, the position of the interface control in the list control can be an absolute position, such as being located at the first position in the list, or being located at the last position in the list. The position can be a relative position, such as being located between two interface controls in the list control, or being located after an interface control in the list control.

[0291] Optionally, when a list control is identified in the interface code, the computer device may determine the absolute position or relative position of the interface control based on the arrangement order or rule of each interface control in the list control.

[0292] Seventh node element subfactor (parentChain)

[0293] Each interface control in the interface is in a control chain, where the upper interface control in the control chain is the parent control of the lower interface control, and the top interface control in the control chain is the root control corresponding to the root node of the interface. When the text content, position, size and other parameters of the interface control are fine-tuned, as long as the control function implemented by the interface control remains unchanged, the control chain to which it belongs will usually not change. Therefore, computer devices can describe interface controls from the dimension of the control chain corresponding to the interface control.

[0294] In a possible implementation, the computer device determines the seventh node element sub-factor based on the parent control chain corresponding to the interface control.

[0295] Among them, the parent control chain is composed of the parent controls of the interface controls, and the end point of the parent control chain is the root control corresponding to the interface root node, that is, the first and last controls of the parent control chain of the current interface control are the root control of the interface root node and the current interface control respectively.

[0296] Optionally, the computer device may determine the parent-child relationship between each interface control based on the interface code, and thereby construct a parent control chain corresponding to each interface control based on the parent-child relationship.

[0297] Eighth node element sub-factor (preNodeElementFactor & postNodeElementFactor)

[0298] In order to further improve the comprehensiveness of the control features, the current interface control may be described with the help of other interface controls surrounding the current interface control. In a possible implementation, the computer device determines the eighth node element sub-factor based on the adjacent interface controls of the interface control.

[0299] Optionally, the adjacent interface control may include other interface controls that are adjacent to the current interface control in at least one direction. For example, the adjacent interface control may include a left interface control and a right interface control of the current interface control.

[0300] Optionally, the computer device determines the relative position relationship between the interface controls based on the control layout information corresponding to each interface control in the interface code, or the computer device identifies the interface screen to determine the relative position relationship between the interface controls.

[0301] Optionally, the computer device may determine the node element factor corresponding to the adjacent interface control as the eighth node element sub-factor of the current interface control. For example, the eighth node element sub-factor of the current interface control includes the node tuple factor (preNodeElementFactor) of the left interface control and the node tuple factor (postNodeElementFactor) of the right interface control.

[0302] Regarding the timing of determining the above node element sub-factors, in a possible implementation manner, the above various node element sub-factors are all determined in real time.

[0303] In another possible implementation, the first to fifth node element sub-factors are determined in real time. Since the sixth to eighth node element sub-factors cannot be directly extracted from the interface code but need to be analyzed by certain calculations, the sixth to eighth node element sub-factors are determined by lazy count.

[0304] Among them, lazy calculation means that node element sub-factor calculation is performed only when node element sub-factor comparison is required. For example, when the node element cannot be uniquely located based on the first to fifth node element sub-factors, the computer device calculates the sixth to eighth node element sub-factors.

[0305] Optionally, when the sixth to eighth node element sub-factors adopt lazy calculation, the sixth to eighth node element sub-factors will not participate in serialization, and only the first to fifth node element sub-factors participate in serialization.

[0306] In a possible implementation, when the sixth to eighth node element sub-factors are calculated in a lazy manner, the computer device stores the interface code so as to subsequently calculate the sixth to eighth node element sub-factors based on the interface code analysis.

[0307] In some embodiments, the node element information may further include at least one of the following:

[0308] Control properties

[0309] Optionally, the control attributes include the type, purpose, and coordinates of the interface control.

[0310] In some embodiments, the control properties may include at least one of the following: stateDescription (control state description), isPassword (whether it is a password input control), isHintText (whether it is a hint text), visible (visibility), checkable (whether it can be checked), editable (whether it can be edited), clickable (whether it can be clicked), scrollable (whether it can be scrolled), longClickable (whether it can be long pressed), focusable (whether it can get focus), enabled (whether it is available), focused (whether it is the focus), checked (whether it is checked), selected (whether it is selected) and bounds (coordinate range).

[0311] Of course, the node element information may also include other control attributes other than the above examples, which is not limited in this embodiment.

[0312] Child node element identifier (children)

[0313] In a possible implementation, the computer device determines a sub-control of the current interface control, and determines a sub-node element identifier of a sub-node element corresponding to the sub-control as the node element information of the current interface control.

[0314] Optionally, the computer device uses the child node element identifier of the direct child control of the current interface control as the node element information, wherein the direct child control refers to a lower-level child control directly connected to the current interface control in a parent-child relationship.

[0315] Optionally, the child node element identifiers can be in list form.

[0316] Parent node element identifier (parentNodeElementId)

[0317] In a possible implementation, the computer device determines the parent control of the current interface control, and determines the parent node element identifier of the child node element corresponding to the parent control as the node element information of the current interface control.

[0318] Optionally, the computer device uses the child node element identifier of the direct parent control of the current interface control as the node element information, wherein the direct parent control refers to the upper parent control directly connected to the current interface control in the parent-child relationship.

[0319] Since the interface controls in the application may be fine-tuned after the version update, or the interface controls may change slightly in different scenarios, in order to ensure the accuracy of the operation of the interface controls, in a possible implementation, when the computer device simulates the operation behavior of the interface controls in the interface, it is necessary to determine the node element factor of the node element based on the node element identifier of the node element, and then locate the interface control from the interface based on the node element factor and the node element factor corresponding to each actual interface control in the interface. Optionally, this step may include the following sub-steps:

[0320] Sub-step 1: based on the node element identifier in the behavior information, obtain the node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize the control features of the interface control from different dimensions.

[0321] In some embodiments, the computer device obtains the node element factor of the node element corresponding to the node element identifier from the operation trajectory model based on the node element identifier. Optionally, the node element factor may include one or more of the first to eighth node element sub-factors mentioned above.

[0322] Sub-step 2: determining the interface controls from the interface based on the node element factor and the node element factor corresponding to each interface control in the interface.

[0323] Before performing node element factor comparison, the computer device needs to generate node element factors corresponding to each interface control in the interface. The generated node element factors may include one or more of the first to eighth node element sub-factors, and different node element sub-factors may be generated synchronously or under the request of comparison requirements.

[0324] The process of generating the node element factor may refer to the above embodiment, which will not be described in detail in this embodiment.

[0325] Optionally, the computer device may locate the interface control from the interface based on a single node element sub-factor in the node element factor, or a combination of multiple node element sub-factors.

[0326] In some embodiments, the node element sub-factors that characterize the control features from different dimensions can be divided into first precision sub-factors, second precision sub-factors, and third precision sub-factors. In the process of locating the interface control based on the node element factor, the interface control can be located based on all or part of the precision sub-factors in descending order of precision.

[0327] Optionally, the first node element subfactor and the second node element subfactor belong to the first precision subfactor, the third node element subfactor, the fourth node element subfactor and the fifth node element subfactor belong to the second precision subfactor, and the sixth node element subfactor, the seventh node element subfactor and the eighth node element subfactor belong to the third precision subfactor.

[0328] In the process of positioning the interface control based on the node element factor, the priority of the first precision sub-factor is higher than the priority of the second precision sub-factor, and the priority of the second precision sub-factor is higher than the priority of the third precision sub-factor.

[0329] In the process of locating the interface control, if a unique interface control can be located based on the first precision sub-factor, there is no need to perform positioning based on the second precision sub-factor and the third precision sub-factor.

[0330] If the unique interface control cannot be located based on the first precision sub-factor, the interface control location is continued based on the second precision. If the unique interface control can be located based on the second precision sub-factor, there is no need to locate it based on the third precision sub-factor.

[0331] If the only interface control cannot be located based on the second precision sub-factor, the interface control is located based on the third precision.

[0332] Optionally, the first precision subfactors of two identical interface controls must be exactly the same. In other words, if the first precision subfactors of two interface controls are different, the two interface controls must not be the same interface control.

[0333] Optionally, when the first precision subfactors of the two interface controls are completely identical, the computer device further determines whether the two interface controls are identical based on the second precision subfactors of the two interface controls.

[0334] Optionally, if the second precision subfactors of the two interface controls are exactly the same, then the two interface controls are determined to be the same interface control. If the second precision subfactors of the two interface controls are completely different, then the two interface controls are determined to be different interface controls. If the second precision subfactors of the two interface controls are partially the same and partially different, the computer device further determines whether the two interface controls are the same based on the third precision subfactors of the two interface controls.

[0335] Optionally, if the third precision subfactors of the two interface controls are exactly the same, the two interface controls are determined to be the same interface control; if the third precision subfactors of the two interface controls are different, the two interface controls are determined to be different interface controls. Alternatively, if the matching degree of the third precision subfactors of the two interface controls is higher than a matching degree threshold, the two interface controls are determined to be the same interface control; if the matching degree of the third precision subfactors of the two interface controls is lower than the matching degree threshold, the two interface controls are determined to be different interface controls.

[0336] Sub-step 3: simulating the operation behavior of the interface control in the interface based on the behavior type included in the behavior information.

[0337] After locating the interface control in the interface, the computer device generates an operation behavior event for the target interface control based on the behavior type in the behavior information, thereby simulating the operation behavior for the interface control in the interface based on the operation behavior event.

[0338] Indicatively, Fig. 9 As shown, the computer device determines the operation path from the startup interface 93 to the backtracking interface 95. When the interface jumps based on the path information of the operation path, the computer device first starts the A application, displays the first interface 94, and then clicks (clicks) the first control 941 (control identifier 49) in the first interface 94 to jump to the backtracking interface 95, which includes "Sports News About Zhang San" 951.

[0339] In this embodiment, the computer device records the interface content information of the interface, so that it can subsequently determine the backtracking interface containing the content to be backtracked according to the content to be backtracked input by the user, and further determine the operation path to jump to the backtracking interface with the help of the operation trajectory model, thereby automatically jumping to the backtracking interface based on the operation path, so as to facilitate the user to quickly view the content to be backtracked.

[0340] Machine usage report function

[0341] With the help of the generated application operation path information and the recorded interface content information, the computer device can provide more abundant user usage data. In a possible implementation, the computer device can generate a more comprehensive and abundant usage report through the following steps:

[0342] 1. Obtain the first user data through the system interface.

[0343] In some embodiments, the computer device may obtain first machine usage data representing the device status through a system interface.

[0344] Indicatively, Fig.10As shown, the computer device can obtain first user data such as sensors, application usage, device information, and location information through the system interface.

[0345] 2. Perform data mining on the interface content information, the global application operation path information, and the application operation path information to obtain second machine usage data, which is used to characterize the application state.

[0346] Since interface content information, global application operation path information, and application operation path information are all information of application usage dimension, the computer device can further mine the above information to obtain second usage data (usage data that cannot be obtained through the system interface) that characterizes the application status.

[0347] In a possible implementation, the computer device may desensitize the interface content information, global application operation path information, and application operation path information, and then input the data into a large language model (which may be set locally on the device), thereby obtaining the second machine usage data mined by the large language model.

[0348] Indicatively, Fig.10 As shown, the computer device can obtain second machine data such as file information, matter information, health information, browsing information and communication status by performing data mining on interface content information, global application operation path information and application operation path information.

[0349] 3. Generate target machine data based on the first machine data and the second machine data.

[0350] In a possible implementation, the computer device fuses the first user data and the second user data to obtain target user data. The computer device can match and fuse the first user data and the second user data generated at the same time according to the generation time of the user data, so that the fused target user data can reflect the device status and application status of the device at the same time.

[0351] 4. Generate a machine usage report based on the target machine usage data.

[0352] In a possible implementation, the computer device divides the target machine usage data according to vertical fields to obtain data in multiple analysis domains. Further, different analysis domains are further measured and scored to obtain a machine usage report of the current user. The machine usage report may include dimensional scores of multiple dimensions.

[0353] In some embodiments, the computer device can input the target user data into the large language model and instruct the large language model to divide the data according to a preset analysis domain (using a prompt indicating data division, which may include division criteria); and further instruct the large language model to generate dimensional scores for different measurement dimensions based on data from different analysis domains (using a prompt indicating dimensional scoring, which may include scoring criteria).

[0354] Indicatively, Fig.10 As shown, the computer device divides the target machine usage data into analysis domains according to vertical domains, such as time management, hobbies and interests, entertainment experience, financial situation, career development, life management, learning and growth, emotional interpersonal relationships, sports and health, and goal planning, and further measures and scores the information in the analysis domains, ultimately obtaining dimensional scores for the six measurement dimensions of sports, life, knowledge, work, finance, and health.

[0355] In some embodiments, the computer device may generate a usage report in the form of text, charts, etc. based on the target usage data.

[0356] In some embodiments, the computer device can also use a large language model to generate usage suggestions for the usage report and display them to improve the rationality of the user's usage.

[0357] In one possible implementation, the computer device may periodically generate target usage data according to a statistical period. When target usage data for at least two usage periods is obtained, the computer device may generate and display usage recommendations based on the target usage data for at least two usage statistical periods through a large language model.

[0358] The statistical period may be a day, a week or a month.

[0359] Optionally, the computer device may generate a comparison chart and a usage recommendation for the comparison chart based on the target machine usage data of at least two statistical periods.

[0360] Content filling function

[0361] When using an application, users often copy and paste content between different interfaces. For example, after a user copies a piece of text in a chat application, he pastes the text into a browser for search. However, copying and pasting content between such interfaces relies on manual execution by the user.

[0362] In order to improve the efficiency of content filling, in a possible implementation, the computer device can determine the interface with content filling requirements based on the historical interface content filling behavior in the interface, and automatically record the interface content in the interface before the jump, so as to automatically fill the recorded interface content into the interface after the jump. Optionally, the process may include the following steps:

[0363] Step 1: Based on the jump relationship between interfaces represented by the global operation path information and the copy and paste behavior within the interface represented by the global operation path information, determine the associated interfaces with content filling relationship.

[0364] Since the global operation path information can represent the jump relationship between different interfaces in the same application and between interfaces of different applications, and the global operation path information can represent the operation behavior within the interface, the computer device can determine the associated interfaces for content filling before and after the jump by parsing the global operation path information.

[0365] Among them, the associated interface with a content filling relationship is composed of a pre-jump interface and a post-jump interface, and the interface content in the pre-jump interface is used to fill the post-jump interface.

[0366] In a possible implementation, the computer device parses the side information in the global operation path information, determines that there is side information of the interface content copy behavior and the interface content paste behavior, and further matches the copied content of the interface content copy behavior with the pasted content of the interface content paste behavior. When the copied content matches the pasted content, and it is determined based on the side information that there is a jump relationship between the interfaces of the interface content copy behavior and the interface content paste behavior, the computer device determines that the interfaces of the interface content copy behavior and the interface content paste behavior are associated interfaces with a content filling relationship.

[0367] Step 2: During the use of the device, record the interface content of the interface before the jump in the associated interface.

[0368] During the use of the computer, when the computer device identifies the interface before the jump in the current associated interface, the interface content in the interface before the jump is stored for use in subsequent automatic content filling.

[0369] In a possible implementation, the computer device may record the interface content of the filling content type in the interface before the jump based on the filling content type of the content copied and pasted between the associated interfaces in the global operation path information.

[0370] Of course, in other possible implementations, the computer device may record all the interface content in the interface before the jump, and selectively use it during subsequent automatic filling.

[0371] Step 3: When jumping to the post-jump interface in the associated interface, fill the post-jump interface with content based on the interface content.

[0372] When the post-jump interface in the associated interface is identified, even if the user does not perform a content copy operation in the pre-jump interface, since the interface content in the pre-jump interface has been recorded, the computer device can automatically fill in the content in the post-jump interface based on the recorded interface content.

[0373] In a possible implementation, the computer device matches the recorded interface content with the interface content that needs to be filled in the fillable content area in the jump interface, thereby filling the matched interface content into the fillable content area in the jump interface. The matching process can be based on the node element information of the node element in the interface, which is not described in detail in this embodiment.

[0374] In an illustrative scenario, the chat interface of the chat application and the browser homepage of the browser application are associated interfaces with a content-filling relationship. When a user receives a link in the chat interface of the chat application and switches to the browser application, the computer device automatically fills the link into the address bar of the browser homepage.

[0375] In addition to real-time recording and automatic filling of interface content, in other possible implementations, the computer device generates available filling content and content usage of the available filling content based on the content filling behavior in the interface represented by the global operation path information.

[0376] In some embodiments, the computer device traverses each side information in the global operation path information, determines the side information of the content filling behavior, and determines the content filled by the content filling behavior as the available filling content. Further, the computer device identifies the purpose of the available filling content to obtain the content purpose of the available filling content. The purpose identification can be determined by a large language model or a pre-trained purpose classification model.

[0377] For example, the computer device generates historically populated addresses and contact numbers based on global operation path information.

[0378] During the use of the machine, when it is recognized that there is a need to fill content in the current interface and the content purpose of the content to be filled matches the content purpose of the available filling content, the computer device fills the content in the current interface based on the available filling content.

[0379] In a possible implementation, the computer device determines whether there is a need to fill content in the current interface based on the node information and / or node element information corresponding to the current interface, and further determines the content purpose of the content to be filled. Schematically, the computer device can determine whether there is a need to fill content in the current interface based on the second node sub-factor and the fifth node sub-factor in the node information, or determine whether there is a need to fill content in the current interface based on the fourth node element sub-factor and the fifth node element sub-factor in the node element information.

[0380] Further, the computer device may determine the content purpose of the content to be filled based on the node information and / or node element information corresponding to the current interface. Schematically, the computer device may determine the content purpose of the content to be filled corresponding to the node element in the interface based on the fourth node element sub-factor and the fifth node element sub-factor in the node element information.

[0381] When the content purpose of the content to be filled matches the content purpose of the available filling content, the computer device automatically fills the available filling content into the current interface.

[0382] In an illustrative scenario, the address and contact number entered by the user in the delivery interface of application A are determined as available fill-in content. When the user subsequently enters the delivery interface of application B, the computer device recognizes that there is a need to fill in the address and contact number on the delivery interface, and automatically fills the address and contact number into the delivery interface of application B.

[0383] Please refer to Fig.11 , which shows a structural block diagram of a device for displaying the usage of a machine provided by an exemplary embodiment of the present application. The device includes:

[0384] The recording module 1101 is used to generate operation path information during the use of the device, where the operation path information is used to represent at least one of an operation path between applications and an operation path within an application;

[0385] A display module 1102 is used to display a usage status display interface when receiving a usage status display instruction, wherein the usage status display interface includes a usage status display control;

[0386] The display module 1102 is also used to respond to the triggering operation of the usage display control and display an operation path diagram based on the operation path information. The operation path diagram includes at least one of the jump relationships between different applications and the jump relationships between different interfaces in the same application.

[0387] Optionally, the recording module 1101 is used to:

[0388] During the use of the machine, global operation path information is generated, and the global operation path information is used to characterize the operation path between different applications;

[0389] The global operation path information is split according to the application dimension to obtain application operation path information of different applications, where the application operation path is used to characterize the operation path within the application.

[0390] Optionally, the recording module 1101 is used to:

[0391] Creating a global application, wherein all interfaces displayed during the use of the device belong to the global application;

[0392] During the use of the machine, the target node corresponding to the target interface involved in the operation path and the target behavior edge corresponding to the target operation behavior involved in the operation path are determined from the operation trajectory model, wherein the operation trajectory model includes the node information of the node corresponding to the interface in the application and the edge information of the behavior edge corresponding to the operation behavior in the interface;

[0393] The global operation path information corresponding to the global application is generated based on the target node information of the target node and the target edge information of the target behavior edge.

[0394] Optionally, the recording module 1101 is used to:

[0395] Based on the interface features of different dimensions of the target interface, a node factor corresponding to the target interface is generated; based on the node factor of the recorded node corresponding to the recorded interface in the operation trajectory model and the node factor corresponding to the target interface, the recorded interface matching the target interface is determined; and the recorded node corresponding to the matched recorded interface is determined as the target node corresponding to the target interface;

[0396] From the operation trajectory model, determine the starting node corresponding to the starting interface before the target operation behavior, and the ending node corresponding to the ending interface after the target operation behavior; when the target operation behavior is related to the target interface control, determine the node element corresponding to the target interface control from the operation trajectory model; based on the starting node, the ending node, the node element and the target behavior type of the target operation behavior, determine the target behavior edge corresponding to the target operation behavior.

[0397] Optionally, the recording module 1101 is used to:

[0398] For each of the target behavior edges in the global operation path information, determining a start node and an end node corresponding to the target behavior edge;

[0399] When the starting interface corresponding to the starting node belongs to the first application, and the terminating interface corresponding to the terminating node belongs to the first application, adding the target behavior edge to the first application operation path information corresponding to the first application;

[0400] When the starting interface corresponding to the starting node belongs to the first application and the terminating interface corresponding to the terminating node belongs to the second application, the target behavior edge is added to the first application operation path information corresponding to the first application, and the target behavior edge is added to the second application operation path information corresponding to the second application.

[0401] Optionally, the usage display control includes a global usage display control and application usage display controls corresponding to different applications;

[0402] The display module 1102 is used to:

[0403] In response to a triggering operation on the global usage display control, a global operation path map is displayed based on the global operation path information, wherein the global operation path map includes interfaces of different applications and jump relationships between the interfaces;

[0404] In response to a trigger operation on the application usage display control corresponding to the target application, a target application operation path graph of the target application is displayed based on the application operation path information of the target application, wherein the target application operation path graph includes interfaces in the target application and jump relationships between interfaces.

[0405] Optionally, the display module 1102 is further used for:

[0406] Based on the target node information in the global operation path information, an interface diagram in the global operation path diagram is generated, wherein the target node information is node information corresponding to the target node of the target interface involved in the operation path;

[0407] Based on the target edge information in the global operation path information, an edge diagram connecting the interface diagrams in the global operation path graph is generated, wherein the edge diagram is used to indicate the jump direction between interfaces, and the target edge information is the edge information of the target behavior edge corresponding to the target operation behavior involved in the operation path.

[0408] Optionally, the machine usage status display interface also includes a backtracking control;

[0409] The device also includes a backtracking module, which is used to:

[0410] In response to a triggering operation on the backtracking control, obtaining content to be backtracked;

[0411] Based on the content to be backtracked and the interface content information corresponding to the interface in the application operation path information, determining a backtracking interface containing the content to be backtracked, wherein the interface content information is used to characterize the content contained in the interface;

[0412] Jump to the backtracking interface.

[0413] Optionally, the interface content information includes at least one of the following:

[0414] The text contained in the interface;

[0415] Picture description information of pictures in the interface;

[0416] A summary of the content contained in the interface;

[0417] Content embedding representation of what the interface contains;

[0418] Named entities extracted from the interface.

[0419] Optionally, the backtracking module is used to:

[0420] In a case where the content to be traced back includes time information, the application operation path information matching the time information is determined.

[0421] Optional, backtracking module, for:

[0422] Determine an operation path to jump to the backtracking interface based on an operation trajectory model, wherein the operation trajectory model includes node information corresponding to a node in the interface in the application and edge information corresponding to a behavior edge of an operation behavior in the interface, wherein the operation path is composed of nodes and behavior edges between nodes;

[0423] Based on the path information corresponding to the operation path, jumping to the backtracking interface by simulating the operation behavior, the path information includes edge information corresponding to the behavior edge between nodes on the operation path, and the edge information includes behavior information indicating the operation behavior.

[0424] Optionally, the device further includes a report generating module, which is used to:

[0425] Acquire first machine usage data through a system interface, where the first machine usage data is used to characterize a device status;

[0426] Performing data mining on the interface content information, the global application operation path information, and the application operation path information to obtain second machine usage data, where the second machine usage data is used to characterize the application state;

[0427] generating target machine data based on the first machine data and the second machine data;

[0428] A machine usage report is generated based on the target machine usage data.

[0429] Optionally, the report generation module is further used to: generate and display machine usage recommendations through a large language model based on the target machine usage data of at least two machine usage statistical periods.

[0430] Optionally, the device further comprises an automatic filling module, for:

[0431] Based on the content filling behavior in the interface represented by the global operation path information, available filling content and the content purpose of the available filling content are generated; during the use of the device, when it is recognized that there is a need for content filling in the current interface and the content purpose of the content to be filled matches the content purpose of the available filling content, content filling is performed in the current interface based on the available filling content; and / or,

[0432] Based on the jump relationship between the interfaces represented by the global operation path information, and the copy and paste behavior within the interface represented by the global operation path information, the associated interface with the content filling relationship is determined; during the process, the interface content within the pre-jump interface in the associated interface is recorded; when jumping to the post-jump interface in the associated interface, the content is filled in the post-jump interface based on the interface content.

[0433] To summarize, in an embodiment of the present application, by recording the operation paths between applications and the operation paths within an application during the use of the machine, and generating operation path information, when the machine usage status is subsequently displayed, an operation path diagram representing the jump relationship between different applications and the jump relationship between different interfaces in the same application can be restored based on the operation path information, thereby facilitating users to review historical operations within and between applications and achieving a more fine-grained display of machine usage status.

[0434] See also Fig.12 , Fig.12 12 is a schematic diagram of a computer device provided by an exemplary embodiment of the present application. The computer device may also include one or more of the following components: a processor 1210 and a memory 1220 .

[0435] Optionally, the processor 1210 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1220, and calling data stored in the memory 1220. Optionally, the processor 1210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1210, but may be implemented by a separate chip.

[0436] The memory 1220 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1220 includes a non-transitory computer-readable storage medium. The memory 1220 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0437] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device, and the computer device may include more (such as communication components, display components) or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0438] An embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for displaying the usage of the machine as described in the above embodiment.

[0439] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the machine usage display method as described in the above embodiment.

[0440] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0441] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for displaying machine usage, characterized in that: The method comprises: During the use of the machine, operation path information is generated, where the operation path information is used to represent at least one of an operation path between applications and an operation path within an application; When receiving a usage status display instruction, displaying a usage status display interface, wherein the usage status display interface includes a usage status display control; In response to a triggering operation on the usage display control, an operation path graph is displayed based on the operation path information, and the operation path graph includes at least one of a jump relationship between different applications and a jump relationship between different interfaces in the same application.

2. The method according to claim 1, characterized in that In the process of using the machine, the operation path information is generated, including: During the use of the machine, global operation path information is generated, and the global operation path information is used to characterize the operation path between different applications; The global operation path information is split according to the application dimension to obtain application operation path information of different applications, where the application operation path is used to characterize the operation path within the application.

3. The method according to claim 2, characterized in that In the process of using the machine, global operation path information is generated, including: Creating a global application, wherein all interfaces displayed during the use of the device belong to the global application; During the use of the machine, the target node corresponding to the target interface involved in the operation path and the target behavior edge corresponding to the target operation behavior involved in the operation path are determined from the operation trajectory model, wherein the operation trajectory model includes the node information of the node corresponding to the interface in the application and the edge information of the behavior edge corresponding to the operation behavior in the interface; The global operation path information corresponding to the global application is generated based on the target node information of the target node and the target edge information of the target behavior edge.

4. The method according to claim 3, characterized in that The step of determining, from the operation trajectory model, a target node corresponding to a target interface involved in the operation path, and a target behavior edge corresponding to a target operation behavior involved in the operation path, includes: Based on the interface features of different dimensions of the target interface, a node factor corresponding to the target interface is generated; based on the node factor of the recorded node corresponding to the recorded interface in the operation trajectory model and the node factor corresponding to the target interface, the recorded interface matching the target interface is determined; and the recorded node corresponding to the matched recorded interface is determined as the target node corresponding to the target interface; From the operation trajectory model, determine the starting node corresponding to the starting interface before the target operation behavior, and the ending node corresponding to the ending interface after the target operation behavior; when the target operation behavior is related to the target interface control, determine the node element corresponding to the target interface control from the operation trajectory model; based on the starting node, the ending node, the node element and the target behavior type of the target operation behavior, determine the target behavior edge corresponding to the target operation behavior.

5. The method according to claim 2, characterized in that: The step of splitting the global operation path information according to the application dimension to obtain the application operation path information of different applications includes: For each of the target behavior edges in the global operation path information, determining a start node and an end node corresponding to the target behavior edge; When the starting interface corresponding to the starting node belongs to the first application, and the terminating interface corresponding to the terminating node belongs to the first application, adding the target behavior edge to the first application operation path information corresponding to the first application; When the starting interface corresponding to the starting node belongs to the first application and the terminating interface corresponding to the terminating node belongs to the second application, the target behavior edge is added to the first application operation path information corresponding to the first application, and the target behavior edge is added to the second application operation path information corresponding to the second application.

6. The method according to claim 2, characterized in that The usage display control includes a global usage display control and application usage display controls corresponding to different applications; The step of displaying an operation path diagram based on the operation path information in response to a triggering operation on the usage display control includes: In response to a triggering operation on the global usage display control, a global operation path map is displayed based on the global operation path information, wherein the global operation path map includes interfaces of different applications and jump relationships between the interfaces; In response to a trigger operation on the application usage display control corresponding to the target application, a target application operation path graph of the target application is displayed based on the application operation path information of the target application, wherein the target application operation path graph includes interfaces in the target application and jump relationships between interfaces.

7. The method according to claim 6, characterized in that The method further comprises: Based on the target node information in the global operation path information, an interface diagram in the global operation path diagram is generated, wherein the target node information is node information corresponding to the target node of the target interface involved in the operation path; Based on the target edge information in the global operation path information, an edge diagram connecting the interface diagrams in the global operation path graph is generated, wherein the edge diagram is used to indicate the jump direction between interfaces, and the target edge information is the edge information of the target behavior edge corresponding to the target operation behavior involved in the operation path.

8. The method according to claim 2, characterized in that: The machine usage display interface also includes a backtracking control; The method further comprises: In response to a triggering operation on the backtracking control, obtaining content to be backtracked; Based on the content to be backtracked and the interface content information corresponding to the interface in the application operation path information, determining a backtracking interface containing the content to be backtracked, wherein the interface content information is used to characterize the content contained in the interface; Jump to the backtracking interface.

9. The method according to claim 8, characterized in that The interface content information includes at least one of the following: The text contained in the interface; Picture description information of pictures in the interface; A summary of the content contained in the interface; Content embedding representation of what the interface contains; Named entities extracted from the interface.

10. The method according to claim 8, characterized in that Before determining the backtracking interface including the content to be backtracked based on the content to be backtracked and the interface content information corresponding to the interface in the application operation path information, the method further includes: In a case where the content to be traced back includes time information, the application operation path information matching the time information is determined.

11. The method according to claim 8, characterized in that The jumping to the backtracking interface includes: Determine an operation path to jump to the backtracking interface based on an operation trajectory model, wherein the operation trajectory model includes node information corresponding to a node in the interface in the application and edge information corresponding to a behavior edge of an operation behavior in the interface, wherein the operation path is composed of nodes and behavior edges between nodes; Based on the path information corresponding to the operation path, jumping to the backtracking interface by simulating the operation behavior, the path information includes edge information corresponding to the behavior edge between nodes on the operation path, and the edge information includes behavior information indicating the operation behavior.

12. The method according to claim 8, characterized in that The method further comprises: Acquire first machine usage data through a system interface, where the first machine usage data is used to characterize a device status; Performing data mining on the interface content information, the global application operation path information, and the application operation path information to obtain second machine usage data, where the second machine usage data is used to characterize the application state; generating target machine data based on the first machine data and the second machine data; A machine usage report is generated based on the target machine usage data.

13. The method according to claim 12, characterized in that The method further comprises: Based on the target machine usage data of at least two machine usage statistical periods, machine usage suggestions are generated and displayed through a large language model.

14. The method according to claim 2, characterized in that The method further comprises at least one of the following: Based on the content filling behavior in the interface represented by the global operation path information, generating available filling content and content usage of the available filling content; During the use of the device, when it is recognized that there is a need to fill content in the current interface, and the content purpose of the content to be filled matches the content purpose of the available filling content, the content is filled in the current interface based on the available filling content; Based on the jump relationship between the interfaces represented by the global operation path information and the copy and paste behavior in the interface represented by the global operation path information, the associated interface with the content filling relationship is determined; during the process, the interface content in the interface before the jump in the associated interface is recorded; In the case of jumping to the post-jump interface in the associated interface, content is filled in the post-jump interface based on the interface content.

15. A device for displaying machine usage, characterized in that: The device comprises: A recording module, used to generate operation path information during the use of the machine, wherein the operation path information is used to represent at least one of an operation path between applications and an operation path within an application; A display module, configured to display a usage status display interface upon receiving a usage status display instruction, wherein the usage status display interface includes a usage status display control; The display module is also used to respond to the triggering operation of the usage display control and display an operation path diagram based on the operation path information. The operation path diagram includes at least one of the jump relationships between different applications and the jump relationships between different interfaces in the same application.

16. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the machine usage display method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the machine usage display method as described in any one of claims 1 to 14.

18. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the machine usage display method as described in any one of claims 1 to 14.

Images