Generation method and device of operation track model, equipment and storage medium

Abstract

The invention provides an operation track model generation method and device, equipment and a storage medium, and belongs to the technical field of computers. According to the method, the interfaces passed by the operation track and the used operation behaviors are obtained, then the interfaces are represented by nodes, the jump relation between the interfaces based on the operation behaviors is represented by directed edges, and then a graph model, namely an operation track model, can be constructed; the operation track model can visually represent an operation track from one interface to another interface.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for generating an operation trajectory model. Background Art

[0002] With the development of computer technology, computer devices include more and more interfaces. Users can jump between different interfaces through a series of operations to browse the content in different interfaces. The user's operation behavior and interface information can reflect the user's operation trajectory, which can be widely used in testing, teaching and other scenarios. Therefore, how to efficiently determine the operation trajectory is a problem that needs to be solved. Summary of the invention

[0003] Embodiments of the present application provide a method, apparatus, device, and storage medium for generating an operation trajectory model. The operation trajectory model can intuitively represent an operation trajectory from one interface to another interface.

[0004] According to one aspect of an embodiment of the present application, a method for generating an operation trajectory model is provided, the method comprising:

[0005] Acquire interface information and event information, wherein the interface information is used to indicate multiple interfaces in the operation track, and the event information is used to indicate at least one operation behavior in the operation track;

[0006] Based on the interface information and the event information, an operation trajectory model is generated, wherein the operation trajectory model includes a plurality of nodes and directed edges between the nodes, wherein the nodes are used to represent interfaces, and the directed edges between the nodes are used to represent jump relationships between interfaces based on operation behaviors.

[0007] According to another aspect of an embodiment of the present application, a device for generating an operation trajectory model is provided, the device comprising:

[0008] An information acquisition module, used to acquire interface information and event information, wherein the interface information is used to indicate multiple interfaces in the operation track, and the event information is used to indicate at least one operation behavior in the operation track;

[0009] A model generation module is used to generate an operation trajectory model based on the interface information and the event information, wherein the operation trajectory model includes multiple nodes and directed edges between the nodes, the nodes are used to represent the interface, and the directed edges between the nodes are used to represent the jump relationship between the interfaces based on the operation behavior.

[0010] In some embodiments, the information acquisition module is used to acquire input events and system events in real time based on accessibility services and system gesture monitoring management services to obtain first data in the event information; in response to changes in the interface caused by the first data, the interface changes are acquired in real time based on the accessibility service to obtain second data in the interface information.

[0011] In some embodiments, the model generation module is used to determine two nodes in the operation trajectory model in real time based on the second data, and the two nodes respectively represent two interfaces before and after the change; based on the first data, a directed edge is constructed between the two nodes in real time.

[0012] In some embodiments, the information acquisition module is used to acquire input events and system events based on the accessibility service and the system gesture monitoring management service, and obtain the event information, wherein the event information includes multiple events; based on the accessibility service, acquire changes in the interface and obtain the interface information, wherein the interface information includes the multiple interfaces.

[0013] In some embodiments, the model generation module is used to generate an operation behavior stream based on the interface information and the event information, wherein the operation behavior stream is used to indicate the timing relationship between the interface and the event; and to generate the operation trajectory model based on the operation behavior stream.

[0014] In some embodiments, the information acquisition module is used to obtain changes in the interface based on accessibility services to obtain the interface information, wherein the interface information includes the multiple interfaces; determine the behavior type of the operation behavior corresponding to each control in the multiple interfaces according to the control type of each control in the multiple interfaces; filter out controls that cannot be operated to obtain the event information, wherein the event information includes multiple events, and different events correspond to different operation behaviors.

[0015] In some embodiments, the information acquisition module is used to input the task information and the current interface into a large language model for the i-th event determination process to obtain an event in the event information, and the large language model is used to predict events based on the interface and task information; based on accessibility services, obtain changes in the interface to obtain an interface in the interface information, and the interface information includes the multiple interfaces.

[0016] In some embodiments, the model generation module is further used to generate an operation trajectory path based on the large language model, and the operation trajectory path is used to indicate the temporal relationship between the interface and the event.

[0017] In some embodiments, the model generation module is used to determine the nodes in the operation trajectory model based on the interface information and through a node grouping strategy, wherein the node grouping strategy is used to divide the nodes corresponding to the interfaces belonging to the same user interface activity into the same node group; based on the event information, determine the directed edges between the nodes in the operation trajectory model.

[0018] In some embodiments, the model generation module is further used to add at least one leading edge in front of any node when a leading operation is required for the interface corresponding to the node to jump to another interface.

[0019] In some embodiments, the model generation module is further used to obtain screenshots, skeleton diagrams, and interface compression trees of the multiple interfaces based on accessibility services and system gesture monitoring management services.

[0020] In some embodiments, the information acquisition module is used to perform any of the following:

[0021] Based on the system debugging function, determining the interface information of the multiple interfaces;

[0022] Obtaining screenshots of the multiple interfaces based on the barrier-free server, converting the screenshots of the multiple interfaces into an interface tree structure, and determining interface information of the multiple interfaces based on the interface tree structure;

[0023] The structured data of at least one application is parsed to obtain interface information of the multiple interfaces, and the at least one application includes the multiple interfaces.

[0024] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the method for generating an operation trajectory model as described in the above aspects.

[0025] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for generating an operation trajectory model as described in the above aspects.

[0026] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for generating an operation trajectory model as described in the above aspects.

[0027] The generation scheme of the operation trajectory model provided in the embodiment of the present application obtains the interfaces passed through and the operation behaviors used in the operation trajectory, and then uses nodes to represent the interfaces, and uses directed edges to represent the jump relationships between interfaces based on the operation behaviors, so as to construct a graph model, that is, an operation trajectory model. The operation trajectory model can intuitively represent the operation trajectory from one interface to another. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of an operation trajectory model provided according to an embodiment of the present application;

[0031] Figure 3 is a flow chart of a method for generating an operation trajectory model provided in an embodiment of the present application;

[0032] Figure 4 is a flowchart of another method for generating an operation trajectory model provided in an embodiment of the present application;

[0033] Figure 5 is a flowchart of another method for generating an operation trajectory model provided in an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of an operation behavior flow provided according to an embodiment of the present application;

[0035] Figure 7 is a flowchart of another method for generating an operation trajectory model provided in an embodiment of the present application;

[0036] Figure 8 is a flowchart of another method for generating an operation trajectory model provided in an embodiment of the present application;

[0037] Fig. 9 is a schematic diagram of generating an operation trajectory model based on a large language model according to an embodiment of the present application;

[0038] Fig.10 It is a structural schematic diagram of a device for generating an operation trajectory model provided in an embodiment of the present application;

[0039] Fig.11It is a structural block diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] In this article, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the interfaces involved in this application are all obtained with full authorization.

[0043] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 , the implementation environment includes a terminal 101 and a server 102. The terminal 101 can be directly or indirectly connected to the server 102 through wired or wireless communication.

[0044] In some embodiments, the terminal 101 may be a mobile phone, a desktop computer, a laptop computer, a tablet computer, etc. The terminal 101 may have various applications installed and running, and interfaces included in different applications may not be completely the same.

[0045] In some embodiments, server 102 is an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0046] In some embodiments, the server 102 undertakes the main computing work and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture to perform collaborative computing.

[0047] The operation trajectory model (Trajectory Structure) is introduced below.

[0048] Figure 2 is a schematic diagram of an operation trajectory model provided according to an embodiment of the present application. Figure 2 As shown, Figure 2 (a) in the figure shows an operation process by way of example, wherein the interface through which the operation process passes is identified as 201, and the operation behavior is identified as 202. Accordingly, Figure 2 (b) in the figure exemplarily shows some nodes in the operation trajectory model 203 and the directed edges between the some nodes. The directed edges correspond to the operation behaviors in the interface and are used to connect different nodes (interface switching occurs after the operation) or connect the same node (no interface switching occurs after the operation). Among them, nodes 2031-node 2035 represent interface 1-interface 5 respectively. It should be noted that the node includes node information, which is used to characterize the interface features of the interface from different dimensions to ensure the accuracy of subsequent interface retrieval and positioning; the directed edge includes edge information, which is used to indicate the operation behavior and the node jump situation before and after the operation behavior, to ensure the accurate restoration of the operation in the subsequent interface. Optionally, multiple nodes and directed edges in the operation trajectory model constitute at least one operation trajectory path. The operation trajectory path includes path information, which is used to characterize the sequence between the indication interface and different operation behaviors to ensure that complex operations are performed in sequence.

[0049] Figure 3 is a flow chart of a method for generating an operation trajectory model provided in an embodiment of the present application. The method is executed by a computer device, see Figure 3 , the method comprising:

[0050] 301. The computer device obtains interface information and event information.

[0051] In the embodiment of the present application, the interface information is used to indicate multiple interfaces in the operation track, wherein the interface is a visual user interface, such as a desktop, an application program interface, a system interface, and the like.

[0052] Event information is used to indicate at least one operation behavior in the operation track. The operation behavior refers to a manual operation by the user, such as triggering an interface control; or the operation behavior refers to a system behavior, such as an automatic interface jump operation after a preset duration.

[0053] The operation track can also be called the operation path. Since the user will perform a series of operations and jump between multiple interfaces during the operation process, the computer device can sort the operation behaviors and displayed interfaces according to the execution sequence between the operation behaviors to obtain the operation path (Path). In other words, the operation track can indicate which operation behavior is used to jump between which pages. Figure 2 (a) shows an operation trajectory by way of example.

[0054] In some embodiments, the path information of the operation path includes at least a path identifier and a behavior edge list.

[0055] Among them, similar to the node identifier, node element identifier and behavior edge identifier, the path identifier is a unique identifier assigned to the operation path (with no actual meaning), which can be expressed in the form of a string, where the path identifiers of different operation paths are different. The behavior edge list contains a series of behavior edges corresponding to the operation behaviors, and the behavior edges are arranged in the time sequence of the operation behaviors. For example, the behavior edge at the head of the list in the behavior edge list corresponds to the starting operation behavior, and the behavior edge at the end of the list corresponds to the ending operation behavior.

[0056] In some embodiments, the behavior edge list includes the behavior edge identifier in the edge information corresponding to the behavior edge.

[0057] Since the edge information of the behavior edge can indicate the nodes (interfaces) involved before and after the operation behavior and the behavior information of the executed operation behavior, the edge information of the behavior edge arranged in time sequence in the behavior edge list can accurately describe the execution order of a series of operation behaviors and what kind of operation behaviors are executed in the interface.

[0058] It should be noted that the interface in the embodiment of the present application is not limited to the visible range of the screen, but includes completed interfaces within and outside the visible range of the screen. Accordingly, before and after the interface content is scrolled, even if the displayed content within the visible range of the screen changes significantly, the interface before and after the scrolling will not be judged as a different interface. The different interfaces in the embodiment of the present application refer to interfaces with large differences in interface layout, content, and style. Usually, when the interface switches, the interfaces before and after the switch are considered to be different interfaces. For example, when the text and pictures in the interface change dynamically, or when the list elements in the interface change, they will not be recognized as different interfaces; when clicking the control in the interface to trigger the interface jump, the interfaces before and after the jump are recognized as different interfaces.

[0059] 302. The computer device generates an operation trajectory model based on the interface information and the event information.

[0060] In the embodiment of the present application, the operation trajectory model is a graph data, which is used to represent the relationship between the interface and the operation behavior through nodes and edges. Accordingly, the operation trajectory model includes multiple nodes and directed edges between nodes. Among them, the node is used to represent the interface, and the directed edge between the nodes is used to represent the jump relationship between the interfaces based on the operation behavior.

[0061] It should be noted that a node includes node information, which includes information that characterizes the interface features of an interface from different dimensions. Different nodes characterize the interface features of their respective corresponding interfaces through their respective node information, that is, different nodes can be distinguished based on the node information. Optionally, the dimensions that characterize the interface features may include interface content dimensions, interface layout dimensions, interface function dimensions, interface description dimensions, the window to which the interface belongs, and the like.

[0062] In some embodiments, the node information of the node includes at least a node identification and a node factor of the node. Accordingly, determining the node information of the node corresponding to the interface during the operation may include: determining the node factor of the node corresponding to the interface during the operation, the node factor including node sub-factors representing the interface features of the interface from different dimensions; determining the node identification of the node; and determining the node factor and the node identification as the node information of the node.

[0063] Among them, the node identifier is a unique identifier assigned to the node (it has no practical meaning) and can be represented in the form of a string. Different nodes have different node identifiers, and the same node has the same node identifier. The node factor (NodeFactor) is composed of several fields that characterize interface features from different dimensions, among which the fields that characterize interface features can be called node sub-factors. When determining whether two nodes are the same node, the computer device matches the field contents with the same field name in the two node factors, that is, matches the node sub-factors in different node factors, thereby determining whether the two nodes are the same node based on the matching results of multiple dimensions.

[0064] In some embodiments, the node factor may include at least one of windowName (window name), operableElements (operable elements), layoutVector (layout vector), screenVector (screen vector) and summaryFunction (function description).

[0065] Among them, the window name of the window to which the interface belongs can be extracted from the interface code. The interface usually includes a large number of operable elements, such as button controls, expandable list controls, text input box controls, upload controls, etc., and these operable elements can reflect the interface characteristics of the interface to a certain extent. In the case of version updates, the interface content of the same interface may change significantly, but the change in the overall layout of the interface is usually small. In addition to judging different interfaces (nodes) from the dimension of interface layout, different interfaces can also be judged from a more intuitive visual level. During the interface development stage, developers usually add interface function description information to the interface code. Even if the interface effect is iteratively updated later, the interface function description information usually does not change.

[0066] In some embodiments, the interface includes multiple elements, and the node includes node elements. The node element information of the node element includes at least the node element identifier and the node element factor of the node element. The node element identifier is a unique identifier assigned to the node element (with no actual meaning), which can be represented in the form of a string. The node element identifiers of different node elements are different. Optionally, the node element factor (NodeElement Factor) is composed of several fields that characterize the control features of the interface control from different dimensions, wherein the fields that characterize the control features can be referred to as node element sub-factors. Similar to the role of the node factor, the role of the node element factor is to retrieve and locate the interface controls of the features in the interface.

[0067] In some embodiments, the node element factors may include packageName (package name), className (class name), ResourceId (resource identifier), contentDescription (content description), Text (text), listOrder (list), parentChain (parent chain), and preNodeElementFactor&postNodeElementFactor (node ​​tuple factor of the left interface control and node tuple factor of the right interface control).

[0068] It should be noted that a directed edge includes edge information, which includes information about nodes corresponding to the interfaces before and after the operation is performed. The interface identifiers corresponding to the interfaces before and after the operation can be different interface identifiers (i.e., the interface jump occurs after the operation) or the same interface identifier (i.e., the interface jump does not occur during the operation).

[0069] In some embodiments, the edge information includes at least: behavior information of the operation behavior, a starting node identifier (fromNodeID) of the starting node corresponding to the starting interface before the operation behavior, an ending node identifier (toNodeID) of the ending node corresponding to the ending interface after the operation behavior, and an edge identifier (a unique identifier similar to a node identifier and a node element identifier).

[0070] It should be noted that behavior information, as an important part of side information, needs to be able to clearly indicate the operation behavior.

[0071] In some embodiments, the behavior information includes the behavior type (ActionType) of the operation behavior. The common behavior class (CommonAction) includes all the operation behaviors of the user and the system, so as to ensure the accuracy of the recorded operation behaviors. Moreover, the behavior type of each operation behavior belongs to the behavior type class.

[0072] For example, taking the common operations in mobile phones as an example, the behavior type class may include at least one of the following behavior types: GoHome (return to the home page), RecentApps (recent applications), GoBack (return), StartApp (start application), Click (click), DoubleClick (double-click), LongClick (long press), InputText (input text), Scroll (scroll), Concurrent (collaborative operation, that is, operation behaviors that occur at the same time), Continuous (continuous operation, referring to operation behaviors performed in sequence), Stop (terminate), Wait (wait, a scenario where the interface changes automatically over time), and Unknown (unknown behavior, as a fallback behavior).

[0073] In some embodiments, when determining the behavior type of the operation behavior as the behavior information, if the operation behavior is related to an interface control in the interface, the computer device may also determine the node element identifier of the node element corresponding to the interface control as the behavior information.

[0074] For example, when the operation behavior is a click operation, the behavior information corresponding to the operation behavior includes the operation type "Click" and the node element identifier of the node element corresponding to the clicked button; for another example, when the operation behavior is a text input operation, the behavior information corresponding to the operation behavior includes the operation type "InputText" and the node element identifier of the node element corresponding to the text input box where the text is entered.

[0075] In some embodiments, the computer device may store the generated operation trajectory model so as to subsequently perform interface retrieval, positioning, and operation reproduction based on the operation trajectory model.

[0076] In summary, the embodiment of the present application provides a method for generating an operation trajectory model by obtaining the interfaces passed through and the operation behaviors used in the operation trajectory, and then using nodes to represent the interfaces, and using directed edges to represent the jump relationships between interfaces based on the operation behaviors, thereby constructing a graph model, that is, an operation trajectory model. The operation trajectory model can intuitively represent the operation trajectory from one interface to another.

[0077] Figure 4 This is a flow chart of another method for generating an operation trajectory model provided by an embodiment of the present application. The method is executed by a computer device, and is described by taking manual construction of an operation event and real-time generation of an operation trajectory model as an example. Figure 4 , the method comprising:

[0078] 401. Based on the barrier-free service and the system gesture monitoring management service, input events and system events are acquired in real time to obtain first data in the event information.

[0079] In an embodiment of the present application, Accessibility Service (ACS) refers to a technical service designed to help users with special needs better use electronic devices and software. Common accessibility services include screen reading technology, voice control technology, magnified display technology, vibration feedback technology, subtitle and sign language translation technology, and adjustable contrast and color setting technology. In an embodiment of the present application, a computer device can monitor changes in windows and content in the interface through accessibility services. A computer device can also capture input (INPUT) events through accessibility services.

[0080] The system gesture monitor management service (Gesture Monitor Manager) refers to a technical service used to manage and process gesture operations in an operating system or application. The main function of the system gesture monitor management service is to detect various gesture actions of the user on the device, such as clicking, sliding, etc., and trigger corresponding events for the application to process. Optionally, the computer device monitors the gesture actions through the system gesture monitor management service to generate accessible content, such as accurately finding the minimum range through the gesture click position, so as to determine the triggered user interface element (View). Optionally, the computer device can also identify system events such as Recent (recent tasks), Goback (return) and GoHome (return to the home screen) through the system gesture monitor management service.

[0081] It should be noted that the computer device can also filter special interfaces such as the status bar and pop-up windows to ensure the accuracy of the interface information obtained.

[0082] It should be noted that the event information includes multiple pieces of data. In the process of generating the operation trajectory model in real time, each time an operation event is obtained, a corresponding piece of data can be generated, which is called the first data for the convenience of description. If the operation behavior in the operation event corresponding to the first data causes the interface to jump, then the first data can be used to generate a directed edge in the operation trajectory model in real time. If the operation behavior in the operation event corresponding to the first data does not cause the interface to jump, that is, the interface is the same before and after the operation event occurs, then no directed edge is generated.

[0083] 402. In response to the interface change caused by the first data, based on the barrier-free service, the interface change is acquired in real time to obtain the second data in the interface information.

[0084] In the embodiment of the present application, the interface change usually refers to the interface jump, that is, jumping from one interface to another interface. If the first data causes the interface to jump, the computer device can obtain the changed interface in real time through the accessibility service, thereby determining the changed interface.

[0085] It should be noted that the interface information includes multiple pieces of data. During the real-time generation of the operation trajectory model, each time an interface jump occurs, a corresponding piece of data can be generated, which is referred to as second data for ease of description. The second data is used to describe the interface before the jump and the interface after the jump.

[0086] Optionally, if the interface does not jump, but the content in the interface changes, such as switching from one sub-interface in the interface to another sub-interface, it can also be classified as an interface change.

[0087] 403. Based on the second data, two nodes are determined in real time in the operation trajectory model, where the two nodes respectively represent two interfaces before and after the change.

[0088] In an embodiment of the present application, since the second data describes which interface to jump from and which interface, and the interface is represented by nodes in the operation trajectory model, the computer device can determine two nodes in the operation trajectory model in real time, and the two nodes are used to represent the two interfaces described by the second data.

[0089] It should be noted that determining two nodes means that if there are nodes corresponding to the two interfaces in the operation trajectory model, the corresponding nodes are obtained; if there is no corresponding node for either interface or both interfaces, the corresponding nodes are added to the trajectory operation model.

[0090] 404. Based on the first data, a directed edge is constructed between two nodes in real time.

[0091] In the embodiment of the present application, since the first data describes the operation behavior that causes the interface to jump, and the operation behavior is represented by a directed edge in the operation trajectory model, the computer device can construct a directed edge between two nodes in the operation trajectory model in real time.

[0092] The embodiment of the present application provides a method for generating an operation trajectory model, by obtaining the two interfaces before and after the jump and the operation behavior that causes the interface jump in real time when the interface jumps, and then using nodes to represent the interface, and using directed edges to represent the jump relationship between the interfaces based on the operation behavior, so as to construct two nodes and a directed edge of the operation trajectory model. The above method can significantly improve the construction efficiency of the operation trajectory model by constructing the operation trajectory model in real time.

[0093] The above embodiment exemplarily shows the process of constructing an operation trajectory model in real time. Accordingly, when the conditions for constructing an operation trajectory model in real time are not met, the computer device can also construct the operation trajectory model offline.

[0094] Figure 5 This is a flowchart of another method for generating an operation trajectory model provided by an embodiment of the present application. The method is executed by a computer device, and is described by taking manual construction of an operation event and offline generation of an operation trajectory model as an example. Figure 5 , the method comprising:

[0095] 501. Based on the barrier-free service and the system gesture monitoring management service, input events and system events are obtained to obtain event information, where the event information includes multiple events.

[0096] In the embodiment of the present application, the explanation of the barrier-free service and the system gesture monitoring management service refers to the above step 401, which will not be repeated here. The difference between the event information obtained in this step and the first data obtained in step 401 is that the event information obtained in this step is an operation event that occurs within a period of time, that is, the event information includes multiple data, each data represents an event. Among them, the period of time can be a fixed time, such as 1 hour, 12 hours, 1 day or 7 days. The period of time can also be a dynamic time, that is, the computer device can cache event information when it is not idle, if it monitors changes in the interface, and regenerate the operation trajectory model when it is idle.

[0097] 502. Based on the barrier-free service, obtain the change of the interface and obtain the interface information, where the interface information includes multiple interfaces.

[0098] In the embodiment of the present application, this step is the same as the above step 402, and will not be repeated here. The difference between the interface information obtained in this step and the second data obtained in step 402 is that the interface information obtained in this step is the interface that has jumped within a period of time, that is, the interface information includes multiple data, each of which is used to indicate the jump from one interface to another. For an explanation of the period of time, see step 501, which will not be repeated here.

[0099] 503. Generate an operation behavior flow based on the interface information and the event information, where the operation behavior flow is used to indicate a temporal relationship between the interface and the event.

[0100] In an embodiment of the present application, interface information can be converted into an interface stream, which is used to record the jump timing of the interface. The interface information includes information such as the layout of interface elements in the interface. Event information can be converted into an event stream, which is used to record the timing of operation behaviors (operation events). The three-track strategy can be used to merge the interface flow and the event flow into an operation behavior flow to represent the timing relationship between the operation behavior and the interface jump.

[0101] For example, Figure 6 is a schematic diagram of an operation behavior flow provided according to an embodiment of the present application, see Figure 6 As shown, the three tracks refer to the interface flow, operation behavior flow and event flow, all of which are based on time. According to the time of interface jump and the time of executing the operation behavior, the interface flow and event flow can be merged into the operation behavior flow. Circle 0 represents interface 0. Triangle a is event a. Circle 1 represents interface 1, that is, interface 1 jumped from interface 0 through event a. Circle 2 represents interface 2, and circle 3 represents interface 3. Since the time from interface 1 to interface 3 is very short, the human eye may not capture interface 2, or even interface 1, so the continuous interfaces can be delayed merged (merge interfaces within 500mm). Triangle e and triangle f are continuous events, such as continuous input events. The last event f can be used as the final event. For example, if the number 1 is entered first, then the number 2, and then the number 3 is entered, the input events should be merged into the input number 123.

[0102] 504. Generate an operation trajectory model based on the operation behavior flow.

[0103] In the embodiment of the present application, the interface in the operation behavior stream can be used to generate a node in the operation trajectory model, and the operation behavior corresponding to the event in the operation behavior stream can be used to generate a directed edge in the operation trajectory model.

[0104] The embodiment of the present application provides a method for generating an operation trajectory model. By obtaining the interface jump situation and the events that occurred within a period of time, the interface flow and the event flow can be obtained. By merging the interface flow and the event flow, the operation behavior flow can be obtained to represent the temporal relationship between the interface and the event. The above method can construct the operation trajectory model without affecting the system function by separating the data recording and generation process in the process of constructing the operation trajectory model. It has high versatility and is more suitable for actual production and use.

[0105] The above embodiment exemplarily shows the process of manually constructing events to generate an operation trajectory model. Optionally, the operation events may also be automatically constructed. Accordingly, the computer device may construct the operation trajectory model according to the automatically constructed operation events.

[0106] Figure 7 is a flowchart of another method for generating an operation trajectory model provided by an embodiment of the present application. The method is executed by a computer device, and is described by automatically constructing an operation event and generating an operation trajectory model as an example. Figure 7 , the method comprising:

[0107] 701. Based on the barrier-free service, obtain the change of the interface and obtain the interface information, where the interface information includes multiple interfaces.

[0108] In the embodiment of the present application, this step is the same as the above-mentioned step 402 and step 502, and will not be repeated here.

[0109] 702. Determine, according to the control type of each control in the multiple interfaces, the behavior type to which the operation behavior corresponding to each control belongs.

[0110] In the embodiment of the present application, the computer device can automatically generate operation behaviors according to the control types of each control in multiple interfaces, such as the control types corresponding to the click behavior (ClickAction), double-click behavior (Double ClickAction) and long press behavior (Long ClickAction) include switch, radio button, checkbox, button, text box, video box, seekbar and spinner, etc. The control types corresponding to the scrolling behavior or sliding behavior (ScrollAction) include image, list view, web view, map view, horizontal scroll view, scroller and view pager, etc. The control types corresponding to the input text behavior (InputTextAction) include input.

[0111] In some embodiments, the computer device may adopt a breadth-first strategy to traverse the controls in the above-mentioned multiple pages to improve the coverage of the spatial traversal.

[0112] 703. Filter out controls that cannot be operated and obtain event information, where the event information includes multiple events, and different events correspond to different operation behaviors.

[0113] In the embodiment of the present application, by filtering the inoperable controls, the following five types of operable controls are obtained: clickable controls, long-pressable controls, checkable controls, scrollable controls, and editable controls. The computer device can generate operation events by automatically triggering the remaining operable controls, which is different from the above embodiment in which the operation events are captured by the system gesture monitoring management service.

[0114] 704. Generate an operation trajectory model based on the interface information and the event information.

[0115] In the embodiment of the present application, after obtaining the interface information and event information, the nodes in the operation trajectory model and the directed edges between the nodes can be generated. This step is similar to the above steps 403 and 404, generating operation events in real time, obtaining interface information in real time, and generating the operation trajectory model in real time. Or, similar to the above steps 503 and 504, multiple operation events are generated and executed, the interface information and event information within a period of time are cached, and the operation trajectory model is generated based on the cached information.

[0116] An embodiment of the present application provides a method for generating an operation trajectory model. By automatically generating events according to the types of controls in the interface, and then combining accessibility services to obtain changes in the interface when executing events, interface information and event information can be efficiently obtained, thereby improving the efficiency of creating an operation trajectory model.

[0117] Optionally, the operation events may also be predicted by a large language model. Accordingly, the computer device may construct an operation trajectory model according to the operation events output by the large language model.

[0118] Figure 8 is a flowchart of another method for generating an operation trajectory model provided by an embodiment of the present application. The method is executed by a computer device, and is described by taking the generation of operation events based on a large language model and the generation of an operation trajectory model as an example, see Figure 8 , the method comprising:

[0119] 801. For the i-th event determination process, the task information and the current interface are input into the large language model to obtain an event in the event information. The large language model is used to predict the event based on the interface and task information.

[0120] In an embodiment of the present application, the input of the large language model is task information and the current interface, and the output is a predicted event, which is used to indicate what operation behavior to perform in the current interface so as to complete the task in the task information or improve the progress of completing the task.

[0121] For example, the task information is to jump to interface a, and the current interface is interface c. There are many ways to jump from interface c to interface a, such as interface c→interface b→interface a, or interface c→interface b→interface d→interface a, or →interface c→interface v→interface a. Then the large language model can generate an operation event to make interface c jump to interface b or to interface v.

[0122] 802. Based on the barrier-free service, obtain the change of the interface and obtain an interface in the interface information, where the interface information includes multiple interfaces.

[0123] In an embodiment of the present application, a computer device can execute an event predicted by the large language model. If the event causes an interface change, such as a jump in the interface, the computer device can obtain relevant information about the interface to which the interface is jumped based on the accessibility service, thereby obtaining a piece of data in the interface information. The interface information is described in the above embodiment and will not be repeated here.

[0124] 803. Generate an operation trajectory model based on the interface information and event information.

[0125] In the embodiment of the present application, after obtaining the interface information and event information, the nodes in the operation trajectory model and the directed edges between the nodes can be generated. This step is similar to the above steps 403 and 404, generating operation events in real time, obtaining interface information in real time, and generating the operation trajectory model in real time. Or, similar to the above steps 503 and 504, multiple operation events are generated and executed, the interface information and event information within a period of time are cached, and the operation trajectory model is generated based on the cached information.

[0126] In some embodiments, in the process of generating an operation trajectory model based on events output by a large language model, an operation trajectory path can also be generated based on the large language model, and the operation trajectory path is used to indicate the temporal relationship between the interface and the event, such as event 1 → interface a → event 2 → interface b → event 3 → interface c.

[0127] To more clearly describe the process of generating an operation trajectory model based on a large language model, see Fig. 9 As shown, Fig. 9 FIG. 1 is a schematic diagram of generating an operation trajectory model based on a large language model according to an embodiment of the present application. Fig. 9 , the computer device obtains the task information input by the user through the proxy service, and parses the task information to determine the target page. Optionally, during the parsing process, the proxy service will also integrate the information into a format that meets the input requirements of the large language model. The proxy service sends the current page and task information to the large language model, and the large language model returns the predicted event to the proxy service. The proxy service forwards the event to the execution engine of the computer device, and the execution engine executes the event. The screen of the computing device displays the interface after the event is executed. If the interface changes, the corresponding nodes and directed edges are generated, and the interface information is sent to the proxy service, which forwards the interface as the current interface to the large language model to start a new round of predictions.

[0128] The embodiment of the present application provides a method for generating an operation trajectory model. By predicting events based on a large language model, it is possible to more accurately imitate the user's browsing behavior. By combining accessibility services to obtain changes in the interface when executing events, interface information and event information can be efficiently obtained, thereby improving the efficiency of creating an operation trajectory model.

[0129] The above embodiments mainly introduce how to construct an event and how to obtain interface information and event information. The following describes the generation strategy of the operation trajectory model.

[0130] In an embodiment of the present application, a computer device can store the above-mentioned interface information and event information in the form of spatial information. Among them, the spatial information includes node space, edge space, operation behavior space, preceding behavior space and layout vector. Among them, the node space is used to represent the uniquely identified control in the interface. The edge space is used to record the key information from node to node. The operation behavior space is used to record the events required for the node to node edge, including a series of operation events. The preceding behavior space is used to record the addition of preceding steps to event dependencies to increase the success rate of interface playback.

[0131] In the process of generating the operation trajectory model, the generation of nodes is an important part. The computer device can solve the problem of interface duplication, that is, node duplication, through the node grouping strategy, so as to ensure that the operation trajectory path in the operation trajectory model is uniquely reachable. Accordingly, the computer device determines the nodes in the operation trajectory model based on the interface information through the node grouping strategy. The node grouping strategy is used to divide the nodes corresponding to the interface belonging to the same user interface activity into the same node group. Then, the computer device determines the directed edges between the nodes in the operation trajectory model based on the event information.

[0132] In some embodiments, since the interface of a computer device is usually represented by an activity, in actual scenarios, there is a situation where a single activity corresponds to multiple fragments, or a single activity corresponds to multiple pages. In this case, it is difficult to distinguish the two interfaces based on the name of the activity, so it is impossible to distinguish the interfaces directly based on the name of the activity. In order to solve this problem, in the embodiments of the present application, the interface is represented by a node, and based on the concept of the node, the concept of the node group (NodeGroup) is extended.

[0133] In some embodiments, the node group corresponds to the activity at the code level, that is, different activities correspond to different node groups. The node group is composed of at least one node, and the interface of the node at the visual level corresponds.

[0134] In an optional implementation, when generating the operation trajectory model, the computer device determines node group information of the node group based on the node information, and generates the operation trajectory model based on the node group information, the edge information, and the path information.

[0135] In order to enable the node group information to characterize each node contained in the node group and to distinguish different node groups, the node group information at least includes a node list and a node group factor (NodeGroupFactor). The node list includes nodes belonging to the node group, and the node group factor is used for the node group characteristics of the node group.

[0136] In some embodiments, the computer device divides nodes with the same activity name into the same node group. Optionally, at the code level, the node group is uniquely indicated by the combination of the application package name + activity name, that is, the node group factor includes the application package name (packageName) and the activity name (activityName) of the application. Optionally, at the code level, the node group is uniquely indicated by the combination of the application package name + activity name + activity title, that is, the node group factor includes the application package name (packageName), the activity name (activityName) and the activity title (activityTitle) of the application.

[0137] In some embodiments, for any node, when a preceding operation is required for the interface corresponding to the node to jump to another interface, at least one preceding edge is added before the node. For example, when the page jumps, in addition to clicking the submit button, a password and a verification code need to be entered, and these input operations can be used as preceding edges.

[0138] In some embodiments, the computer device can also obtain screenshots, skeleton diagrams, and interface compression trees of multiple interfaces based on the accessibility service and the system gesture monitoring management service. By obtaining screenshots, skeleton diagrams, and interface compression trees of multiple interfaces, the nodes and edges in the generated operation trajectory model can be intuitively viewed, and the operation trajectory model can be debugged conveniently and quickly, thereby improving the accuracy of the operation trajectory model.

[0139] It should be noted that, in addition to the method of obtaining interface information mentioned in the above embodiment, the computer device can also use the system debugging (Debug) capability to capture the information of the controls on the screen to determine the interface information of multiple interfaces. Alternatively, the computer device can also obtain screenshots of multiple interfaces based on the barrier-free server through visual processing, convert the screenshots of multiple interfaces into an interface tree structure, and determine the interface information of multiple interfaces based on the interface tree structure. Alternatively, the computer device can also parse the structured data of at least one application to obtain interface information of multiple interfaces, and at least one application includes multiple interfaces. For example, text and pictures are obtained through Android Native controls, pictures are obtained through SurfaceView, pictures are obtained through TextureView, and text, picture URLs (Uniform Resource Locator) and video URLs are obtained through WebView.

[0140] It should be noted that in addition to the methods of obtaining event information mentioned in the above embodiments, the computer device can also monitor the back function from the SystemUI (System User Interface); or monitor the home function from the desktop launcher; or monitor the response window; or filter invalid events through the SystemUI. Finally, it is converted into the form specified by the event information.

[0141] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0142] Fig.10 is a schematic diagram of the structure of a device for generating an operation trajectory model provided in an embodiment of the present application. Fig.10 , the device includes: an information acquisition module 1001 and a model generation module 1002.

[0143] The information acquisition module 1001 is used to acquire interface information and event information, wherein the interface information is used to indicate multiple interfaces in the operation track, and the event information is used to indicate at least one operation behavior in the operation track;

[0144] The model generation module 1002 is used to generate an operation trajectory model based on the interface information and event information. The operation trajectory model includes multiple nodes and directed edges between nodes. The nodes are used to represent the interfaces, and the directed edges between the nodes are used to represent the jump relationship between the interfaces based on the operation behavior.

[0145] In some embodiments, the information acquisition module 1001 is used to acquire input events and system events in real time based on accessibility services and system gesture monitoring management services to obtain first data in the event information; in response to changes in the interface caused by the first data, the interface changes are acquired in real time based on accessibility services to obtain second data in the interface information.

[0146] In some embodiments, the model generation module 1002 is used to determine two nodes in the operation trajectory model in real time based on the second data, the two nodes respectively representing two interfaces before and after the change; and to construct a directed edge between the two nodes in real time based on the first data.

[0147] In some embodiments, the information acquisition module 1001 is used to obtain input events and system events based on accessibility services and system gesture monitoring management services, and obtain event information, which includes multiple events; based on accessibility services, obtain changes in the interface and obtain interface information, which includes multiple interfaces.

[0148] In some embodiments, the model generation module 1002 is used to generate an operation behavior stream based on the interface information and event information, where the operation behavior stream is used to indicate the temporal relationship between the interface and the event; and to generate an operation trajectory model based on the operation behavior stream.

[0149] In some embodiments, the information acquisition module 1001 is used to obtain changes in the interface based on accessibility services and obtain interface information, which includes multiple interfaces; determine the behavior type of the operation behavior corresponding to each control according to the control type of each control in the multiple interfaces; filter out controls that cannot be operated to obtain event information, which includes multiple events, and different events correspond to different operation behaviors.

[0150] In some embodiments, the information acquisition module 1001 is used to input the task information and the current interface into the large language model for the i-th event determination process to obtain an event in the event information, and the large language model is used to predict events based on the interface and task information; based on accessibility services, obtain changes in the interface to obtain an interface in the interface information, and the interface information includes multiple interfaces.

[0151] In some embodiments, the model generation module 1002 is further used to generate an operation trajectory path based on the large language model, and the operation trajectory path is used to indicate the temporal relationship between the interface and the event.

[0152] In some embodiments, the model generation module 1002 is used to determine the nodes in the operation trajectory model based on the interface information and through the node grouping strategy, and the node grouping strategy is used to divide the nodes corresponding to the interfaces belonging to the same user interface activity into the same node group; based on the event information, determine the directed edges between the nodes in the operation trajectory model.

[0153] In some embodiments, the model generation module 1002 is further used to add at least one leading edge in front of any node when a leading operation is required for the interface corresponding to the node to jump to another interface.

[0154] In some embodiments, the model generation module 1002 is further used to obtain screenshots, skeleton diagrams, and interface compression trees of multiple interfaces based on accessibility services and system gesture monitoring management services.

[0155] In some embodiments, the information acquisition module 1001 is used to perform any of the following:

[0156] Based on the system debugging function, determine the interface information of multiple interfaces;

[0157] Based on the barrier-free server, screenshots of the multiple interfaces are obtained, the screenshots of the multiple interfaces are converted into an interface tree structure, and based on the interface tree structure, interface information of the multiple interfaces is determined;

[0158] The structured data of at least one application is parsed to obtain interface information of multiple interfaces, and the at least one application includes multiple interfaces.

[0159] The device for generating an operation trajectory model provided in an embodiment of the present application obtains the interfaces passed through and the operation behaviors used in the operation trajectory, and then uses nodes to represent the interfaces and directed edges to represent the jump relationships between interfaces based on the operation behaviors, thereby constructing a graph model, that is, an operation trajectory model. The operation trajectory model can intuitively represent the operation trajectory from one interface to another.

[0160] An embodiment of the present application provides a computer device, which includes a processor and a memory; the memory stores at least one program code, and the at least one program code is used to be executed by the processor to implement the method for generating an operation trajectory model provided by the above-mentioned various method embodiments.

[0161] A computer device may be configured as a terminal, Fig.11 11 is a block diagram of a terminal provided in an embodiment of the present application. In some embodiments, the terminal 1100 is a terminal such as a smart phone, a tablet computer, a wearable device, etc. that can access a wireless local area network as a wireless station. The terminal 1100 in the present application includes at least one or more of the following components: a processor 1110, a memory 1120, and at least two wireless links 1130.

[0162] In some embodiments, the processor 1110 includes one or more processing cores. The processor 1110 uses various interfaces and lines to connect various parts of the entire terminal 1100, and executes various functions and processes data of the terminal 1100 by running or executing program codes stored in the memory 1120, and calling data stored in the memory 1120. In some embodiments, the processor 1110 is implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 1110, but may be implemented by a separate chip.

[0163] In some embodiments, the processor 1110 is used to control the working conditions of at least two wireless links 1130. Accordingly, the processor 1110 is a processor integrated with a wireless fidelity (Wireless Fidelity, Wi-Fi) chip. The Wi-Fi chip is a chip with dual Wi-Fi processing capabilities. For example, the Wi-Fi chip is a dual-band dual concurrent (Dual Band Dual Concurrent, DBDC) chip, or a dual-band synchronous (Dual Band Simultaneous, DBS) chip, etc.

[0164] In some embodiments, the memory 1120 includes a random access memory (RAM), and in some embodiments, the memory 1120 includes a read-only memory (ROM). In some embodiments, the memory 1120 includes a non-transitory computer-readable storage medium. The memory 1120 can be used to store program codes. The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the terminal 1100 (such as audio data, a phone book), etc.

[0165] In some embodiments, the memory 1120 stores reception schemes for receiving beacon frames of different wireless links 1130, as well as identifiers of access nodes connected to different wireless links 1130, identifiers of wireless links 1130, and the like.

[0166] The at least two wireless links 1130 are used to connect different access points (APs) to receive downlink data sent by the APs, wherein the different access points are access points in the same router or access points in different routers.

[0167] In some embodiments, the terminal 1100 also includes a display screen. The display screen is a display component for displaying a user interface. In some embodiments, the display screen is a display screen with a touch function, through which the user can use any suitable object such as a finger or a touch pen to perform touch operations on the display screen. In some embodiments, the display screen is generally arranged on the front panel of the terminal 1100. In some embodiments, the display screen is designed to be a full screen, a curved screen, a special-shaped screen, a double-sided screen or a folding screen. In some embodiments, the display screen is also designed to be a combination of a full screen and a curved screen, a combination of a special-shaped screen and a curved screen, etc., which is not limited in this embodiment.

[0168] In addition, those skilled in the art will appreciate that the structure of the terminal 1100 shown in the above drawings does not limit the terminal 1100, and the terminal 1100 includes more or fewer components than shown in the drawings, or combines certain components, or arranges components differently. For example, the terminal 1100 also includes components such as a microphone, a speaker, an input unit, a sensor, an audio circuit, a module, a power supply, and a Bluetooth module, which will not be described in detail here.

[0169] The present application also provides a computer-readable storage medium, which stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the method for generating an operation trajectory model as shown in the above embodiments.

[0170] The present application also provides a computer program product, which stores at least one program code, and the at least one program code is used to be executed by a processor to implement the method for generating an operation trajectory model as shown in the above embodiments.

[0171] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0172] Those skilled in the art can understand that all or part of the steps in the method for generating the operation trajectory model of the above embodiment can be completed by hardware, or can be completed by instructing related hardware through a program, and the above program can be stored in a computer-readable storage medium, and the above storage medium can be a read-only memory, a disk or an optical disk, etc. The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating an operation trajectory model, characterized in that: The method comprises: Acquire interface information and event information, wherein the interface information is used to indicate multiple interfaces in the operation track, and the event information is used to indicate at least one operation behavior in the operation track; Based on the interface information and the event information, an operation trajectory model is generated, wherein the operation trajectory model includes a plurality of nodes and directed edges between the nodes, wherein the nodes are used to represent interfaces, and the directed edges between the nodes are used to represent jump relationships between interfaces based on operation behaviors.

2. The method according to claim 1, characterized in that The obtaining of interface information and event information includes: Based on the accessibility service and the system gesture monitoring management service, input events and system events are acquired in real time to obtain the first data in the event information; In response to the interface change caused by the first data, the interface change is acquired in real time based on the accessibility service to obtain the second data in the interface information.

3. The method according to claim 2, characterized in that The generating an operation trajectory model based on the interface information and the event information includes: Based on the second data, two nodes are determined in the operation trajectory model in real time, wherein the two nodes respectively represent two interfaces before and after the change; Based on the first data, a directed edge is constructed between the two nodes in real time.

4. The method according to claim 1, characterized in that: The obtaining of interface information and event information includes: Based on the accessibility service and the system gesture monitoring management service, input events and system events are acquired to obtain the event information, where the event information includes multiple events; Based on the barrier-free service, the change of the interface is acquired to obtain the interface information, where the interface information includes the multiple interfaces.

5. The method according to claim 4, characterized in that The generating an operation trajectory model based on the interface information and the event information includes: Based on the interface information and the event information, generating an operation behavior flow, wherein the operation behavior flow is used to indicate a temporal relationship between an interface and an event; Based on the operation behavior flow, the operation trajectory model is generated.

6. The method according to claim 1, characterized in that The obtaining of interface information and event information includes: Based on the barrier-free service, acquiring the change of the interface, and obtaining the interface information, wherein the interface information includes the multiple interfaces; Determining, according to the control type of each control in the multiple interfaces, the behavior type to which the operation behavior corresponding to each control belongs; Controls that cannot be operated are filtered out to obtain the event information, where the event information includes multiple events, and different events correspond to different operation behaviors.

7. The method according to claim 1, characterized in that The obtaining of interface information and event information includes: For the i-th event determination process, the task information and the current interface are input into the large language model to obtain an event in the event information, and the large language model is used to predict the event based on the interface and task information; Based on the barrier-free service, a change of the interface is acquired to obtain an interface in the interface information, where the interface information includes the multiple interfaces.

8. The method according to claim 7, characterized in that The method further comprises: Based on the large language model, an operation trajectory path is generated, where the operation trajectory path is used to indicate a temporal relationship between an interface and an event.

9. The method according to claim 1, characterized in that: The generating an operation trajectory model based on the interface information and the event information includes: Based on the interface information, determining nodes in the operation trajectory model through a node grouping strategy, wherein the node grouping strategy is used to divide nodes corresponding to interfaces belonging to the same user interface activity into the same node group; Based on the event information, directed edges between nodes in the operation trajectory model are determined.

10. The method according to claim 9, characterized in that The method further comprises: For any node, when a pre-operation is required for the interface corresponding to the node to jump to another interface, at least one pre-edge is added before the node.

11. The method according to claim 1, characterized in that: The method further comprises: Based on the accessibility service and the system gesture monitoring management service, screenshots, skeleton diagrams, and interface compression trees of the multiple interfaces are obtained.

12. The method according to claim 1, characterized in that The method of obtaining the interface information includes any of the following: Based on the system debugging function, determining the interface information of the multiple interfaces; Obtaining screenshots of the multiple interfaces based on the barrier-free server, converting the screenshots of the multiple interfaces into an interface tree structure, and determining interface information of the multiple interfaces based on the interface tree structure; The structured data of at least one application is parsed to obtain interface information of the multiple interfaces, and the at least one application includes the multiple interfaces.

13. A device for generating an operation trajectory model, characterized in that: The device comprises: An information acquisition module, used to acquire interface information and event information, wherein the interface information is used to indicate multiple interfaces in the operation track, and the event information is used to indicate at least one operation behavior in the operation track; A model generation module is used to generate an operation trajectory model based on the interface information and the event information, wherein the operation trajectory model includes multiple nodes and directed edges between the nodes, the nodes are used to represent the interface, and the directed edges between the nodes are used to represent the jump relationship between the interfaces based on the operation behavior.

14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the method for generating an operation trajectory model according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the method for generating an operation trajectory model according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method for generating an operation trajectory model as described in any one of claims 1 to 12.

Images