A device-independent method for recording and replaying user operation trajectories in Android apps

Abstract

The present invention discloses a device-independent Android APP user operation trajectory recording and playback method, which connects to an Android device to obtain basic information, monitors the control information of the current screen active view and user operation events in real time, and converts them into a standardized logical structure. According to the control layout of the current screen active view, a structured view model is constructed, which covers all the control information of the page. For user operation events, a mapping relationship between them and the page view controls is generated, and each user operation event and its corresponding control information are obtained. Starting from the first user operation event, each user operation event and its corresponding control information are recorded, and an operation trajectory sequence is gradually generated. In the trajectory playback stage, the APP is restarted and adjusted to the initial interface, and the playback of all actions is achieved by traversing the operation trajectory sequence in sequence. The method described in the present invention realizes the efficient recording and accurate playback of device-independent Android APP user operation trajectories.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology for Android apps, specifically to a device-independent method for recording and replaying user operation trajectories in Android apps. Background Technology

[0002] With the widespread adoption of smartphones, apps have become ubiquitous in daily life. To optimize user experience, the demand for automated testing of Android apps is constantly increasing, especially in the field of user behavior analysis. A key step in user behavior analysis is recording user action patterns on the device and subsequently replaying these actions to accurately reproduce user behavior. Furthermore, recording user action patterns also plays a crucial role in app traffic fingerprinting analysis. By replaying user action patterns, a large amount of user traffic data can be collected, providing data support for research such as traffic pattern recognition and user preference prediction.

[0003] However, existing operation trajectory recording and playback technologies have certain limitations. Existing technologies rely on specific device hardware and system environments, resulting in inconsistencies in the same operation trajectory across different devices. Specifically, existing technologies record touch events based on absolute screen coordinates during event recording. If the screen size or resolution of the playback device differs, the same coordinates may not trigger the expected control operation. This limitation can lead to poor performance in cross-device testing, necessitating a device-independent operation trajectory recording and playback technology to ensure consistent operation reproduction across different devices. To address this, we propose a device-independent method for recording and playing back user operation trajectories in Android apps. Summary of the Invention

[0004] To address the aforementioned technical problems, a device-independent method for recording and playing back user operation trajectories in an Android app is provided. This technical solution resolves the problems described above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A device-independent method for recording and playing back user operation trajectories in an Android app, comprising the following steps:

[0007] S1: Connect to an Android device via USB or Wi-Fi to obtain basic device information; monitor control information and user operation events in the current active view of the screen in real time, and convert them into a standardized logical structure;

[0008] S2: Based on the control layout of the current screen activity view, extract all its control information and build a structured view model for all pages that have been viewed in the Android APP, which includes all control information of the page;

[0009] S3: Based on the view model and real-time listening of user operation events, for swipe and system key events, it generates a mapping relationship between user operation events and page view controls, and obtains each user operation event and its corresponding control information;

[0010] S4: Starting from the first user operation event when the user launches the Android APP, record each user operation event and its corresponding control information, and gradually generate a complete operation trajectory sequence;

[0011] S5: During the trajectory playback stage, restart the Android APP and adjust it to the initial interface, then iterate through the operation trajectory sequence to replay all actions.

[0012] Preferably, in step S1:

[0013] The basic information includes screen resolution, operating system version, device ID, and device name;

[0014] The user interaction events include clicks, swipes, system key presses, and text input;

[0015] The standardized logical structure stores event representations in the form of triples:

[0016]

[0017] In the formula, Indicates the timestamp of the event. Indicates the event type, Indicates event parameters;

[0018] If the event type is click, then Represents the coordinates of the touch point; if the event type is swipe, then... This represents the sequence of coordinates for the start and end points of the slide. If the event type is a system button, then... This indicates the keystroke code captured by the system framework. If the event type is text input, then... This includes the control's center position and text content.

[0019] Preferably, in step S1, the condition for determining the click event is:

[0020] When the parameter of a user action event contains only one coordinate, it indicates that the user has performed a single-point touch action. This event is a screen click operation, and its parameter is the screen coordinate of the touch point. If the touch point is located within the boundary of a clickable control and a view change is triggered after the touch event, the operation event will eventually be classified as a valid click event. Click operations that do not meet the above conditions will be considered invalid and discarded.

[0021] Preferably, in step S1, the judgment condition for the swipe event is: its parameters include the coordinate sequence of the starting and ending points of the swipe, and a series of continuous coordinate pairs, used to represent the change of the touch path and reflect that the user has performed a screen swipe operation. Let the swipe path be:

[0022]

[0023] From Move to The path is simplified to:

[0024]

[0025] If the sliding path length A swipe event is classified as valid if it exceeds the preset minimum swipe distance and causes a change in the view or a shift in the control's position. Swipe events that do not meet these conditions will be considered invalid and discarded.

[0026] Preferably, in step S1, the judgment condition for the text input event is:

[0027] The system first analyzes the UI controls in the interface to determine whether the target control supports text input;

[0028] When a control that supports text input is active, the system monitors changes in the input characters during the event triggering process, records the complete sequence of user input, and confirms that the text content of the target control has been successfully updated to the user input by analyzing the view response after input.

[0029] When the above conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as a text input parameter. Text input operations that do not meet the above conditions will be considered invalid and discarded.

[0030] Preferably, in step S2, the structured view model is constructed through the following steps:

[0031] Use XPath expressions to parse the UI hierarchy displayed on the device screen and capture all clickable controls;

[0032] UIAutomator is used to extract basic information for each control, including but not limited to: position, size, text content, package name, and resource ID. Each control is ultimately represented as a triple.

[0033]

[0034] In the formula, Indicates the boundary information of the control. This indicates the identifier of the constructed control. Represents the text property of the control;

[0035] Filter the extracted UI controls and remove controls that users do not frequently interact with, controls that are not clickable, and controls that are not scrollable;

[0036] For each control, its information Represented as:

[0037]

[0038] In the formula, , , and These represent the four boundary vertices of the control;

[0039] Basic identifiers of controls Represented as:

[0040]

[0041] In the formula, control The application package name, control Resource ID, control The class name;

[0042] Control properties It contains two sub-attributes: text content and status information.

[0043] Define the extended identifier of the control , is represented as:

[0044]

[0045] in It is a control Basic identifiers, This indicates the current control's position in the list of child controls of its parent control. The basic identifier representing all parent controls of the current control. The set, The basic identifier representing all clickable controls on the current page. A set of.

[0046] Preferably, in step S3, for click events, the specific control is located using the touch coordinates in the event parameters, and the coordinates are mapped to the control by matching the control's boundary information in conjunction with the page's view model; for text input events, control positioning is completed by real-time monitoring of changes in input focus.

[0047] Preferably, in step S4, the system uses different methods to record and organize events for different categories;

[0048] For click and text input events, once the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the event record data and the interval since the last control operation.

[0049] For swipe and system key events, the user operation event log only contains the event log data and the interval since the last control operation;

[0050] Starting from the first operation event when a user launches an Android application, the system monitors and organizes user operation event records in real time. Each record is encapsulated as a node in a linked list, eventually generating a complete operation trajectory sequence.

[0051] Preferably, in step S5, during the trajectory playback stage, the system first launches the target Android application via command and ensures that its interface state is consistent with the initial interface of the user's operation trajectory record.

[0052] Once the application state adjustment is complete, the system begins to replay each node in the operation trajectory sequence sequentially. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into an executable simulated operation.

[0053] After each operation replay, the system monitors the changes in the interface state to ensure that the operation result is consistent with the trajectory record. If the interface state is detected to deviate from the expectation, the system will attempt to correct it through a preset recovery strategy to ensure that subsequent operations can be executed smoothly. The entire replay process is carried out in chronological order until all operation records have been traversed and successfully executed. The recovery strategy includes restarting the application or returning to the initial interface.

[0054] Preferably, it also includes a device-independent Android APP user operation trajectory recording and playback system, which internally includes: an operation event listening and recording module, a view construction module, a user operation and control mapping module, an operation trajectory sequence generation module, and a path playback module;

[0055] The operation event listening and recording module is used to initialize the device and environment, and at the same time listen for and record valid operation events in real time;

[0056] The view building module is used to record the structural information of the application interface, including page information, control information and their position information, control identifiers and control text attributes, and finally generate the view model;

[0057] The User Operation and Control Mapping Module is used to obtain the mapping relationship between events and controls for click and text input events, using the event representation information obtained from the Operation Event Listening and Recording Module and the view model generated by the View Building Module.

[0058] The operation trajectory sequence generation module is used to record each operation event and its associated control information starting from the first user operation event of the Android APP, and to construct a complete operation trajectory sequence by using each operation information as a node;

[0059] The path replay module is used to gradually reconstruct user operations from the initial page based on the sequence of operation trajectories.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The Android APP user operation trajectory recording and playback method proposed in this invention listens to and records user operation events in real time, and simultaneously constructs a structured view model for all browsing pages of the Android APP. This method can accurately map user operation events to specific page view controls. This device-independent characteristic overcomes the limitations of existing technologies that rely on specific device hardware and system environments. It enables the same operation trajectory to accurately trigger the expected control operation on devices with different screen sizes or resolutions. By storing event identifiers through a standardized logical structure, the standardization and readability of event recording are ensured. By monitoring changes in the interface state, the consistency between the operation result and the trajectory record is ensured. If the interface state is detected to deviate from the expectation, the system will also attempt to correct it through a preset recovery strategy, thereby ensuring that subsequent operations can be executed smoothly, improving the efficiency and accuracy of automated testing of Android APP. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the device-independent Android APP user operation trajectory recording and playback system in an example of the present invention.

[0063] Figure 2 This is a diagram illustrating the mapping rules between operation events and controls, and the recording format of operation events in an example of the present invention.

[0064] Figure 3 This is a schematic diagram of trajectory playback in an example of the present invention. Detailed Implementation

[0065] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0066] A device-independent method for recording and playing back user operation trajectories in an Android app includes: S1: Connecting to an Android device via USB or Wi-Fi to obtain basic device information; real-time monitoring of control information and user operation events in the current active view of the screen, and converting them into a standardized logical structure.

[0067] Connect your Android device via USB or Wi-Fi and obtain basic device information, including screen resolution, operating system version, device ID, and device name, through the ADB (Android Debug Bridge) interface.

[0068] When a user performs an event operation on an Android device, the system executes the "getevent" command through the ADB interface to obtain the raw event data stream from the device's input layer. This data contains detailed information about low-level interactive events such as screen touch and button operations. The system then parses the captured event data in real time and categorizes it into four types: clicks, swipes, system buttons, and text input.

[0069] A click event represents a user's action of touching a clickable control on the screen. When the parameter of a user action event contains only one coordinate, it indicates that the user has performed a single-point touch, which is a screen click operation. The parameter is the screen coordinate of the touch point. If the touch point is within the boundary of a clickable control and a view change is triggered after the touch event, then the operation event is ultimately classified as a valid click event. View changes can be determined by the subsequently constructed view model, defining different views based on the different controls they contain. Click operations that do not meet the above conditions will be considered invalid and discarded.

[0070] A swipe event refers to a series of interactive operations performed by a user through screen dragging gestures, used to describe the changes in the touch trajectory. Its parameters typically include the starting and ending coordinate sequences of the swipe, and a series of consecutive coordinate pairs to represent the changes in the touch path, reflecting the user's screen swipe operation. If there is a swipe path... , From Move to The path can be simplified to If the sliding path length A swipe event is classified as valid if it exceeds the preset minimum swipe distance and causes a change in the view or a shift in the control's position. Swipe events that do not meet these conditions will be considered invalid and discarded.

[0071] System key events refer to user actions such as pressing physical or virtual system keys, including the back, home, and volume keys. These events are independent of specific view controls and do not depend on touch path matching; rather, they are global functions provided by the device's operating system. System key events are captured by the ADB interface, and their key type is identified by a key code, which is reflected in the parameters of the user operation event.

[0072] Text input events refer to user interactions with input fields on the interface to input characters, numbers, or symbols. The system first analyzes the UI controls to determine if the target control supports text input. When a control supporting text input is active (i.e., has input focus), the system monitors changes in the input characters during the event triggering process and records the complete sequence of user input. Simultaneously, the system analyzes the view response after input to confirm that the text content of the target control has been successfully updated to reflect the user's input. If these conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as a text input parameter. Text input operations that do not meet these conditions are considered invalid and discarded.

[0073] After parsing the event log data, the system obtains the event representation stored in the form of triples:

[0074]

[0075] in, The timestamp of the event. For event type, This is an event parameter. If the event type is click, then... These are the coordinates of the touch point; if the event type is swipe, then... This is the sequence of coordinates for the start and end points of the slide; if the event type is a system button, then... This refers to the keystroke code captured by the system framework; if the event type is text input, then... This includes the center position of the control and the text content. The above data only stores event log data and does not involve the identifier of the manipulated control. The mapping relationship between events and controls can be obtained by matching it with page information later.

[0076] S2: Based on the control information of the current active screen view, associate it with the interaction information to build a structured view model for all pages that have been viewed in the Android APP, which contains all the control information of the page.

[0077] First, XPath expressions are used to parse the UI hierarchy displayed on the device screen, capturing all clickable controls. Then, UIAutomator is used to extract basic information about each control (including position, size, text content, package name, resource ID, etc.), ultimately representing each control as a triple:

[0078]

[0079] in For the boundary information of the control, The identifier for the constructed control. This refers to the text property of the control.

[0080] The system will filter the extracted interface controls, removing those that users rarely interact with, such as input method keyboards and emoji selection boxes. Additionally, the system will remove non-clickable and non-scrollable controls, such as static label text and tooltips. Then, for each control, its information... It can be represented in the following form:

[0081]

[0082] in , , and These represent the four boundary vertices of the control, describing its specific position on the screen.

[0083] Basic identifiers of controls , can be represented as:

[0084]

[0085] in control The application package name, control Resource ID, control The class name, the combination of these three elements ensures that each control is globally unique in the view tree.

[0086] Control properties This property contains two sub-properties: text content and status information. Text content includes button labels, text box tooltips, etc. If the control contains text content, the system will associate the text content with the control's position boundaries. If the control does not have text content, this property will be empty. Status information includes the control's current interactive state, such as whether it is selected or available. For example, an instance of a control's property might look like this:

[0087]

[0088] The text content of this control is The optional value is empty, and the status information is that interaction is allowed.

[0089] During the generation or updating of the view model, whenever a user browses a new page, the system parses and extracts all controls on that page, records the key attributes of each control, and combines them into the aforementioned triplet description. Furthermore, the layout of controls on the page may change with user actions (e.g., pop-ups, scrolling). Therefore, after each action, the system will reread the page information and update the view model.

[0090] Subsequently, in order to locate the same control on different devices, an extended identifier for the control was defined. . A control that can uniquely identify different controls is represented as:

[0091]

[0092] in It is a control Basic identifiers, This indicates the current control's position in the list of child controls of its parent control. The basic identifier representing all parent controls of the current control. The set, The basic identifier representing all clickable controls on the current page. A collection of these information tags. Through these tags, the extended identifier of a control can uniquely identify different controls.

[0093] S3: Based on the view model and real-time listening of user operation events, for swipe and system key events, it generates a mapping relationship from user operation events to controls on the page view, and obtains the control information of each user operation event and its operation.

[0094] User interaction events can be categorized into four types: clicks, swipes, system buttons, and text input. Only clicks and text input require locating the target control. Swipes affect the entire view screen, while system button events are captured by the system framework and identified through button codes, and are not directly related to any specific control.

[0095] For click events, the specific control is located using the touch coordinates in the event parameters. The raw coordinates only reflect the user's specific touch point on the screen and cannot be directly associated with the specific control on the page. To map the touch coordinates to the control, the system combines the page's view model and achieves the mapping by matching the control's boundary information. During the matching process, the system reads all control information in the current page's view table and compares them one by one with the coordinates of the click event. If a click event exists... coordinates and controls its location information If it exists and This indicates the event Clicking control .

[0096] For text input events, control location is achieved by monitoring changes in input focus in real time. The system determines the target control for input by detecting the control currently receiving focus. Furthermore, if a change in the input character is detected during the event, it further confirms that the control is the one actually performing text input.

[0097] S4: Starting from the first user operation event when the user launches the Android APP, record each user operation event and the control information it operates on, thereby generating an operation trajectory sequence.

[0098] The system employs different recording methods for different types of events. For click and text input events, once the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the event record data mentioned above and the interval since the last control operation. For swipe and system button events, the user operation event record only includes the event record data and the interval since the last control operation.

[0099] Starting with the first user action when launching the Android application, the system monitors and organizes user action event records in real time. Each record is encapsulated as a node in a linked list. As user actions occur, new records are dynamically added to the end of the linked list, ensuring that the action trajectory is arranged in chronological order. Ultimately, the system generates a complete sequence of action trajectories based on the order of events. This sequence of action trajectories is saved for later playback during the replay phase.

[0100] S5: During the trajectory playback stage, restart the Android APP and adjust it to the initial interface, then iterate through the operation trajectory sequence to replay all actions.

[0101] During the trajectory playback phase, the system first launches the target Android application via command and ensures that its interface state is consistent with the initial interface recorded in the user's operation trajectory. This is typically achieved by clearing the application's cache data or sending a command to restart the application to avoid interference from the historical operation state in the playback process.

[0102] Once the application state adjustment is complete, the system begins to replay each node in the operation trajectory sequence sequentially. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into an executable simulated operation. For click events, the system uses the extended identifier of the recorded control. The system locates controls and executes click operations; swipe events generate swipe paths using start and end coordinates in a trajectory sequence, then send continuous touch commands at appropriate time intervals to complete the action; system button events directly call the device control interface to complete the operation; for text input events, the system also uses the extended identifier of the control. Position the control and enter the recorded text content.

[0103] After each operation replay, the system monitors changes in the interface state to ensure consistency between the operation result and the recorded trajectory. If the interface state deviates from expectations, the system attempts to correct it using a preset recovery strategy (such as restarting the application or returning to the initial interface) to ensure subsequent operations can be executed smoothly. Furthermore, the system strictly controls the execution time of adjacent operations according to the time intervals recorded in the operation trajectory sequence. The entire replay process proceeds chronologically until all operation records have been traversed and successfully executed.

[0104] Reference Figure 1 As shown, steps S1 to S5 describe in detail the operation process of the device-independent Android APP user operation trajectory recording and playback system. The system is designed to achieve comprehensive recording and accurate playback of Android APP user operations through the collaborative work of multiple functional modules. Specifically, it uses an operation event listening module to capture various interactive actions between the user and the APP, and combines this with a view construction module to extract detailed information about the APP page and various controls, ultimately forming an operation trajectory sequence, and using this sequence to complete the playback of all operation events.

[0105] A device-independent Android app user operation trajectory recording and playback system can be effectively used to implement the Android app user operation trajectory recording and playback method in the above embodiments. The system consists of five key modules: operation event listening and recording module, view construction module, user operation and control mapping module, operation trajectory sequence generation module, and path playback module.

[0106] The operation event listening and recording module plays a crucial role in initializing the device and environment at the beginning of the system's operation. It ensures that all device parameters and the operating environment meet the requirements of subsequent operations, monitors all user actions on the Android app in real time, and accurately records valid operation events. Valid operation events encompass various meaningful user interactions on the app interface, such as tapping the screen, swiping pages, and entering text.

[0107] The view construction module's primary responsibility is to meticulously record the structural information of the application interface. It not only focuses on the overall layout of the page but also delves into the details of each control, including its specific information, its position on the page, its unique identifier, and the text attributes it displays. By collecting and organizing this information, this module generates a comprehensive view model that clearly presents the various components of the interface and their interrelationships.

[0108] The user action and control mapping module primarily focuses on the two most common and important event types: clicks and text input. It cleverly utilizes event representation information obtained from the action event listening and recording module, along with a carefully generated view model from the view building module. Through a series of complex and precise algorithms and logic, it successfully obtains the mapping relationship between events and controls, accurately matching user actions with specific controls on the app interface. This lays a solid foundation for the accurate recording and playback of subsequent operation trajectories.

[0109] The operation trajectory sequence generation module begins its meticulous recording work from the very first user operation event triggered by the Android app. It captures every operation event, along with the associated control information. Then, treating each operation as a node, it connects these nodes in an orderly manner using a specific organizational method, ultimately constructing a complete operation trajectory sequence. This sequence acts like a timeline, comprehensively recording the user's operation process within the app, from the initial action to the final outcome, with every step clearly traceable.

[0110] The path replay module operates based on the sequence of user actions. Starting from the initial page, it recreates the user's actions on the app step by step, following the steps recorded in the sequence. Whether it's clicking a control or typing text, it accurately reproduces the actions, as if the user were re-enacting the entire process on the app, providing a highly realistic replay experience.

[0111] Reference Figure 2As shown, the system employs a rigorous and meticulous processing flow for click and text input events during operation. Once the system successfully links the user's real-time actions with the layout information of page controls, it enters the information integration phase. At this stage, the system comprehensively collects and integrates key information from various aspects, including detailed information about the user's actions themselves, such as the intensity of the click and the specific content of the text input; control information, such as the control's type, function, and page position; and the time interval since the last operation on that control. This information is deeply integrated and saved as a complete user action event record. This record comprehensively and accurately records the relevant information of a specific operation on a specific control, providing a solid data foundation for subsequent analysis, backtracking, and system optimization.

[0112] For swipe and system button events, the user operation event logs are relatively simple. These logs only contain event data, including the direction and distance of the swipe, the type of system button pressed, and the time interval since the last control operation. This recording method satisfies the need to record key information for these two types of events without increasing the system load with excessive redundant information, ensuring the efficiency and relevance of the system's recording.

[0113] Regarding the specific location of controls, for click events, the system employs a method based on the touch coordinates in the event parameters to accurately locate the specific control. When a user performs a click operation on the screen, the system quickly captures the touch coordinate information at the moment of the click. Then, based on the pre-constructed page control layout information, a precise coordinate matching algorithm compares the touch coordinates with the position areas of each control, thereby accurately determining which control the user clicked. This coordinate-based positioning method is as precise and efficient as determining a specific location on a map using latitude and longitude, providing accurate target guidance for subsequent click event processing.

[0114] For text input events, the system employs a strategy of real-time monitoring of input focus and changes in input characters to locate controls. When a user begins text input, the system continuously monitors changes in the input focus position on the page. Once the input focus lands on a specific control, that control becomes the target of the current text input. Simultaneously, the system continuously monitors real-time changes in input characters to accurately record the user's input. Through this dual monitoring mechanism of input focus and input character changes, the system can promptly and accurately identify the specific control associated with the text input event, ensuring that the recording and processing of text input operations are precisely mapped to the corresponding control.

[0115] Reference Figure 3As shown, during the trajectory replay phase, the Android app is restarted and returned to its initial interface. Each node in the operation trajectory sequence is then replayed sequentially. After each replay, the system monitors changes in the interface state to ensure the operation result matches the recorded trajectory. If the interface state deviates from expectations, the system returns to the initial interface and re-executes the operation.

[0116] The process of using this invention is as follows: connect the device, acquire and convert information, extract control information, build a view model, generate the mapping relationship between operation and control, record the operation trajectory sequence, and replay the operation trajectory.

[0117] In summary, the advantages of this invention are: it improves the efficiency and accuracy of automated testing for Android apps, and provides strong technical support for fields such as user behavior analysis and app traffic fingerprinting analysis. By recording and replaying user operation trajectories, user behavior can be accurately reproduced, and a large amount of user traffic data can be collected, providing data support for research such as traffic pattern recognition and user preference prediction.

[0118] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A device-independent method for recording and playing back user operation trajectories in an Android app, characterized in that, Includes the following steps: S1: Connect to an Android device via USB or Wi-Fi to obtain basic device information; Monitor the control information and user operation events of the current active view on the screen in real time, and transform them into a standardized logical structure; The basic information includes screen resolution, operating system version, device ID, and device name; The user interaction events include clicks, swipes, system key presses, and text input; The standardized logical structure stores event information in the form of triples: In the formula, Indicates an event The timestamp of the occurrence Indicates the event type, Indicates event parameters; If the event type is click, then Represents the coordinates of the touch point; if the event type is swipe, then... This represents the sequence of coordinates for the start and end points of the slide. If the event type is a system button, then... This indicates the keystroke code captured by the system framework. If the event type is text input, then... This includes the control's center position and text content; S2: Based on the control layout of the current screen activity view, extract all its control information and build a structured view model for all pages that have been viewed in the Android APP, which contains all control information of the page; S3: Based on the view model and real-time listening of user operation events, for swipe and system key events, it generates a mapping relationship between user operation events and page view controls, and obtains each user operation event and its corresponding control information; S4: Starting from the first user operation event when the user launches the Android APP, record each user operation event and its corresponding control information, and gradually generate a complete operation trajectory sequence; S5: Trajectory playback stage, restart the Android APP and adjust it to the initial interface, and traverse the operation trajectory sequence in turn to replay all actions; During the trajectory playback phase, the system first launches the target Android application via command and ensures that its interface state is consistent with the initial interface recorded by the user's operation trajectory. Once the application state adjustment is complete, the system begins to replay each node in the operation trajectory sequence sequentially. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into an executable simulated operation. After each operation replay, the system monitors the changes in the interface state to ensure that the operation result is consistent with the trajectory record. If the interface state is detected to deviate from the expectation, the system will attempt to correct it through a preset recovery strategy to ensure that subsequent operations can be executed smoothly. The entire replay process is carried out in chronological order until all operation records have been traversed and successfully executed. The recovery strategy includes restarting the application or returning to the initial interface.

2. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S1, the process of determining the click event is as follows: When the parameter of a user action event contains only one coordinate, it indicates that the user has performed a single-point touch action. This event is a screen click operation, and its parameter is the screen coordinate of the touch point. If the touch point is located within the boundary of a clickable control and a view change is triggered after the touch event, the user action event will eventually be classified as a valid click event. Click operations that do not meet the above conditions will be considered invalid and discarded.

3. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S1, the judgment condition for the swipe event is: its parameters include the coordinate sequence of the start and end points of the swipe, and a series of consecutive coordinate pairs to represent the change of the touch path, reflecting that the user has performed a screen swipe operation. Let the swipe path be: From Move to The path is simplified to: If the sliding path length If the sliding distance exceeds the preset minimum sliding distance, and the sliding operation causes a change in the view or an offset in the control position, the user operation event is classified as a valid sliding event. Slide operations that do not meet the above conditions will be considered invalid and discarded.

4. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S1, the condition for determining the text input event is: The system first analyzes the UI controls in the interface to determine whether the target control supports text input; When a control that supports text input is active, the system monitors changes in the input characters during the event triggering process, records the complete sequence of user input, and confirms that the text content of the target control has been successfully updated to the user input by analyzing the view response after input. When the above conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as a text input parameter. Text input operations that do not meet the above conditions will be considered invalid and discarded.

5. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S2, the structured view model is constructed through the following steps: Use XPath expressions to parse the UI hierarchy displayed on the device screen and capture all clickable controls; UIAutomator is used to extract basic information for each control, including its position, size, text content, package name, and resource ID. Each control is ultimately represented as a triple. In the formula, Indicates the boundary information of the control. This indicates the identifier of the constructed control. Represents the text property of the control; Filter the extracted UI controls and remove controls that users do not frequently interact with, controls that are not clickable, and controls that are not scrollable; For each control, its information Represented as: In the formula, , , and These represent the four boundary vertices of the control; Basic identifiers of controls Represented as: In the formula, control The application package name, control Resource ID, control The class name; Control properties It contains two sub-attributes: text content and status information. Define the extended identifier of the control , represented as: in It is a control Basic identifiers, This indicates the current control's position in the list of child controls of its parent control. The basic identifier representing all parent controls of the current control. The set, The basic identifier representing all clickable controls on the current page. A set of.

6. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S3, for click events, the specific control is located using the touch coordinates in the event parameters. Combined with the view model of the page, the coordinates are mapped to the control by matching the control's boundary information. For text input events, control positioning is completed by monitoring changes in the input focus in real time.

7. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, In step S4, the system uses different methods to record and organize events for different categories. For click and text input events, once the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the event record data and the interval since the last control operation. For swipe and system key events, the user operation event log only contains the event log data and the interval since the last control operation; Starting from the first operation event when a user launches an Android application, the system monitors and organizes user operation event records in real time. Each record is encapsulated as a node in a linked list, eventually generating a complete operation trajectory sequence.

8. The method for recording and playing back user operation trajectories of a device-independent Android APP according to claim 1, characterized in that, It also includes a device-independent Android APP user operation trajectory recording and playback system, which includes: operation event listening and recording module, view construction module, user operation and control mapping module, operation trajectory sequence generation module and path playback module; The operation event listening and recording module is used to initialize the device and environment, and at the same time listen for and record valid operation events in real time; The view building module is used to record the structural information of the application interface, including page information, control information and their position information, control identifiers and control text attributes, and finally generate the view model; The User Operation and Control Mapping Module is used to obtain the mapping relationship between events and controls for click and text input events, using the event representation information obtained from the Operation Event Listening and Recording Module and the view model generated by the View Building Module. The operation trajectory sequence generation module is used to record each operation event and its associated control information starting from the first user operation event of the Android APP, and to construct a complete operation trajectory sequence by using each operation information as a node; The path replay module is used to gradually reconstruct user operations from the initial page based on the sequence of operation trajectories.

Images