Device-independent Android APP user operation track recording and playback method

Abstract

The invention discloses a device-independent Android APP user operation track recording and playback method, which comprises the following steps: connecting an Android device to obtain basic information, monitoring control information of a current screen activity view and a user operation event in real time, and converting the control information and the user operation event into a standardized logic structure; according to the control layout of the current screen activity view, a structured view model is constructed, and the model covers all control information of the page. Aiming at the user operation event, generating a mapping relation between the user operation event and the page view control, and obtaining each user operation event and corresponding control information; starting from the first user operation event, recording each user operation event and the control information corresponding to the user operation event, and gradually generating an operation track sequence; and in a track playback stage, restarting the APP and adjusting the APP to an initial interface, and realizing playback of all actions by traversing an operation track sequence in sequence. According to the method provided by the invention, efficient recording and accurate playback of the equipment-independent Android APP user operation track are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of Android APP automated testing, and in particular to a device-independent Android APP user operation track recording and playback method. Background Art

[0002] With the popularity of smartphones, apps have been widely used in daily life. In order to optimize the user experience, the demand for automated testing of Android apps is increasing, especially in the field of user behavior analysis. When analyzing user behavior, a key step is to record the user operation trajectory on the device and replay these operations later to accurately reproduce the user behavior. In addition, the recording of user operation trajectories also plays an important role in APP traffic fingerprint analysis. By replaying the user's operation trajectory, a large amount of user traffic data can be collected, providing data support for research such as traffic pattern recognition and user preference prediction.

[0003] However, existing operation track recording and playback technologies have certain limitations. Existing technologies rely on specific device hardware and system environment, resulting in the inability to maintain consistency in the same operation track on different devices. Specifically, when recording events, existing technologies record touch events based on the absolute coordinates of the screen. If the screen size or resolution of the playback device is different, the same coordinates may not trigger the expected control operation. This limitation may cause it to perform poorly in cross-device testing. There is an urgent need for a device-independent operation track recording and playback technology to ensure the consistency of operation reproduction on different devices. To this end, we propose a device-independent Android APP user operation track recording and playback method. Summary of the invention

[0004] In order to solve the above technical problems, a device-independent Android APP user operation trajectory recording and playback method is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is: A device-independent Android APP user operation track recording and playback method comprises the following steps: S1: Connect to the Android device via USB or Wi-Fi to obtain basic information about the device; monitor the control information and user operation events of the current screen active view in real time and convert them into a standardized logical structure; S2: Based on the control layout of the current screen active view, extract all its control information and build a structured view model for all browsed pages in the Android APP, which contains all the control information of the page; S3: Based on the view model and the real-time monitored user operation events, for sliding and system key events, a mapping relationship between user operation events and page view controls is generated, and each user operation event and its corresponding control information are obtained; S4: starting from the first user operation event when the user launches the Android APP, each user operation event and its corresponding control information are recorded, and a complete operation trajectory sequence is gradually generated; S5: Trajectory playback phase, restart the Android APP and adjust it to the initial interface, and traverse the operation trajectory sequence in sequence to play back all actions.

[0006] Preferably, in step S1: The basic information includes screen resolution, operating system version, device ID, and device name; The user operation events include click, slide, system button and text input; The standardized logical structure stores event representations in the form of triples:

[0007] In the formula, The timestamp indicating when the event occurred. Indicates the event type. Represents event parameters; If the event type is click, then Indicates the coordinates of the touch point. If the event type is sliding, then Indicates the coordinate sequence of the start and end points of the slide. If the event type is a system button, then Indicates the key code captured by the system framework. If the event type is text input, then Includes the center position of the control and the text content.

[0008] Preferably, in step S1, the judgment condition of the click event is: When the parameter of a user operation event contains only one coordinate, it means that the user has performed a single-point touch behavior. The event is a screen click operation, and its parameter is the screen coordinate of the touch point. If the touch point is within the boundary of a clickable control and triggers a view change after the touch event, the operation event is ultimately classified as a valid click event. Click operations that do not meet the above conditions will be deemed invalid and discarded.

[0009] Preferably, in step S1, the judgment condition of the sliding event is: its parameters include the starting point and end point coordinate sequence of the sliding, and a series of continuous coordinate pairs, which are used to represent the change of the touch path, reflecting that the user has performed a screen sliding operation. Suppose the sliding path is:

[0010] is from Move to , simplifying the path to:

[0011] If the sliding path length When the sliding distance is greater than the preset minimum sliding distance and the sliding operation causes the view to change or the control position to shift, the operation event is classified as a valid sliding event. Sliding operations that do not meet the above conditions will be considered invalid and discarded.

[0012] Preferably, in step S1, the judgment condition of the text input event is: The system first determines whether the target control supports text input by analyzing the UI controls in the interface; When a control that supports text input is activated, the system monitors the changes in input characters during event triggering, records the complete sequence of user input, and confirms that the text content of the target control has been successfully updated to the content entered by the user by analyzing the view response after the input; When the above conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as text input parameters. Text input operations that do not meet the above conditions will be considered invalid and discarded.

[0013] Preferably, in step S2, the structured view model is constructed by the following steps: Use XPath expressions to parse the UI hierarchy displayed on the device screen and capture all clickable controls; Use UIAutomator to extract the basic information of each control, including but not limited to: position, size, text content, package name, and resource ID. Finally, each control is represented as a triple:

[0014] In the formula, Represents the boundary information of the control. Indicates the constructed control identifier. Represents the text property of the control; Filter the extracted interface controls and remove controls that are not frequently operated by users, and controls that are not clickable or scrollable; For each control, its information It is expressed as:

[0015] In the formula, , , and Respectively represent the four boundary vertices of the control; Basic identification of the control It is expressed as:

[0016] In the formula, Representation Controls The name of the application package to which it belongs. Representation Controls The resource ID of Representation Controls The class name; Control Properties Contains two sub-attributes, namely text content and status information; Defines the extended identifier of the control , expressed as:

[0017] in It is a control The basic identification, Indicates the ordinal position of the current control in the list of child controls of its parent control. Represents the base identifier of all parent controls of the current control A collection of Indicates the basic identifier of all clickable controls on the current page A collection of .

[0018] Preferably, in step S3, for a click event, a specific control is located by the touch coordinates in the event parameters, and in combination with the view model of the page, the mapping of coordinates to controls is achieved by matching the boundary information of the controls; for a text input event, control positioning is accomplished by real-time monitoring of changes in input focus.

[0019] Preferably, in step S4, the system uses different methods to record and organize events of different categories; For click and text input events, when the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the event record data and the interval time from the last control operation; For sliding and system button events, the user operation event record only contains the event record data and the interval time from the last control operation; Starting from the first operation event when the user launches the Android application, the system monitors and organizes the user operation event records in real time. Each record will be encapsulated as a node in a linked list, and finally a complete operation track sequence will be generated.

[0020] Preferably, in step S5, during the trajectory playback phase, the system first starts the target Android application through a command, and ensures that its interface state is consistent with the initial interface recorded by the user operation trajectory; When the application state adjustment is completed, the system starts to traverse each node in the operation trajectory sequence in turn for playback. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into executable simulation operations; After each operation is played back, the system monitors changes in the interface status to ensure that the operation results are consistent with the trajectory records. If it is detected that the interface status deviates from expectations, the system will attempt to correct it through a preset recovery strategy to ensure that subsequent operations can be executed smoothly. The entire playback process is performed in chronological order until all operation records are traversed and successfully executed. The recovery strategy includes restarting the application or returning to the initial interface.

[0021] Preferably, it also includes a device-independent Android APP user operation track recording and playback system, which includes: an operation event monitoring and recording module, a view construction module, a user operation and control mapping module, an operation track sequence generation module and a path playback module; The operation event monitoring and recording module is used to initialize the device and environment, and monitor and record effective operation events in real time; The view construction module is used to record the structural information of the application interface, including page information, control information and its location information, control identification, and text attributes of the control, and finally generate a view model; The user operation and control mapping module is used to obtain the mapping relationship between events and controls for two events: click and text input events, using the event representation information obtained by the operation event monitoring and recording module and the view model generated by the view construction module; The operation trajectory sequence generation module is used to record each operation event and its associated control information starting from the first user operation event of the Android APP, and use each operation information as a node to construct a complete operation trajectory sequence; The path playback module is used to gradually restore user operations from the initial page based on the operation trajectory sequence.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The Android APP user operation trajectory recording and playback method proposed in the present invention monitors and records user operation events in real time, and at the same time builds a structured view model for all browsing pages of the Android APP. The method can accurately map user operation events to specific page view controls. This device-independent feature overcomes the limitations of the prior art that rely on specific device hardware and system environment, so that the same operation trajectory can accurately trigger the expected control operation on devices with different screen sizes or resolutions. The event identifiers are stored in a standardized logical structure to ensure the standardization and readability of the event records. The consistency of the operation results and the trajectory records is ensured by monitoring the changes in the interface status. If it is detected that the interface status deviates from the expectation, the system will also try to correct it through a preset recovery strategy, thereby ensuring that subsequent operations can be executed smoothly, thereby improving the efficiency and accuracy of Android APP automated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of a device-independent Android APP user operation track recording and playback system in an example of the present invention; Figure 2 It is a record format diagram of operation event and control mapping rules and operation event in the example of the present invention; Figure 3 It is a schematic diagram of trajectory playback in an example of the present invention. DETAILED DESCRIPTION

[0024] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0025] A device-independent Android APP user operation trajectory recording and playback method includes: S1: connecting an Android device via USB or Wi-Fi to obtain basic information of the device; monitoring control information and user operation events of the current screen active view in real time, and converting them into a standardized logical structure.

[0026] Use USB or Wi-Fi to connect to the Android device, and obtain basic information of the device through the ADB (Android Debug Bridge) interface, including screen resolution, operating system version, device ID, device name, etc.

[0027] When a user performs an event operation on an Android device, the system executes the "getevent" command through the ADB interface to obtain the original event data stream from the device's input layer. The data contains detailed information about low-level interactive events such as screen touch and key operations. The system then analyzes the captured event data in real time and classifies it into four categories: click, slide, system key, and text input.

[0028] A click event indicates that the user touches a clickable control on the screen. When the parameter of the user operation event contains only one coordinate, it means that the user has performed a single-point touch behavior, and the event is a screen click operation. Its parameters are the screen coordinates of the touch point. If the touch point is within the boundary of a clickable control and a view change is triggered after the touch event, the operation event is ultimately classified as a valid click event. View changes can be determined by the view model constructed later, and different views can be defined based on the different controls contained in different views. Click operations that do not meet the above conditions will be considered invalid and discarded.

[0029] A sliding event refers to a continuous interactive operation completed by the user through a screen drag gesture, which is used to describe the change process of the touch track. Its parameters usually include the coordinate sequence of the starting point and end point of the sliding, as well as a series of continuous coordinate pairs to represent the change of the touch path, reflecting that the user has performed a screen sliding operation. If there is a sliding path , is from Move to , the path can be simplified to If the sliding path length When the sliding distance is greater than the preset minimum sliding distance and the sliding operation causes the view to change or the control position to shift, the operation event is classified as a valid sliding event. Sliding operations that do not meet the above conditions will be considered invalid and discarded.

[0030] System key events refer to the operation of users pressing physical or virtual system keys, such as the back key, home key, and volume key. Such events are independent of specific view controls and do not rely on the matching of touch paths. Instead, they are global functions provided by the device operating system. System key events are captured by the ADB interface, and their key types are identified by key codes and reflected in the parameters of user operation events.

[0031] A text input event refers to an operation in which a user interacts with an input box in the interface to input characters, numbers, or symbols. The system first determines whether the target control supports text input by analyzing the UI controls in the interface. When a control that supports text input is activated (i.e., it obtains input focus), the system monitors changes in the input characters during the event triggering process and records the complete content sequence of the user input. At the same time, the system confirms that the text content of the target control has been successfully updated to the content entered by the user by analyzing the view response after the input. When the above conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as a text input parameter. Text input operations that do not meet the above conditions will be considered invalid and discarded.

[0032] After the system parses the event record data, it obtains the event representation stored in the form of triples:

[0033] in, is the timestamp of the event. is the event type, is the event parameter. If the event type is click, then is the coordinate of the touch point; if the event type is sliding, then is the coordinate sequence of the start and end points of the slide; if the event type is a system button, then is the key code captured by the system framework; if the event type is text input, then The above data only saves the event record data, and does not involve the identification of the operated control. The mapping relationship between the event and the control can be obtained by matching with the page information.

[0034] S2: Based on the control information of the current screen active view, associate it with the interaction information to build a structured view model for all browsed pages in the Android APP, which includes all control information of the page.

[0035] First, use XPath expressions to parse the UI interface hierarchy displayed on the device screen and capture all clickable controls. Then, use UIAutomator to extract the basic information of each control (including position, size, text content, package name, resource ID, etc.), and finally represent each control as a triple:

[0036] in For the boundary information of the control, is the constructed control identifier, The text property of the control.

[0037] The system will filter the extracted interface controls and remove some controls that users do not often operate, such as input method keyboards, emoji selection boxes, etc. In addition, the system will also remove those controls that are not clickable or scrollable, such as static label text, prompt boxes, etc. After that, for each control, its information It can be expressed as follows:

[0038] in , , and They represent the four boundary vertices of the control and describe the specific position of the control on the screen.

[0039] Basic identification of the control , which can be expressed as:

[0040] in Representation Controls The name of the application package to which it belongs. Representation Controls The resource ID of Representation Controls The class name of the control is unique in the view tree.

[0041] Control Properties , contains two sub-attributes, namely text content and state information. Text content includes button labels, text box prompt text, etc. If the control contains text content, the system will associate the text content with the location boundary of the control. If the control does not have text content, this item is empty. State information includes the current interactive state of the control, such as whether it is selected or available. For example, the property instance of a control is:

[0042] The text content of the control is , the optional value is empty, and the status information is that interaction is allowed.

[0043] In the process of generating or updating the view model, every time a user browses a new page, the system will parse and extract all the controls on the page, record the key properties of the controls one by one, and combine them into the description form of the above triples. In addition, the control layout of the page may change with user operations (such as pop-up windows, scrolling loading). Therefore, after each operation, the system will re-read the page information and update the view model.

[0044] In order to locate the same control on different devices, the extended identifier of the control is defined. . Different controls can be uniquely identified, expressed as:

[0045] in It is a control The basic identification, Indicates the ordinal position of the current control in the list of child controls of its parent control. Represents the base identifier of all parent controls of the current control A collection of Indicates the basic identifier of all clickable controls on the current page By marking these information, the control's extended identifier can uniquely identify different controls.

[0046] S3: Based on the view model and the real-time monitored user operation events, for sliding and system key events, a mapping relationship from user operation events to controls on the page view is generated, and the control information of each user operation event and its operation is obtained.

[0047] User operation events can be divided into four categories: click, slide, system button, and text input. Only click and text input require the target control to be located. Slide operations target the entire view screen, while system button events are captured by the system framework and identified by button codes, and are not directly associated with specific controls.

[0048] For click events, the specific control is located through the touch coordinates in the event parameters. The original coordinates only reflect the specific touch point of the user on the screen and cannot be directly associated with the specific control of the page. In order to map the touch coordinates to the control, the system combines the view model of the page and realizes the mapping of coordinates to controls by matching the boundary information of the control. During the matching process, the system reads all the control information in the current page view table and compares them one by one with the coordinates of the click event. If there is a click event Coordinates and controls , its location information If it exists and , then the event Clicking the control .

[0049] For text input events, control positioning is completed by real-time monitoring of changes in input focus. The system determines that the control that currently has focus is the input target control. At the same time, if changes in the input characters are detected during the event, it is further confirmed that the control is the control that actually performs text input.

[0050] S4: Starting from the first user operation event of the user launching the Android APP, each user operation event and the information of the control operated are recorded to generate an operation trajectory sequence.

[0051] The system uses different ways to record and organize different types of events. For click and text input events, when the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the above-mentioned event record data and the interval time from the last control operation. For sliding and system button events, the user operation event record only contains the event record data and the interval time from the last control operation.

[0052] Starting from the first operation event when the user launches the Android application, the system monitors and organizes the user operation event records in real time. Each record will be encapsulated as a node in a linked list. As the user operation occurs, new records are dynamically added to the end of the linked list to ensure that the operation track is arranged in chronological order. Finally, the system generates a complete operation track sequence based on the order of events. The operation track sequence is saved for later playback in the playback phase.

[0053] S5: Trajectory playback phase, restart the Android APP and adjust it to the initial interface, and traverse the operation trajectory sequence in sequence to play back all actions.

[0054] In the trajectory playback phase, the system first launches the target Android application through a command and ensures that its interface state is consistent with the initial interface recorded by the user's operation trajectory. This is usually achieved by clearing the application cache data or sending a command to restart the application to avoid the interference of the historical operation state on the playback process.

[0055] When the application state adjustment is completed, the system begins to traverse each node in the operation trajectory sequence for playback. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into executable simulation operations. For click events, the system uses the extended identifier of the recorded control to Locate the control and perform the click operation; the sliding event generates the sliding path through the start and end coordinates in the trajectory sequence, and then sends continuous touch instructions at appropriate time intervals to complete the action; the system key event directly calls the device control interface to complete the operation; for text input events, the system also uses the extended identifier of the control Locate the control and enter the text content for the record.

[0056] After each operation is replayed, the system monitors changes in the interface state to ensure that the operation results are consistent with the trajectory record. If the interface state is detected to deviate from the expected, the system will try to correct it through a preset recovery strategy (such as restarting the application or returning to the initial interface) to ensure that subsequent operations can be executed smoothly. In addition, the system strictly controls the execution time of adjacent operations according to the time interval recorded in the operation trajectory sequence. The entire playback process is carried out in chronological order until all operation records are traversed and successfully executed.

[0057] Reference Figure 1As shown in the figure, steps S1 to S5 describe in detail the operation process of the device-independent Android APP user operation trajectory recording and playback system. The design concept of this system aims to achieve comprehensive recording and accurate playback of Android APP user operations through the collaborative work of multiple functional modules. Specifically, it uses the operation event monitoring module to capture various interactive actions between users and APPs, and combines the view construction module to extract detailed information of APP pages and various controls, and finally forms an operation trajectory sequence, and completes the playback of all operation events based on this sequence.

[0058] A device-independent Android APP user operation track recording and playback system can be effectively used to implement the Android APP user operation track recording and playback method in the above embodiment. The system consists of five key modules, namely, an operation event monitoring and recording module, a view construction module, a user operation and control mapping module, an operation track sequence generation module, and a path playback module.

[0059] The operation event monitoring and recording module is responsible for initializing the device and environment at the beginning of the entire system operation. It needs to ensure that the various parameters of the device and the operating environment can meet the needs of subsequent operations, monitor all operations of users on the Android APP in real time, and accurately record the effective operation events. The effective operation events here cover all meaningful interactive behaviors of users on the APP interface, such as clicking the screen, sliding the page, entering text, etc.

[0060] The main responsibility of the view construction module is to record the structural information of the application interface in detail. It not only focuses on the overall layout information of the page, but also goes deep into the details of each control, including the specific information of the control, the location information in the page, the unique control identifier, and the text attributes displayed by the control. By collecting and organizing this information, the module generates a comprehensive view model that can clearly present the various components of the interface and their relationships.

[0061] The user operation and control mapping module mainly focuses on the two most common and important event types: click and text input. It cleverly uses the event representation information obtained by the operation event monitoring and recording module, and the view model carefully generated by the view construction module. Through a series of complex and precise algorithms and logic, it successfully obtains the mapping relationship between events and controls, and can accurately match the user's operation behavior with the specific controls on the APP interface, laying a solid foundation for the accurate recording and playback of subsequent operation trajectories.

[0062] The operation track sequence generation module begins its meticulous recording work from the first user operation event triggered by the Android APP. It does not miss any operation event, and also records the control information associated with each operation event. Then, it regards each operation information as a node, connects these nodes in an orderly manner through a specific organization method, and finally constructs a complete operation track sequence. This operation track sequence is like a timeline, which completely records the user's operation process on the APP, from the initial operation to the final end, and each step is clearly traceable.

[0063] The path playback module works based on the operation track sequence. It starts from the initial page and follows the steps recorded in the operation track sequence to restore the user's operations on the APP step by step. No matter which control is clicked or what text is entered, it can accurately reproduce it, as if the user has performed a complete operation on the APP again, providing users with a highly restored operation playback experience.

[0064] Reference Figure 2 As shown in the figure, during the operation of the system, there is a rigorous and detailed processing flow for the two types of events, click and text input. When the system successfully closely associates the user's real-time operation with the layout information of the page control, it will enter the information integration stage. At this time, the system will comprehensively collect and integrate key information from many aspects, including detailed information of the user operation itself, such as the strength of the click, the specific content of the text input, etc.; control information, such as the type, function, and page location of the control; and the interval time from the last operation on the control. After deeply integrating this information, it is saved as a complete user operation event record. This record fully and accurately records the relevant information of a specific operation on a specific control, providing a solid data foundation for subsequent analysis, backtracking, and system optimization.

[0065] As for sliding and system button events, the structure of their user operation event records is relatively simple. In this type of event record, only event record data is included, that is, the record of specific event contents such as sliding direction, distance, type of system button pressed, and the interval time from the last control operation. This recording method can not only meet the recording needs of key information of these two types of events, but also will not increase the system burden due to too much redundant information, ensuring the efficiency and pertinence of system records.

[0066] In terms of specific control positioning, for click events, the system uses a method based on the touch coordinates in the event parameters to accurately locate specific controls. When the user performs a click operation on the screen, the system will quickly capture the touch coordinate information at the moment of the click. Then, based on the pre-built page control layout information, the touch coordinates are compared with the location area of ​​each control through an accurate coordinate matching algorithm, so as to accurately determine which specific control the user clicked. This coordinate-based positioning method is as accurate and efficient as determining a specific location on a map based on longitude and latitude, providing accurate target pointing for subsequent processing of click events.

[0067] For text input events, the system adopts a strategy of completing control positioning by real-time monitoring of input focus and input character changes. When the user starts to input text, the system will pay attention to the position changes of the input focus on the page in real time. Once the input focus falls on a specific control, it means that the control becomes the target object of the current text input. At the same time, the system will continue to monitor the real-time changes of the input characters in order to accurately record the content of the user's input. Through this dual monitoring mechanism of input focus and input character changes, the system can promptly and accurately determine the specific controls related to the text input event, ensuring that the recording and processing of text input operations can accurately correspond to the corresponding controls.

[0068] Reference Figure 3 As shown in the figure, in the trajectory playback phase, the Android APP is restarted and adjusted to the initial interface, and each node in the operation trajectory sequence is traversed and replayed in turn. After each operation is replayed, the system monitors the changes in the interface state to ensure that the operation results are consistent with the trajectory record. If it is detected that the interface state deviates from the expectation, the system will return to the initial interface and re-execute.

[0069] The use process of the present invention is: connecting devices, acquiring and converting information, extracting control information, building a view model, generating an operation and control mapping relationship, recording an operation track sequence, and replaying the operation track.

[0070] In summary, the advantages of the present invention are: improving the efficiency and accuracy of Android APP automated testing, and providing strong technical support for user behavior analysis, APP traffic fingerprint analysis and other fields. By recording and replaying user operation trajectories, user behavior can be accurately reproduced, and a large amount of user traffic data can be collected, providing data support for research such as traffic pattern recognition and user preference prediction.

[0071] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A device-independent Android APP user operation track recording and playback method, characterized in that: The following steps are involved: S1: Connect to the Android device via USB or Wi-Fi to obtain basic information of the device; Monitor the control information and user operation events of the current screen active view in real time and convert them into standardized logical structures; S2: Based on the control layout of the current screen active view, extract all its control information and build a structured view model for all browsed pages in the Android APP, which contains all the control information of the page; S3: Based on the view model and the real-time monitored user operation events, for sliding and system key events, a mapping relationship between user operation events and page view controls is generated, and each user operation event and its corresponding control information are obtained; S4: starting from the first user operation event when the user launches the Android APP, each user operation event and its corresponding control information are recorded, and a complete operation trajectory sequence is gradually generated; S5: Trajectory playback phase, restart the Android APP and adjust it to the initial interface, and traverse the operation trajectory sequence in sequence to play back all actions.

2. According to claim 1, a device-independent Android APP user operation trajectory recording and playback method is characterized in that: In step S1: The basic information includes screen resolution, operating system version, device ID, and device name; The user operation events include click, slide, system button and text input; The standardized logical structure stores event representations in the form of triples: In the formula, The timestamp indicating when the event occurred. Indicates the event type. Represents event parameters; If the event type is click, then Indicates the coordinates of the touch point. If the event type is sliding, then Indicates the coordinate sequence of the start and end points of the slide. If the event type is a system button, then Indicates the key code captured by the system framework. If the event type is text input, then Includes the center position of the control and the text content.

3. According to the device-independent Android APP user operation trajectory recording and playback method of claim 1, it is characterized in that: In step S1, the judgment condition of the click event is: When the parameter of a user operation event contains only one coordinate, it means that the user has performed a single-point touch behavior. The event is a screen click operation, and its parameter is the screen coordinate of the touch point. If the touch point is within the boundary of a clickable control and triggers a view change after the touch event, the operation event is ultimately classified as a valid click event. Click operations that do not meet the above conditions will be deemed invalid and discarded.

4. According to the device-independent Android APP user operation trajectory recording and playback method of claim 1, it is characterized in that: In step S1, the judgment condition of the sliding event is: its parameters include the coordinate sequence of the starting point and the end point of the sliding, and a series of continuous coordinate pairs, which are used to represent the change of the touch path, reflecting that the user has performed a screen sliding operation. Suppose the sliding path is: is from Move to , simplifying the path to: If the sliding path length When the sliding distance is greater than the preset minimum sliding distance and the sliding operation causes the view to change or the control position to shift, the operation event is classified as a valid sliding event; Swipe operations that do not meet the above conditions will be considered invalid and discarded.

5. According to claim 1, a device-independent Android APP user operation track recording and playback method is characterized in that: In step S1, the judgment condition of the text input event is: The system first determines whether the target control supports text input by analyzing the UI controls in the interface; When a control that supports text input is activated, the system monitors the changes in input characters during event triggering, records the complete sequence of user input, and confirms that the text content of the target control has been successfully updated to the content entered by the user by analyzing the view response after the input; When the above conditions are met, the event is classified as a valid text input event, and the captured input content is recorded as text input parameters. Text input operations that do not meet the above conditions will be considered invalid and discarded.

6. The device-independent Android APP user operation track recording and playback method according to claim 1, characterized in that: In step S2, the structured view model is constructed by the following steps: Use XPath expressions to parse the UI hierarchy displayed on the device screen and capture all clickable controls; Use UIAutomator to extract the basic information of each control, including but not limited to: position, size, text content, package name, and resource ID. Finally, each control is represented as a triple: In the formula, Represents the boundary information of the control. Indicates the constructed control identifier. Represents the text property of the control; Filter the extracted interface controls and remove controls that are not frequently operated by users, and controls that are not clickable or scrollable; For each control, its information It is expressed as: In the formula, , , and Respectively represent the four boundary vertices of the control; Basic identification of the control It is expressed as: In the formula, Representation Controls The name of the application package to which it belongs. Representation Controls The resource ID of Representation Controls The class name; Control Properties Contains two sub-attributes, namely text content and status information; Defines the extended identifier of the control , expressed as: in It is a control The basic identification, Indicates the ordinal position of the current control in the list of child controls of its parent control. Represents the base identifier of all parent controls of the current control A collection of Indicates the basic identifier of all clickable controls on the current page A collection of .

7. The device-independent Android APP user operation trajectory recording and playback method according to claim 1 is characterized in that: In step S3, for click events, the specific control is located through the touch coordinates in the event parameters, and the coordinates are mapped to the control by matching the boundary information of the control in combination with the view model of the page; for text input events, control positioning is completed by real-time monitoring of changes in input focus.

8. The device-independent Android APP user operation trajectory recording and playback method according to claim 1 is characterized in that: In step S4, the system uses different methods to record and organize events of different categories; For click and text input events, when the real-time operation is successfully associated with the page control layout information, the system integrates the user operation information with the control information and saves it as a complete user operation event record. The user operation information includes the event record data and the interval time from the last control operation; For sliding and system button events, the user operation event record only contains the event record data and the interval time from the last control operation; Starting from the first operation event when the user launches the Android application, the system monitors and organizes the user operation event records in real time. Each record will be encapsulated as a node in a linked list, and finally a complete operation track sequence will be generated.

9. The device-independent Android APP user operation track recording and playback method according to claim 1, characterized in that: In step S5, during the trajectory playback phase, the system first starts the target Android application through a command and ensures that its interface state is consistent with the initial interface recorded by the user operation trajectory; When the application state adjustment is completed, the system starts to traverse each node in the operation trajectory sequence in turn for playback. For each record, the system reads the stored operation type, control information, and related parameters, and converts them into executable simulation operations; After each operation is played back, the system monitors changes in the interface status to ensure that the operation results are consistent with the trajectory records. If it is detected that the interface status deviates from expectations, the system will attempt to correct it through a preset recovery strategy to ensure that subsequent operations can be executed smoothly. The entire playback process is performed in chronological order until all operation records are traversed and successfully executed. The recovery strategy includes restarting the application or returning to the initial interface.

10. The device-independent Android APP user operation track recording and playback method according to claim 1, characterized in that: It also includes a device-independent Android APP user operation track recording and playback system, which includes: an operation event monitoring and recording module, a view construction module, a user operation and control mapping module, an operation track sequence generation module and a path playback module; The operation event monitoring and recording module is used to initialize the device and environment, and monitor and record effective operation events in real time; The view construction module is used to record the structural information of the application interface, including page information, control information and its location information, control identification, and text attributes of the control, and finally generate a view model; The user operation and control mapping module is used to obtain the mapping relationship between events and controls for two types of events: click and text input events, using the event representation information obtained by the operation event monitoring and recording module and the view model generated by the view construction module; The operation trajectory sequence generation module is used to record each operation event and its associated control information starting from the first user operation event of the Android APP, and use each operation information as a node to construct a complete operation trajectory sequence; The path playback module is used to gradually restore user operations from the initial page based on the operation trajectory sequence.

Images