Data processing method and device of terminal equipment, equipment and storage medium

Abstract

The invention provides a data processing method and device for terminal equipment, the equipment and a storage medium. The method comprises the steps of obtaining various metadata corresponding to interactive operation of a human-computer interface of the terminal equipment based on multiple services operated by the terminal equipment; screening out interaction event data matched with a timestamp in the time sequence from the interaction event data set; screening out image data matched with the timestamp from the image data set; screening out process page data matched with the timestamp from the process page data set; combining the interaction event data, the image data and the process page data matched with each timestamp in the time sequence to obtain composite data of each timestamp; and combining the composite data of each timestamp into screen recording data according to a time sequence. According to the method and the device, the recorded playback data with relatively high precision can be provided through the multiple metadata.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, apparatus, device, and storage medium for a terminal device. Background Art

[0002] The data processing of a terminal device aims to generate recording and playback data of a user interface in a human-computer interface. User interface recording and playback is an efficient user interface automation technology that identifies, collects, and analyzes user interface operations by obtaining the original data stream of events of the operating system of the terminal device. However, the user interface operation data obtained through the original data stream of events cannot provide recording and playback data with high accuracy. Summary of the Invention

[0003] Embodiments of this application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for a terminal device, which can generate recording and playback data with high accuracy through multiple metadata.

[0004] The technical solution of the embodiments of this application is implemented as follows:

[0005] Embodiments of this application provide a data processing method for a terminal device, the method comprising:

[0006] Based on multiple services running on the terminal device, obtain multiple types of metadata corresponding to interaction operations of the human-computer interface of the terminal device; wherein, different services are used to obtain different metadata, and the multiple types of metadata include: an interaction event data set, an image data set, and a process page data set;

[0007] Screen out interaction event data that matches the time stamp in the time series from the interaction event data set;

[0008] Screen out image data that matches the time stamp from the image data set;

[0009] Screen out process page data that matches the time stamp from the process page data set;

[0010] Combine the interaction event data, the image data, and the process page data that match each time stamp in the time series to obtain composite data for each time stamp;

[0011] Combine the composite data for each time stamp in chronological order to form screencast data.

[0012] Embodiments of this application provide a data processing apparatus for a terminal device, the apparatus comprising:

[0013] An acquisition module, configured to obtain various metadata corresponding to interaction operations of a human-machine interface of the terminal device based on multiple services running on the terminal device; wherein, different services are used to obtain different metadata, and the various metadata include: an interaction event data set, an image data set, and a process page data set;

[0014] A screening module, configured to screen out interaction event data that matches a timestamp in a time series from the interaction event data set; screen out image data that matches the timestamp from the image data set; and screen out process page data that matches the timestamp from the process page data set;

[0015] A combination module, configured to combine the interaction event data, the image data, and the process page data that match each timestamp in the time series to obtain composite data for each timestamp; and combine the composite data for each timestamp in chronological order to obtain screencast data.

[0016] An embodiment of the present application provides an electronic device, which includes:

[0017] A memory, configured to store computer-executable instructions;

[0018] A processor, configured to implement the data processing method of the terminal device provided by the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium, storing a computer program or computer-executable instructions, which are configured to implement the data processing method of the terminal device provided by the embodiment of the present application when being executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, which are configured to implement the data processing method of the terminal device provided by the embodiment of the present application when being executed by a processor.

[0021] The embodiment of the present application has the following beneficial effects:

[0022] In the embodiment of the present application, multiple servers are used to collect various metadata from different sources, including an interaction event data set, an image data set, and a process page data set. By screening out various metadata corresponding to timestamps and combining them as screencast data for recording and playback, the various metadata complement and reference each other, improving the comprehensiveness and accuracy of the screencast data, thereby improving the accuracy and stability of recording and playback. Description of the Drawings

[0023] Figure 1It is a schematic structural diagram of the data processing system architecture of the terminal device provided by an embodiment of the present application;

[0024] Figure 2 It is a schematic structural diagram of the terminal device provided by an embodiment of the present application;

[0025] Figure 3A It is a first process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0026] Figure 3B It is a second process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0027] Figure 3C It is a third process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0028] Figure 3D It is a fourth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0029] Figure 3E It is a fifth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0030] Figure 3F It is a sixth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0031] Figure 3G It is a seventh process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0032] Figure 3H It is an eighth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0033] Figure 3I It is a ninth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0034] Figure 3J It is a tenth process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0035] Figure 3K It is an eleventh process schematic diagram of the data processing method of the terminal device provided by an embodiment of the present application;

[0036] Figure 4 It is a schematic diagram of the output of the character stream provided by an embodiment of the present application;

[0037] Figure 5 It is a process schematic diagram of starting a process provided by an embodiment of the present application;

[0038] Figure 6 It is a schematic flowchart of the screen recording data acquisition method provided by an embodiment of the present application;

[0039] Figure 7 It is a schematic flowchart of the data calibration method provided by an embodiment of the present application;

[0040] Figure 8 It is a schematic flowchart of the screen recording method provided by an embodiment of the present application. Detailed implementation manners

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limitations on the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0042] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0043] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] Before further elaborating on the embodiments of the present application, the nouns and terms involved in the embodiments of the present application are described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0046] 1) getevent: A tool in the Android system, which is essentially an executable file of the Linux operating system kernel. It provides real-time dump information about input devices and kernel input events in the Android system. It can be understood that through this tool, the real-time information flow in the Android event system can be obtained, including screen input events (such as click, swipe, long press, return to the home screen, back, etc.), key input events (such as power key, volume control key, etc.), and real-time information of some kernel input events. The results will be displayed as a string stream data in a specific format on the terminal for the tool user to view.

[0047] 2) app_process: A native executable program provided by the Android system based on the Linux system, located in the system / bin / directory of the Android system. The most primitive zygote process (the process responsible for spawning processes) is also started by this executable file. Its function is that based on this executable program, a native Java program can be started in the command-line interface (shell) of the operating system. Therefore, based on this function, a process with system-level shell execution permission can be started.

[0048] 3) Minicap: An open-source tool for taking screenshots on Android devices. It is a lightweight screen capture library, mainly used to capture the screen content in real time on Android devices and transfer it to a computer for display or processing. Using Minicap can achieve some useful functions, such as screen recording, screen sharing, and screen analysis, etc.

[0049] 4) User Interface (UI): Refers to the human-computer interaction interface, which is the medium for interaction and information exchange between the system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans.

[0050] UI recording and playback is an efficient UI automation technology. In related technologies, the main focus is on performing some single-item analyses on the original event data stream. For example, directly parsing operations by referring to the original data stream of getevent and directly performing parsing and conversion on some simple character data in it. For instance, directly performing coordinate system conversion calculations on the original data with coordinate data of x and y. However, it is very difficult to directly automate UI recording and playback only with the event metadata obtained by getevent. The reason is that the descriptive attributes of only event data generally cannot provide stable data information because the UI interface usually changes dynamically over time series. Only relying on the single-item information of the operation coordinate area is often insufficient to meet the needs of its stable automated playback. Therefore, from the perspective of stability and success rate, in addition to the descriptive information of the operation area (such as coordinate information, time information, type information), it must be done from the perspective of UI elements, and image data is a good choice as the necessary UI description information. However, in addition to image data, the necessary information related to the UI interface actually also includes data for describing the current interface state, such as the current process information, current page information, etc. Currently, in related technologies, there is no combination of descriptive metadata such as images, processes, and pages, which also has a relatively large impact on the stability and success rate of the recording and playback technology. Identifying and collecting and analyzing UI operations directly by obtaining the original data of getevent in the Android system cannot provide recording and playback data with relatively high accuracy. For example, for a UI click operation, although the core operation metadata (such as the click position, etc.) of the click operation can be obtained at a certain time point through getevent, directly injecting the event of the playback position based on the data of getevent will obviously have relatively poor robustness and stability of the function. Therefore, it is necessary to combine other auxiliary data for comprehensive analysis and judgment to improve the accuracy and success rate of UI playback.

[0051] Based on the above problems, the embodiments of the present application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for a terminal device. By using getevent, Minicap, and data collection to obtain an interaction event data set, an image data set, and a process page data set corresponding to the interaction operations of the human-machine interface, and combining the three types of data according to the time timestamp as the core data for UI recording and playback, the accuracy and stability of recording and playback are improved.

[0052] The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), a smart phone, a smart speaker, a smart watch, a smart TV, a vehicle-mounted terminal, etc.

[0053] See Figure 1 , Figure 1 which is a schematic structural diagram of the data processing system architecture of the terminal device provided in the embodiments of the present application. Exemplarily, Figure 1 involves a debugging device 100, a network 200, a terminal device 300, and a server 400, and a human-machine interface 301 displayed on the terminal device 300. Exemplarily, the debugging device 100 may be a personal computer (PC). Among them, the terminal device 300 can be connected to the debugging device 100 and the server 400 through the network 200. The network 200 can be a wide area network, a local area network, or a combination of the two.

[0054] In some embodiments, the terminal device 300 can establish a link with the debugging device 100 through a wired connection, such as establishing a link through a Universal Serial Bus (USB). The debugging device 100 sends instructions through the Android Debug Bridge (ADB). The terminal device 300 is used to receive the instructions and send a recording request to the server 400 according to the instructions. After receiving the recording request, the server 400 generates screen recording data and sends the screen recording data to the terminal device 300, so as to realize the recording and playback of the human-machine interface 301.

[0055] In some embodiments, the server 400 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Among them, the cloud service can be an interactive processing service for the terminal to call. The debugging device 100 and the terminal device 300 can be smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, vehicle terminals, etc. with an Android system, but are not limited thereto. The terminal device 300 and the server 400 can be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present application.

[0056] In some embodiments, multiple servers can form a blockchain, and the server 400 is a node on the blockchain. There can be an information connection between each node in the blockchain, and information can be transmitted between nodes through the above information connection. Among them, the data related to the data processing method of the terminal device provided in the embodiments of the present application can be stored on the blockchain.

[0057] In some embodiments, the terminal device 300 or the server 400 may implement the data processing method of the terminal device provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions may be commands at the microprogram level, machine instructions, or software instructions. The computer program may be a native program or a software module in the operating system; it may be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a game application or a live broadcast application; it may also be a small program embedded in any APP, that is, a program that only needs to be downloaded to the browser environment to run. In short, the above computer-executable instructions may be instructions in any form, and the above computer programs may be application programs, modules, or plugins in any form.

[0058] The embodiments of the present application may be implemented with the help of Artificial Intelligence (AI) technology. Artificial Intelligence is the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, Artificial Intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial Intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0059] Artificial Intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of Artificial Intelligence generally include, for example, sensors, dedicated Artificial Intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various directions of Artificial Intelligence after fine-tuning. The software technologies of Artificial Intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] In some embodiments, refer to Figure 1 , Figure 1The recording request in can include recording parameters related to the recorded content. By using the recording parameters, when controlling the screen recording, only the data related to the specified area or specified element is recorded, such as the image data of the target person part, the image data of the target object part, the data within the target software, the data related to the target operation, etc. In this way, after the server 400 receives the recording request including the recording parameters, it can perform target recognition on each frame of the initial screen recording data according to the recording parameters, determine the position or relevant video frames related to the recording parameters through the recognition result, then perform matte extraction on the relevant position, and synthesize the video frames after matte extraction as the final screen recording data to be sent to the terminal device 300, thereby realizing the compression of the initial screen recording data. Among them, in the process of performing target recognition on the initial screen recording data according to the recording parameters, target recognition can be performed through artificial intelligence, and a corresponding training set is formed according to the target corresponding to the recording parameters to train the model, obtaining a corresponding target recognition model, such as using the face recognition model of artificial intelligence to recognize the faces in the initial screen recording data, etc., so as to obtain the screen recording data related to the recording parameters.

[0061] See Figure 2 , Figure 2 is a schematic structural diagram of the terminal device provided by an embodiment of the present application. Figure 2 The terminal device 300 shown includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. Each component in the terminal device 300 is coupled together through a bus system 340. It can be understood that the bus system 340 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 340.

[0062] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] The user interface 330 includes one or more output devices 331 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls.

[0064] The memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memories, hard disk drives, optical disc drives, etc. The memory 350 optionally includes one or more storage devices that are physically remote from the processor 310.

[0065] The memory 350 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 350 described in the embodiments of the present application is intended to include any suitable type of memory.

[0066] In some embodiments, the memory 350 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are exemplarily described below.

[0067] The operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0068] The network communication module 352 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, Wi-Fi (Wireless Fidelity), and Universal Serial Bus (USB), etc.;

[0069] The presentation module 353 is used to enable the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more output devices 331 associated with the user interface 330 (such as a display screen, a speaker, etc.);

[0070] The input processing module 354 is used to collect one or more user inputs or interactions from one of one or more input devices 332 and translate the collected inputs or interactions.

[0071] In some embodiments, the device provided by the embodiments of the present application can be implemented in software. Figure 2 Shown is a data processing device 355 of the terminal device stored in the memory 350, which can be software in the form of programs and plugins, etc., including the following software modules: an acquisition module 3551, a screening module 3552, and a combination module 3553. These modules are logical, and thus can be arbitrarily combined or further split according to the functions to be implemented. The functions of each module will be described below.

[0072] The following will describe the exemplary applications and implementations of the terminal device provided in the embodiments of the present application. Taking the terminal device as the execution subject, the data processing method of the terminal device provided in the embodiments of the present application will be described. Refer to Figure 3A , Figure 3A which is the first process schematic diagram of the data processing method of the terminal device provided in the embodiments of the present application, and will be described in combination with the steps shown in Figure 3A .

[0073] In step 101, based on multiple services running on the terminal device, various metadata corresponding to the interaction operations with the human-machine interface of the terminal device are obtained; among them, different services are used to obtain different metadata, and the various metadata include: an interaction event data set, an image data set, and a process page data set.

[0074] In some embodiments, the multiple services include an event service, a screen capture service, and a process page collection service, and three types of metadata corresponding to the interaction operations with the human-machine interface of the terminal device are obtained through the above three services. In the following specific example, the terminal device is taken as an Android system for illustration, but it should not be regarded as a limitation to the embodiments of the present application. Those skilled in the art can implement corresponding technical solutions in other operating systems according to the following records.

[0075] Refer to Figure 3B , Figure 3B which is the second process schematic diagram of the data processing method of the terminal device provided in the embodiments of the present application, Figure 3A and the steps shown in step 101 can be implemented through the following steps 1011 to 1013, which will be specifically described below.

[0076] In step 1011, through the event service in the terminal device, an interaction event data set corresponding to the interaction operation with the human-machine interface of the terminal device is obtained.

[0077] In some embodiments, the event service in the terminal device can be implemented through getevent, and through getevent, an interaction event data set corresponding to the interaction operation with the human-machine interface can be periodically obtained.

[0078] In some embodiments, refer to Figure 3C , Figure 3C which is the third process schematic diagram of the data processing method of the terminal device provided in the embodiments of the present application, Figure 3B and the steps shown in step 1011 can be implemented through the following steps 10111 and 10112, which will be specifically described below.

[0079] In step 10111, the event service is used to obtain the original event data stream corresponding to the interaction operation of the human-machine interface of the terminal device, and the original event data stream is redirected to the buffer area of the event service.

[0080] In some embodiments, the interaction event data set can be periodically obtained through the event service, and all the interaction event data sets corresponding to the interaction operations within a certain time period can be obtained.

[0081] In step 10112, the original event data stream in the buffer area is parsed to obtain the graphic element data of the human-machine interface, the operation data of the human-machine interface, and the operation type data of the human-machine interface, so as to combine them into an interaction event data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0082] In some embodiments, the original event data stream is a character stream collected by getevent (which can also be called the original string data stream). By parsing the original event data stream in the buffer area, for example, cutting and identifying the original event data stream according to the characteristic data of the operation type and the event data, the graphic element data of the human-machine interface, such as UI element image data, the operation data of the human-machine interface, such as UI operation metadata data, and the operation type data of the human-machine interface, such as UI operation type data, or other description data or core metadata for describing the interaction operation related to the interaction operation of the human-machine interface can be obtained. For example, taking the click type operation as an example, the core metadata corresponding to the click operation may include: click time, click position, click operation duration, and other information.

[0083] In some embodiments, the event service collects the original event data stream generated by all UI operations. For example, the original data stream collected by getevent is used for parsing to obtain the original event data stream. The original event data stream is the character stream output by the event subsystem in the Android kernel, and the output character stream is similar to Figure 4 shown as Figure 4 is a schematic diagram of the output of the character stream provided by the embodiment of the present application. These character stream data have the following characteristics: poor readability, large data volume, high complexity, disordered and without obvious rules, and large format differences in the read-only memory (ROM) of the model system. By parsing and identifying these character streams, it can be well recognized from the Figure 4 character stream data shown as which UI operation types are included in a series of original event data streams within a certain time period, such as click operation type, double-click operation type, long-press operation type, and the description data corresponding to the UI operation, such as click position, click time, sliding area, sliding speed, etc.

[0084] Through the above steps 10111 and 10112, based on the getevent capability, the original data stream of all UI operation events of the Android operating system within a certain period is obtained through buffer redirection. By redirecting and parsing these original data streams, the chaotic original data stream of events in the continuous data stream is converted into the corresponding UI operation data at the corresponding time points, which can support the parsing of graphic element data, operation data, and operation type data related to the UI operation type within a certain period from the original data stream of time. By converting the chaotic original data stream of events into an interactive event data set corresponding to the interactive operations of the human-machine interface, the efficiency of screen recording data processing is improved.

[0085] In step 1012, an image data set corresponding to the interactive operations of the human-machine interface of the terminal device is obtained through the screen capture service in the terminal device.

[0086] In some embodiments, refer to Figure 3D , Figure 3D is the fourth process schematic diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3B The steps shown in 1012 can be implemented through the following steps 10121 to 10123, which are specifically described below.

[0087] In step 10121, in the screen capture service, the human-machine interface of the terminal device is captured at regular intervals through a command in the form of a system-level permission command line to obtain image data.

[0088] In some embodiments, the screen capture service is implemented by starting minicap. A service with the running logic of minicap built in is run in the Android system. This service always resides in the Android system memory and runs in the background. The screen capture service realizes the ability to capture the screen image at regular intervals through starting minicap. For example, it starts to collect at a frequency of 0.1s per frame, that is, 10 real-time screen image data will be saved locally every second. The screen capture service in the terminal device can be implemented through minicap. In minicap, a command in the form of a system-level permission command line, such as the command of the operating system's command line interface (shell), is used to capture the screen at regular intervals to obtain image data.

[0089] In step 10122, the first time information when the human-machine interface of the terminal device is captured is recorded.

[0090] In some embodiments, the first time information is the current time information of the system when the screen capture service captures the human-machine interface of the terminal device.

[0091] In step 10123, the first time information and the image data are combined to obtain an image data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0092] Through steps 10121 to 10123, in the embodiments of the present application, the data for generating UI recording and playback data (i.e., screen recording data) may include, in addition to the original data obtained by the above getevent, a series of Android screen image data sets collected based on the minicap service. These data include not only the human-machine interface image data files but also the system time information corresponding to the screenshots. Based on the screenshot ability provided by minicap, the image data of the human-machine interface within a certain period of time is continuously obtained, thereby improving the richness of the screen recording data and further improving the stability and versatility of the UI recording and playback technology.

[0093] Continue to refer to Figure 3B , in step 1013, through the process page collection service in the terminal device, a process page data set corresponding to the interaction operation of the human-machine interface of the terminal device is obtained.

[0094] In some embodiments, refer to Figure 3E , Figure 3E is the fifth process schematic diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3B The steps shown in 1013 can be implemented by the process page collection service in the terminal device by executing the following steps 10131 to 10133. The following is a specific description.

[0095] In step 10131, through a command in the form of a system-level permission command line, the process process information and the active page information corresponding to the interaction operation of the human-machine interface of the terminal device are collected.

[0096] In some embodiments, the process page collection service in the terminal device can be implemented through a process collection service. By starting the process page collection service, the process process information and the active page information are periodically collected. For example, the process page collection is performed once every 0.1 second.

[0097] In step 10132, the second time information when the process process information and the active page information are collected is recorded.

[0098] In some embodiments, the second time information is the current system time information corresponding to the collection of the process process information and the active page information, and the system time information at the time of collection is recorded while performing the process page collection each time.

[0099] In step 10133, the process process information, the active page information, and the second time information are combined to obtain a process page data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0100] In some embodiments, the collected process process information, active page information, and second time information are combined into a composite data, which is used as the process page data corresponding to the interaction operation of the human-machine interface and cached in the memory.

[0101] Through steps 10121 to 10123, the embodiment of the present application starts a process page collection service dedicated to collecting process page information, which is used to collect the process page information corresponding to the UI operation during the human-computer interaction. By improving the richness of the screen recording data, the stability and versatility of the UI recording and playback technology are improved.

[0102] Here, when multiple types of metadata corresponding to the UI are obtained through multiple services, there is still a technical problem of how to calibrate and align the multiple data. When the interaction event data set, the image data set, and the process page data set are respectively obtained in a certain time period, these data are completely independent and there is no direct correlation between them. That is to say, for the interaction event data set corresponding to the UI operation at a certain moment, it is impossible to determine the associated image data set and process page data set. Only the getevent event data will perform UI playback based on regional data such as coordinates. Because the page often changes dynamically with time, the previously recorded UI-related descriptive data may not be stably displayed in the current latest page view. It may exist or may not exist. Even if there is a corresponding UI element, it may also change, such as position adjustment, etc. Therefore, limited by the dynamics of these UIs, using only the getevent data will definitely result in much lower success rate and stability. So, when a continuous interaction event data set is obtained through getevent, it is necessary to determine the descriptive data of the corresponding image data set and process page data set. However, the existing recording and playback technologies in the related art are limited by this problem, so only the event data collected based on getevent is used to generate playback data.

[0103] Based on this, the embodiment of the present application provides a solution for aligning and calibrating multiple types of metadata based on a time series. Refer to Figure 3A Steps 102 to 105 shown are used to align the timestamps in the time series to achieve data calibration and combination between the interaction event data set, the image data set, and the process page data set.

[0104] In step 102, the interaction event data that matches the timestamps in the time series is filtered out from the interaction event data set.

[0105] In some embodiments, all interaction event data that match the timestamps in the time series can be filtered out from the set of interaction event data, or some of the interaction event data that match the timestamps in the time series can be filtered out from the set of interaction event data. Among them, the timestamp can also be replaced with other information that can identify when a specific event occurs, for example, time identification information, time series information, etc.

[0106] In some embodiments, referring to Figure 3F , Figure 3F is the sixth process schematic diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3A The step 102 shown can be implemented through the following steps 1021 to 1023, which will be specifically described below.

[0107] In step 1021, timestamps included in multiple event data are extracted from the set of interaction event data.

[0108] In some embodiments, the set of interaction event data includes multiple event data, and the event data corresponds to the interaction operations of the human-machine interface. Each event data has corresponding time information, such as a timestamp. By extracting the timestamps included in multiple event data from the set of interaction event data, the time corresponding to each event data can be determined, so as to determine the time corresponding to each operation in the interaction operations of the human-machine interface.

[0109] In step 1022, the timestamps are combined in chronological order to form a time series.

[0110] In step 1023, for each timestamp in the time series, the interaction event data including the timestamp is used as the interaction event data that matches the timestamp.

[0111] In some embodiments, a time series including multiple timestamps is formed based on the set of interaction event data. That is to say, the timestamps included in multiple event data are extracted from the set of interaction event data, and then a time series is formed according to the chronological relationship. At the same time, for each timestamp, the interaction event data including the corresponding timestamp is used as the interaction event data that matches the timestamp, so as to arrange the interaction events in the set of interaction event data in a time series, which is convenient for aligning and calibrating multiple metadata.

[0112] In step 103, the image data that matches the timestamp is filtered out from the set of image data.

[0113] In some embodiments, referring to Figure 3G , Figure 3G is the seventh process schematic diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3AThe illustrated step 103 can be implemented by the following steps 1031 and 1032, which are specifically described below.

[0114] In step 1031, in the first time information of each image data in the image data set, determine the first time information closest to the timestamp.

[0115] In some embodiments, according to the principle of proximity in time, traverse and analyze from the first time information of each image data in the image data set to obtain the first time information whose time attribute is closest to the timestamp.

[0116] In step 1032, use the image data corresponding to the first time information closest to the timestamp as the image data matching the timestamp.

[0117] In the above steps 1031 and 1032, determine the first time information closest to the timestamp in the first time information through the principle of proximity in time, establish a time association between the image data set and the interaction event data, and facilitate the calibration and alignment of multiple metadata.

[0118] In step 104, filter out the process page data that matches the timestamp from the process page data set.

[0119] In some embodiments, refer to Figure 3H , Figure 3H which is the eighth flowchart diagram of the data processing method of the terminal device provided in the embodiments of the present application. Figure 3A The illustrated step 104 can be implemented by the following steps 1041 and 1042, which are specifically described below.

[0120] In step 1041, in the second time information of each process page data in the process page data set, determine the second time information closest to the timestamp.

[0121] In some embodiments, according to the principle of proximity in time, traverse and analyze from the second time information of each process page data in the process page data set to obtain the second time information whose time attribute is closest to the timestamp.

[0122] In step 1042, use the process page data corresponding to the second time information closest to the timestamp as the image data matching the timestamp.

[0123] In the above steps 1041 and 1042, determine the second time information closest to the timestamp in the second time information through the principle of proximity in time, establish a time association between the process page data set and the interaction event data, and facilitate the calibration and alignment of multiple metadata.

[0124] Continue to refer to Figure 3A, in step 105, the interaction event data, image data, and process page data that match each timestamp in the time series are combined to obtain composite data for each timestamp.

[0125] In some embodiments, referring to Figure 3I , Figure 3I is the ninth flowchart diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3A The steps shown in step 105 can be implemented through the following steps 1051 and 1052, which will be specifically described below.

[0126] In step 1051, the interaction event data, image data, and process page data that match each timestamp in the time series are aligned according to the timestamps.

[0127] In some embodiments, a calibration comparison strategy is based on timestamps, and multiple types of data are aligned according to the principle of nearest matching in time.

[0128] In step 1052, the aligned interaction event data, image data, and process page data are combined to obtain composite data for each timestamp.

[0129] In some embodiments, a calibration comparison strategy is based on timestamps, and data combination is performed according to the principle of nearest matching in time. The interaction event data set, image data set, and process page data set are calibrated and reorganized.

[0130] Exemplarily, as Figure 7 shown, three types of metadata are collected: an interaction event data set, an image data set, and a process page data set, which can be displayed in the same time diagram in the view of the time series. For example Figure 7 The topmost timeline is the UI operation data obtained by getevent (corresponding to the above-mentioned interaction event data), representing 5 operations performed on the UI during the recording process. Each operation has corresponding interaction event data, and corresponding to the timeline is the system time point corresponding to each operation. The second line represents the screenshot data obtained by minicap every 0.1 seconds (corresponding to the above-mentioned image data) and the display at the corresponding time; the third line has a similar logic, representing the process and page information collected by the process data acquisition sequence every 0.1 seconds (corresponding to the above-mentioned process page data). Figure 7 For the first interaction event data in [], the time that is closest to it in time matching is the first screenshot data in the timeline of the screenshot data obtained by minicap and the first process page data in the timeline of the process data acquisition sequence. Therefore, these three data will be combined into the final recording data. By analogy, the three types of data are aligned according to the timestamps.

[0131] In the above steps 1051 and 1052, by analyzing the sets of interaction event data, image data, and process page data respectively obtained during a certain operation time process, the interaction event data that matches the time stamps in the time series is filtered out from the set of interaction event data based on the time stamps. Then, according to the time stamp information of each event data, the image data with the time attribute closest to the time stamp is traversed and analyzed from the obtained set of image data. Similarly, the process page data closest to the time stamp is traversed and analyzed from the obtained set of process page data. Then, these three types of data are recombined to obtain relatively highly complete descriptive data of interaction operations as the composite data for each time stamp. Through the alignment and recombination of multiple types of data, relatively highly complete descriptive data of interaction operations can be obtained, thereby improving the stability and generality of the recording and playback technology.

[0132] In step 106, the composite data for each time stamp is combined into screen recording data in the order of time sequence.

[0133] In some embodiments, taking the composite data including three types of data as the final screen recording data, the three types of data can be recombined according to requirements as screen recording data that can fully analyze the interaction operation elements.

[0134] See Figure 3J , Figure 3J is the tenth process schematic diagram of the data processing method of the terminal device provided by the embodiment of the present application. Before Figure 3A the step 101 shown, steps 107 to 109 can also be executed to establish a communication channel between the terminal device and the debugging device to start multiple services in the terminal device, which will be specifically described below.

[0135] In step 107, a communication channel between the terminal device and the debugging device is established based on the debugging tool; wherein, the communication channel enables data communication and logical synchronization interaction between the terminal device and the debugging device.

[0136] In some embodiments, a bidirectional communication channel from a debugging device to the terminal device is established through the debugging tool adb (including adb forward and adb reverse), and data communication and logical synchronization interaction between the terminal device and the debugging device are carried out through the communication channel.

[0137] In step 108, initialization instructions for multiple services are received through the communication channel.

[0138] In some embodiments, the debugging device sends initialization instructions through the communication channel, enabling the terminal device to receive the initialization instructions for multiple services.

[0139] In step 109, multiple services are started based on the initialization instructions.

[0140] In some embodiments, after receiving the initialization instruction, the terminal device starts multiple services according to the initialization instruction. Among them, the debugging device also sends a start recording instruction to the terminal device. After receiving the start recording instruction, the multiple services will start collecting various metadata.

[0141] In some embodiments, referring to Figure 3K , Figure 3K is the eleventh process schematic diagram of the data processing method of the terminal device provided by the embodiments of the present application. Figure 3J The step 109 shown can be implemented by the following step 1091 and step 1092. The following is a specific description.

[0142] In step 1091, based on the initialization instruction, a process with system-level permissions is started through a system-level executable program.

[0143] In step 1092, in the process, a command in the form of a system-level permission command line is executed through a process component to start multiple services.

[0144] In some embodiments, when starting a process through a system-level executable program, such as app_process, the processing logic executed is different from the application (APP) startup logic in the related art. The process started in this way belongs to the native java process of the Linux system and has corresponding system-level permissions, so as to be able to serve as the ability on which the getevent core depends.

[0145] Exemplarily, as Figure 5 shown, Figure 5 is the process schematic diagram of starting a process provided by the embodiments of the present application. The specific description is as follows.

[0146] First, start a program (such as a Boot Loader program) through a boot process (such as a Boot ROM program), load a kernel (such as a Keml kernel) through the startup program, thereby starting an initialization process (such as an init process), parse a file to obtain a startup script file (such as init.rc), and execute a loop instruction (such as a Loop instruction). At the same time, the initialization process starts a process, and the started process belongs to the native java process of the Linux system, thereby starting a system-level executable program (such as app_process) to obtain a process with corresponding system-level permissions (a Shell permission process). Among them, the startup script file starts processes, including: a process that incubates processes (such as Zygote), a server management (such as a service_manager program), a system service (a SurfaceFlinger program), and a media service (a MediaServe program). Here, the process that incubates processes starts a virtual machine, and then forks through a fork function (such as a Fork function). The fork includes: a system server (such as system_server) and a desktop launcher (such as Launcher). The system service is started through the system server, then the startup interface is displayed, and then the desktop launcher is started to display the desktop. Among them, the system service also generates an activity management service (ActivityManagerService), a power management service (PowerManagerService), an input management service (InputManagerService), and a window management service (WindowManagerService), and registers them to the inter-process communication (such as the Binder communication method) generated by the server management. The server management executes a loop instruction after generating the inter-process communication. The system service generates a frame buffer (such as FrameBuffer), and then executes a loop instruction. The media service generates an audio service (such as AudioFlinger), an image service (such as CameraService), and a media playback service (MediaPlayerService), and then executes a loop instruction. In this way, the process of starting a process through app_process is realized.

[0147] In the above steps 107 to 109, a process with system-level permissions is started through a system-level executable program, so as to realize data collection of operation events based on getevent, and all kernel event data serialized in a specific format in the terminal device can be collected in real time, including the above interaction event data.

[0148] In some embodiments, the data processing method of the terminal device according to the embodiments of the present application can be applied to any Android mobile phone device. After starting the tool provided by this solution from the PC side, UI operations can be performed on the corresponding Android mobile phone device. When performing UI operations, three independent services on the Android device will collect data respectively, and then all relevant data in the entire continuous UI operation process will be integrated together through the data calibration method provided by the embodiments of the present application. According to the nearest matching rule of the time series, more accurate and rich UI element data can be obtained well. Through the embodiments of the present application, UI automation technology is realized. For example, UI recording and playback can well support all UI automation on Android devices, make full use of the data in the process and further structure it for the automation framework to parse and use. Especially during the playback process, based on the interactive event data set, image data set and process page data set, the data accuracy for judgment during playback can be well improved, thus providing more judgment conditions. At the same time, the embodiments of the present application are adapted to mainstream Android models, so they have relatively good generality and can well provide a data acquisition ability with relatively high robustness for Android system UI automation.

[0149] Next, an exemplary application of the embodiments of the present application in the screen recording application scenario will be described. Refer to Figure 6 , Figure 6 which is the implementation flowchart of the screen recording data acquisition method provided by the embodiments of the present application, as follows.

[0150] First, the PC side (corresponding to the above-mentioned debugging device) establishes a two-way communication channel from the PC side to the mobile terminal service set (corresponding to the above-mentioned terminal device) through the debugging tools adb forward and adb reverse, and conducts data communication and logical synchronization interaction through the way of socket communication.

[0151] After starting from the PC side tool, cross-terminal communication of ports will be performed based on the socket after establishing adb forward. The PC side tool sends two instructions through the adb socket: initialization 601 and start recording 602.

[0152] The PC - side tool first sends an initialization 601 instruction to the service set in the terminal device (such as a mobile terminal). The mobile - terminal service set starts three background services running on the Android side based on app_process in the Android system, including: the getevent collection service (corresponding to the above - mentioned event service), the minicap screen service (corresponding to the above - mentioned screenshot service), and the collection service (corresponding to the above - mentioned process - page collection service). These three services will continuously run on the Android device (corresponding to the above - mentioned terminal device).

[0153] Here, after receiving the initialization 601 command from the PC - side tool, the three services start to construct various function - dependent capabilities and corresponding initializations. The initialization service 6011 of the getevent collection service includes: the getevent collection service starts a collection service in memory to determine whether to start collecting getevent data. The initialization service 6012 of the minicap screen service includes: minicap configures and starts the underlying dependencies such as the dynamic - link library (so library) of the screenshot capability through commands. The initialization service 6013 of the process collection service includes: similar to the getevent collection service, the process collection service also starts a collection service in memory to determine whether to start collecting information such as the current top - level process (process) and activity (activity) of the system.

[0154] The PC - side tool then sends a start - recording 602 instruction to the mobile - terminal service set. After receiving the start - recording 602 instruction from the PC - side tool, the three mobile - terminal service sets will start collecting data respectively. During the entire recording process, three data sources obtained from different channels will be collected simultaneously, namely: event collection 6021, screenshot collection 6022, and process collection 6023.

[0155] First, the event collection 6021 specifically includes: start collection 60211. At this time, the getevent collection service starts to call the shell command getevent - l - t 60212. After executing this command, the terminal device outputs relevant Android input information, including information related to touching the screen. Since each Android ROM may be different, no specific restrictions are made here. Then, the collection of UI - event data starts. During the collection process, all UI - event data is output to the memory cache through data redirection 60213 for type parsing. The original data stream is redirected and parsed through data - parsing recognition 60214, and finally, the parsed data is cached through data caching 60215 for later use.

[0156] Here, based on getevent, the original data stream of all UI operation events of the Android operating system within a certain period can be obtained through buffer redirection. These original data streams are redirected and parsed. Through parsing, the chaotic original event data in the continuous data stream is converted into corresponding UI operation data at the corresponding time points. According to the original event data, the data sources with system characteristic properties among them are cut and identified, which can support parsing out the core metadata related to the UI operation type from the original event data stream within a certain period. For example, a click type operation includes click time, click position, click operation duration, etc.

[0157] Among them, the core basic ability of UI recording depends on the data source of getevent for parsing. Through the combination of app_process and getevent, based on the image recognition ability of the open-source computer vision library (Open Source Computer Vision Library, opencv), a general recording and playback solution for the real machine environment of terminal devices is realized. Through app_process, a set of backend recording and playback engine sets are started on the terminal device. Based on getevent, all operation data of the Android system in a specific period is collected in real time, and these operation metadata are automatically recorded after being converted through algorithm parsing and logical processing. Since ordinary user-level Apps are restricted in various permissions in the initialization logic related to the process creation of child processes (fork function), starting an application through conventional means (such as clicking on the application icon) in the Android system is generally a process with restricted permissions. However, although the executable program of app_process cannot be utilized by ordinary App processes, it can be driven through the adb execution program on the PC side. Therefore, as long as the Linux process is started through this app_process in the adb terminal on the PC side, it can have system-level execution permissions. As Figure 5As shown, when starting a process through app_process, the actual running logic is different from the App startup logic in the related technology. The process started in this way belongs to the native Java process of the Linux system. By default, app_process inherits from the user-level process init with a process identifier (pid) of 1. Therefore, the process started in this form naturally has the shell permission (corresponding to the above system-level permission) ability of the init process, thus realizing the ability that the getevent core in the recording function depends on. When a process with shell permission ability is started through app_process, the system-level shell command (corresponding to the command in the form of the above system-level permission command line) can be executed in this process through the process component of the programming language Java (corresponding to the above process component). Another core ability is to collect operation event data based on getevent. The startup principle of the getevent collection service is based on the fact that app_process executes shell instructions in the shell execution environment of a Java process as introduced above. Therefore, the effect is the same as executing getevent in the terminal. By redirecting the data obtained after starting getevent to the memory cache for continued parsing and identification of types, the recognition of each UI operation can analyze its corresponding operation type, operation metadata description information, etc. based on specific data characteristics.

[0158] In this process, all the data collected for all UI operations under this framework, such as UI element images, UI operation metadata, UI operation types, etc., are parsed based on the original data stream collected by getevent. Among them, the original data stream is the character stream output by the event subsystem in the Android kernel, and the output character stream is similar to Figure 4 shown. The characteristics of these character stream data are mainly in the following aspects: poor readability, large data volume, high complexity, disorder and no obvious rules, and large differences in the format of the device model system ROM. Based on these data stream characteristics, the operation type feature data and event data can be used for cutting and identification, and it is possible to well identify which UI operations are included in the data stream of a series of getevent within a certain period of time from the above original string data stream, and parse out the corresponding UI operation description data (such as click position, sliding area, etc.).

[0159] In a second aspect, the screenshot acquisition 6022 specifically includes: starting a screenshot 60221, and simultaneously recording the screenshot time 60222, corresponding to the above-mentioned second time information. Performing a minicap screenshot 60223, combining the image data obtained from the screenshot and the corresponding screenshot time through data combination 60224. The combined composite data corresponds to the above-mentioned image data set, and then the combined data is cached through data caching 60225. Here, a loop call logic will be entered. Essentially, it is to continuously execute a thread logic at regular intervals. The timing strategy is to execute once every 0.1 seconds, which is equivalent to executing 10 times per second. In the thread logic, mainly the system time at the time of taking the screenshot is recorded first, and then the screenshot data at the instruction level is obtained based on minicap. These two data will be combined into a composite data and cached in the memory for later use.

[0160] Here, minicap is started in the tool service installed on the terminal device. After receiving the instruction to start recording 602, continuous screenshots are taken. Based on the powerful capabilities provided by minicap, all image data within a certain period of time is continuously obtained. The UI image area covered by the operation is automatically recognized through the opencv open-source library, and the general UI recording and playback technology is realized in a way that is completely comparable to the normal operation of the real machine. The start of minicap is through the minicap.so and other dependent files that are pre-downloaded and pushed (adb push) to the Android device through the debugging tool, and then the minicap command is used to take screenshots at regular intervals through shell execution in the screen capture service. The current time information is also recorded while taking screenshots. The time information and the screenshot data will be combined into a data as the data source set generated by the screen capture service, that is, the above-mentioned image data set. Therefore, in addition to the original data obtained by the above-mentioned getevent, the UI recording and playback data will also include a series of Android screen image data collected based on the minicap service.

[0161] In a third aspect, the process acquisition 6023 specifically includes: starting the acquisition 60131, process acquisition 60132 (corresponding to the above-mentioned process process information), Activity acquisition 60133 (corresponding to the above-mentioned activity page information), combining the collected process process information, activity page information, and the corresponding time information through data combination 60134, and caching the combined data, that is, the above-mentioned process page data set, through data caching 60135. Among them, the process acquisition service also performs acquisition once every 0.1 seconds. In the acquisition logic, mainly the top-level process process information and the top-level page activity information are collected based on shell instructions, and the system time information at the time of acquisition (corresponding to the above-mentioned second time information) is also recorded, and then combined into a composite data and cached in the memory.

[0162] Here, the process collection 6023 will also, after receiving the instruction request of starting recording 602 sent by the PC-side tool, start collecting process information and activity page information based on shell commands through app_process, for collecting the process page data corresponding to UI operations during the collection process. These data are also combined into a new composite data as the data source set generated by the process page collection service, improving the richness of the data.

[0163] In addition, when receiving the stop recording instruction from the PC-side tool, the above three types of data will complete the collection. After completion, the three composite data will be parsed and compared. Since the three collection processes are completely independent of each other, it is necessary to associate the data. The purpose of the association is to achieve the effect of improving the stability and success rate of recording and playback. The association strategy is to perform comparison and matching based on the principle of proximity according to the time series. As Figure 7 shown, Figure 7 is the flowchart of the data calibration method provided by the embodiment of the present application, specifically as follows.

[0164] As Figure 7 shown, there are a total of three types of data collected: UI operation data obtained by getevent (corresponding to the above-mentioned interaction event data set), screenshot data obtained by minicap (corresponding to the above-mentioned image data set), and process collection data sequence (corresponding to the above-mentioned process page data set), which can be shown in the same time diagram in the view of the time series. For example, Figure 7 the top timeline 701 represents 5 operations performed on the UI during the recording process, and each operation corresponds to the system time point at the time of operation on the timeline; the second timeline 702 represents the display of the screenshot data taken by minicap every 0.1 second and the corresponding time; similarly, the third timeline 703 represents the process collection service collecting the current process and the current page auxiliary information every 0.1 second.

[0165] Here, taking Figure 7 the first click event data as an example, it includes: click type, click time, and click position. In terms of time matching, the closest to it are the first screenshot data in the minicap timeline and the first collection data in the process collection data sequence timeline. Therefore, these three data will be combined into the final recording data (corresponding to the composite data of each time stamp), which is used as the playback data that can fully analyze the UI elements. By analogy, matching is performed on the data corresponding to other operations based on the principle of the closest time.

[0166] Here, by collecting the three types of UI recording metadata obtained above, namely the getevent raw operation data, the minicap screenshot data, and the process page information; we perform multiple data calibrations based on the time series information alignment strategy, and obtain strongly correlated combined data in the time series from the above two completely different and uncorrelated data. These combined data will be used as the core use case data in the recording and playback technology. These use case data not only include the UI event data corresponding to the UI operations, but also the screenshot information and process page information of the UI at that time. These information sources greatly improve the richness of the dependent data for use case playback, can well improve the condition matching and data judgment during playback, well improve the success rate and stability of UI recording, reduce the cost of manual automation, and improve the efficiency of regression testing.

[0167] As Figure 8 shown, Figure 8 is a schematic flowchart of the screen recording provided by this application. The overall logic includes the above Figure 6 and Figure 7 , specifically refer to the Figure 6 and Figure 7 corresponding description parts.

[0168] Here, based on the data sets respectively obtained by getevent and minicap during a certain operation time process, time annotation and analysis are performed. Based on the combined strategy of time alignment, the corresponding UI event data set is filtered out from getevent, and then according to the timestamp information of each event data, the image data with the time attribute closest to the event data is traversed and analyzed from all the image data sets obtained from minicap. And the process page information with the closest time will also be obtained from the process collection service through a similar strategy. Then, these three types of data are reorganized. Through this data alignment and reorganization, relatively complete UI descriptive data can be obtained, which not only has operation area descriptive data such as coordinates, but also has corresponding on-site image data. Based on this solution, the stability and generality of the UI recording and playback technology can be improved.

[0169] Next, continue to describe the exemplary structure of the software module implementation of the data processing device 355 of the terminal device provided by the embodiments of this application. In some embodiments, as Figure 2 shown, the software modules stored in the data processing device 355 of the terminal device in the memory 350 may include:

[0170] An acquisition module 3551, configured to acquire various metadata corresponding to interaction operations of a human-machine interface of the terminal device based on multiple services running on the terminal device; wherein different ones of the services are used to acquire different ones of the metadata, and the various metadata include: an interaction event data set, an image data set, and a process page data set.

[0171] A screening module 3552, configured to screen out interaction event data that matches a time stamp in a time series from the interaction event data set; screen out image data that matches the time stamp from the image data set; and screen out process page data that matches the time stamp from the process page data set.

[0172] A combination module 3553, configured to combine the interaction event data, the image data, and the process page data that match each time stamp in the time series to obtain composite data for each time stamp; and combine the composite data for each time stamp in chronological order to obtain screen recording data.

[0173] In some embodiments, the acquisition module 3551 is further configured to acquire an interaction event data set corresponding to an interaction operation of the human-machine interface of the terminal device through an event service in the terminal device; acquire an image data set corresponding to the interaction operation of the human-machine interface of the terminal device through a screen capture service in the terminal device; and acquire a process page data set corresponding to the interaction operation of the human-machine interface of the terminal device through a process page acquisition service in the terminal device.

[0174] In some embodiments, the acquisition module 3551 is further configured to acquire an event original data stream corresponding to an interaction operation of the human-machine interface of the terminal device through the event service, redirect the event original data stream to a buffer area of the event service; and parse the event original data stream in the buffer area to obtain graphic element data of the human-machine interface, operation data of the human-machine interface, and operation type data of the human-machine interface, so as to combine them into an interaction event data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0175] In some embodiments, the acquisition module 3551 is further configured to, in the screen capture service, take a screenshot of the human-machine interface of the terminal device at regular intervals through a command in the form of a system-level permission command line to obtain image data; record first time information when taking the screenshot of the human-machine interface of the terminal device; and combine the first time information and the image data to obtain an image data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0176] In some embodiments, the acquisition module 3551 is further configured to establish a communication channel between the terminal device and the debugging device based on a debugging tool; wherein, the communication channel enables data communication and logical synchronization interaction between the terminal device and the debugging device; receive initialization instructions for the multiple services through the communication channel; and start the multiple services based on the initialization instructions.

[0177] In some embodiments, the acquisition module 3551 is further configured to start a process with system-level permissions through a system-level executable program based on the initialization instructions; and execute commands in the form of system-level permission command lines through a process component in the process to start the multiple services.

[0178] In some embodiments, the acquisition module 3551 is further configured to perform the following processing by collecting services through a process page in the terminal device: collect process process information and active page information corresponding to interaction operations of a human-machine interface of the terminal device through a command in the form of a system-level permission command line; record second time information when collecting the process process information and the active page information; and combine the process process information, the active page information, and the second time information to obtain a process page data set corresponding to the interaction operation of the human-machine interface of the terminal device.

[0179] In some embodiments, the screening module 3552 is further configured to extract timestamps included in multiple event data from the interaction event data set; combine the timestamps in chronological order to form a time series; and for each timestamp in the time series, use the interaction event data including the timestamp as the interaction event data matching the timestamp.

[0180] In some embodiments, the screening module 3552 is further configured to determine, in the first time information of each image data in the image data set, the first time information closest to the timestamp; and use the image data corresponding to the first time information closest to the timestamp as the image data matching the timestamp.

[0181] In some embodiments, the screening module 3552 is further configured to determine, in the second time information of each process page data in the process page data set, the second time information closest to the timestamp; and use the process page data corresponding to the second time information closest to the timestamp as the image data matching the timestamp.

[0182] In some embodiments, the combination module 3553 is further configured to align the interaction event data, the image data, and the process page data in the time series that match each of the timestamps according to the timestamps; and combine the aligned interaction event data, image data, and process page data to obtain composite data for each of the timestamps.

[0183] An embodiment of the present application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the data processing method of the terminal device in the embodiments of the present application.

[0184] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, where computer-executable instructions or a computer program are stored. When the computer-executable instructions or the computer program are executed by a processor, the processor will be caused to execute the data processing method of the terminal device provided in the embodiments of the present application. For example, as Figure 3A shown in the data processing method of the terminal device.

[0185] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0186] In some embodiments, the computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0187] As an example, the computer-executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files that store one or more modules, subroutines, or code portions).

[0188] As an example, the computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0189] In summary, through the embodiments of the present application, on any Android device, after starting the tool provided by this solution from the PC side, UI operations can be performed on the corresponding Android device. When performing UI operations, three independent services on the Android device will collect data respectively. The original event data stream obtained by getevent will be redirected to the terminal cache. The image data collected by minicap and the process page data collected by the process collection service will also be utilized. Through the data calibration method provided by the embodiments of the present application, all relevant data in the entire continuous UI operation process are integrated together. According to the nearest matching rule of the time series, more accurate and rich UI element data can be well obtained. Thus, UI automation technologies such as UI recording and playback are realized, better supporting all UI automation on Android devices and making full use of the data in the process and further structuring it for the automation framework to parse and use. Especially during the playback process, based on the image and event data and the real-time information collected by the system, the data accuracy for judgment during playback can be well improved, thus providing more judgment conditions. At the same time, it adapts to mainstream Android models and has relatively good versatility, and can well provide a relatively high-robustness data collection ability for Android UI automation.

[0190] The above is only the embodiments of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method for a terminal device, characterized in that, The method includes: Based on multiple services running on the terminal device, obtaining various metadata corresponding to the interaction operations of the human-machine interface of the terminal device; wherein, different services are used to obtain different metadata, and the various metadata include: an interaction event data set, an image data set, and a process page data set; Filtering out interaction event data that matches the time stamps in the time series from the interaction event data set; Filtering out image data that matches the time stamp from the image data set; Filtering out process page data that matches the time stamp from the process page data set; Combining the interaction event data, the image data, and the process page data that match each time stamp in the time series to obtain composite data for each time stamp; Combining the composite data for each time stamp in chronological order to form screencast data.

2. The method according to claim 1, wherein The obtaining, based on multiple services in the terminal device, of various metadata corresponding to the interaction operations of the human-machine interface of the terminal device includes: Obtaining, through an event service in the terminal device, an interaction event data set corresponding to the interaction operations of the human-machine interface of the terminal device; Obtaining, through a screen capture service in the terminal device, an image data set corresponding to the interaction operations of the human-machine interface of the terminal device; Obtaining, through a process page collection service in the terminal device, a process page data set corresponding to the interaction operations of the human-machine interface of the terminal device.

3. The method according to claim 2, characterized in that, The obtaining, through an event service in the terminal device, of an interaction event data set corresponding to the interaction operations of the human-machine interface of the terminal device includes: Obtaining, through the event service, an event original data stream corresponding to the interaction operations of the human-machine interface of the terminal device, and redirecting the event original data stream to the buffer area of the event service; Parsing the event original data stream in the buffer area to obtain the graphic element data of the human-machine interface, the operation data of the human-machine interface, and the operation type data of the human-machine interface, so as to combine them into an interaction event data set corresponding to the interaction operations of the human-machine interface of the terminal device.

4. The method according to claim 2, characterized in that, The obtaining, through a screen capture service in the terminal device, of an image data set corresponding to the interaction operations of the human-machine interface of the terminal device includes: In the screen capture service, taking a screenshot of the human-machine interface of the terminal device at regular intervals through a command in the form of a system-level permission command line to obtain image data; Recording the first time information when taking a screenshot of the human-machine interface of the terminal device; Combining the first time information and the image data to obtain an image data set corresponding to the interaction operations of the human-machine interface of the terminal device.

5. The method according to any one of claims 2 to 4, characterized in that, The method further includes: Establishing a communication channel between the terminal device and a debugging device based on a debugging tool; wherein, the communication channel enables data communication and logical synchronization interaction between the terminal device and the debugging device; Receiving, through the communication channel, initialization instructions for the multiple services; Starting the multiple services based on the initialization instructions.

6. The method according to claim 5, wherein Starting the multiple services based on the initialization instruction includes: Based on the initialization instruction, starting a process with system-level permissions through a system-level executable program; Executing a command in the form of a system-level permission command line in the process to start the multiple services.

7. The method according to claim 2, wherein Collecting, through a process page collection service in the terminal device, a set of process page data corresponding to an interaction operation with the human-machine interface of the terminal device, including: Performing the following processing through the process page collection service in the terminal device: Collecting, through a command in the form of a system-level permission command line, process process information and active page information corresponding to an interaction operation with the human-machine interface of the terminal device; Recording second time information when collecting the process process information and the active page information; Combining the process process information, the active page information, and the second time information to obtain a set of process page data corresponding to an interaction operation with the human-machine interface of the terminal device.

8. The method according to any one of claims 1 to 4, characterized in that, Filtering out interaction event data that matches a time stamp in the time series from the interaction event data set, including: Extracting time stamps included in multiple event data from the interaction event data set; Combining the time stamps in chronological order to form a time series; For each time stamp in the time series, taking the interaction event data including the time stamp as the interaction event data that matches the time stamp.

9. The method according to any one of claims 1 to 4, characterized in that, Filtering out image data that matches the time stamp from the image data set, including: Determining, in the first time information of each image data in the image data set, the first time information closest to the time stamp; Taking the image data corresponding to the first time information closest to the time stamp as the image data that matches the time stamp.

10. The method according to any one of claims 1 to 4, characterized in that Filtering out process page data that matches the time stamp from the process page data set, including: Determining, in the second time information of each process page data in the process page data set, the second time information closest to the time stamp; Taking the process page data corresponding to the second time information closest to the time stamp as the image data that matches the time stamp.

11. The method according to any one of claims 1 to 4, characterized in that, Combining the interaction event data, the image data, and the process page data that match each time stamp in the time series to obtain composite data for each time stamp, including: Aligning the interaction event data, the image data, and the process page data that match each time stamp in the time series according to the time stamp; Combining the aligned interaction event data, the image data, and the process page data to obtain composite data for each time stamp.

12. A data processing device for a terminal device, characterized in that, The device includes: An acquisition module, configured to acquire, based on multiple services running on a terminal device, multiple types of metadata corresponding to an interaction operation with the human-machine interface of the terminal device; wherein, different services are used to acquire different metadata, and the multiple types of metadata include: an interaction event data set, an image data set, and a process page data set; A screening module, configured to screen out interaction event data that matches the timestamps in the time series from the set of interaction event data; screen out image data that matches the timestamps from the set of image data; and screen out process page data that matches the timestamps from the set of process page data. A combining module, configured to combine the interaction event data, the image data, and the process page data that match each of the timestamps in the time series to obtain composite data for each of the timestamps; and combine the composite data for each of the timestamps in chronological order to obtain screencast data.

13. An electronic device, characterized in that, The electronic device includes: A memory, configured to store computer-executable instructions. A processor, configured to implement the data processing method of the terminal device according to any one of claims 1 to 11 when executing the computer-executable instructions stored in the memory.

14. A computer-readable storage medium stores computer-executable instructions or a computer program, characterized in that, The computer-executable instructions or the computer program, when executed by the processor, implement the data processing method of the terminal device according to any one of claims 1 to 11.

15. A computer program product, comprising computer-executable instructions or a computer program, characterized in that, The computer-executable instructions or the computer program, when executed by the processor, implement the data processing method of the terminal device according to any one of claims 1 to 11.

Images