Data processing method, device and equipment and computer readable storage medium

Abstract

The invention provides a data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring equipment information of target equipment and original operation data stream information generated by the target equipment; calling a target identification interface corresponding to the target equipment based on the equipment information; based on the target identification interface, performing data cutting processing on the original operation data stream information to obtain a plurality of operation data units; and based on the target identification interface, performing operation type identification processing on the plurality of operation data units to obtain the operation type of each operation data unit. By means of the method and device, the user interface operation type recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of Internet technology and the widespread use of mobile phones, the amount of data synchronized on the Internet can reach hundreds of billions every day. In the analysis of device automation anomalies, the log data generated by the device can be used to quickly determine the cause of the anomaly. In addition, the log data also records the execution information of the device and the execution data generated. Therefore, by identifying and analyzing the data flow information, all UI operation types contained in it can be determined, and the recording and playback of user UI operations can be realized.

[0003] In the related art, the UI operation types contained in the data stream are obtained by performing operation analysis on the character data in the original operation data stream. However, this analysis method cannot accurately identify all the UI operation types contained in the data stream when facing the data stream generated by complex continuous operations. Summary of the invention

[0004] Embodiments of the present application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of identifying the operation type of a user interface.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method, the method comprising:

[0007] Acquire device information of a target device and original operation data stream information generated by the target device;

[0008] Calling a target identification interface corresponding to the target device based on the device information;

[0009] Based on the target identification interface, data segmentation processing is performed on the original operation data stream information to obtain multiple operation data units;

[0010] Based on the target identification interface, operation type identification processing is performed on the multiple operation data units to obtain the operation type of each of the operation data units.

[0011] The present application provides a data processing device, including:

[0012] An acquisition module, used to acquire device information of a target device and original operation data stream information generated by the target device;

[0013] A calling module, used for calling a target identification interface corresponding to the target device based on the device information;

[0014] A cutting module, used for performing data cutting processing on the original operation data stream information based on the target identification interface to obtain multiple operation data units;

[0015] The identification module is used to perform operation type identification processing on the multiple operation data units based on the target identification interface to obtain the operation type of each operation data unit.

[0016] An embodiment of the present application provides an electronic device for data processing, the electronic device comprising:

[0017] Memory for storing computer programs or computer executable instructions;

[0018] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer program or computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, can implement the data processing method provided in the embodiment of the present application.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By calling the identification interface corresponding to the target device, the operation type of the operation data unit is identified, so that the operation types of the operation data units generated by different target devices can be identified, thereby improving the versatility of operation type identification; and, based on the identification interface, the original operation data stream information is cut and processed to accurately cut out multiple operation data units contained in the original operation data stream information, and then the type of each operation data unit is identified to obtain all operation types contained in the original operation data stream information, thereby improving the accuracy of operation type identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of a data processing system architecture provided by an embodiment of the present application;

[0024] Figure 2 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0025] Figure 3A It is a first flow chart of a data processing method provided by an embodiment of the present application;

[0026] Figure 3B is a second flow chart of a data processing method provided in an embodiment of the present application;

[0027] Figure 3C It is a third flow chart of a data processing method provided in an embodiment of the present application;

[0028] Figure 3D is a fourth flow chart of a data processing method provided in an embodiment of the present application;

[0029] Figure 3E is a fifth flow chart of a data processing method provided in an embodiment of the present application;

[0030] Figure 3F It is a sixth flow chart of a data processing method provided in an embodiment of the present application;

[0031] Figure 3G It is a seventh flow chart of a data processing method provided in an embodiment of the present application;

[0032] Figure 3H This is an eighth flow chart of a data processing method provided in an embodiment of the present application;

[0033] Fig. 3I This is a ninth flow chart of a data processing method provided in an embodiment of the present application;

[0034] Figure 3J is a tenth flow chart of a data processing method provided in an embodiment of the present application;

[0035] Figure 4A This is a schematic diagram of a first process of operating data unit type identification provided by an embodiment of the present application;

[0036] Figure 4B This is a second flow chart of an operation data unit type identification method provided by an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of an Android system program startup process provided by an embodiment of the present application;

[0038] Figure 6 It is a hierarchical logical structure diagram of a cutting recognition algorithm provided in an embodiment of the present application;

[0039] Figure 7 This is a schematic diagram of the original operation data stream cutting result provided by an embodiment of the present application;

[0040] Figure 8 A schematic diagram of a double-click operation data unit provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0046] 1) getevent command: It is an executable file of Linux system. In Android system, the getevent command can be used to obtain real-time information stream in Android system, such as screen input events (such as click, slide, long press, etc.), key input events (such as power button, volume control button, etc.) and some kernel input events. The results will be displayed in a specific format of string stream data on the terminal for tool users to view.

[0047] 2) Operation data unit: used to store the operation metadata generated when the user performs UI operations. When the user performs each UI operation, a corresponding operation data unit is generated to store all the operation data. When the user performs continuous UI operations, multiple operation data units are combined into the original operation data stream.

[0048] 3) Event data: The string information contained in each operation data unit is called event data, and each operation data unit includes multiple event data.

[0049] In the prior art, electronic devices directly perform operations on character data in the original operation data stream to obtain the UI operation types contained in the original operation data stream. However, this parsing method cannot accurately identify all the UI operation types contained in the original operation data stream generated by complex continuous operations. For example, in some game system scenarios, the actual user operations are relatively complex and have a certain degree of continuity, such as multiple clicks, multiple slides, and a combination of some clicks and slides. The original operation data stream generated in this scenario is sometimes output in a disordered and interspersed manner, and the existing technology cannot accurately identify all the UI operation types contained therein.

[0050] The embodiments of the present application provide a data processing method, apparatus, device, computer-readable storage medium and computer program product, which can cut the original operation data stream, obtain all operation data units contained in the original operation data stream, and identify the operation type for each operation data unit, thereby solving the problem in the prior art that it is impossible to accurately identify all operation types contained in the original operation data stream generated by complex operations.

[0051] The data processing method provided in the embodiment of the present application can be implemented by the terminal alone; it can also be implemented by the terminal and the server in collaboration. For example, the terminal alone undertakes the following data processing method, or the terminal sends a data stream to the server, and the server executes the data processing method according to the received data stream. In the process of data processing, the device information of the target device and the original operation data stream information generated by the target device are obtained; the target identification interface corresponding to the target device is called based on the device information; based on the target identification interface, the original operation data stream information is subjected to data segmentation processing to obtain multiple operation data units; based on the target identification interface, the multiple operation data units are subjected to operation type identification processing to obtain the operation type of each of the operation data units, so as to improve the accuracy of operation type identification.

[0052] The following describes exemplary applications of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, vehicle-mounted devices), smart phones, smart speakers, smart watches, smart televisions, and vehicle-mounted terminals.

[0053] In some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms. Among them, cloud services can be real-time data processing services for terminals to call.

[0054] Next, an exemplary application when the electronic device is implemented as a server will be described.

[0055] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the data processing system 100 provided in an embodiment of the present application. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] In some embodiments, taking the electronic device as a terminal as an example, the real-time data processing method provided in the embodiment of the present application can be implemented by the terminal. For example, the terminal 400 locally executes the data processing method provided in the embodiment of the present application, and in the process of data processing, obtains the device information of the target device and the original operation data stream information generated by the target device; based on the device information, calls the target identification interface corresponding to the target device; based on the target identification interface, performs data segmentation processing on the original operation data stream information to obtain multiple operation data units; based on the target identification interface, performs operation type identification processing on the multiple operation data units to obtain the operation type of each operation data unit, so as to improve the accuracy of operation type identification.

[0057] In some embodiments, the real-time data processing method provided in the embodiments of the present application can also be implemented by the server and the terminal in collaboration. For example, the data stream generated by the terminal device 400 can be transmitted to the server 200 through the network 300; the server 200 is used to obtain the device information of the target device and the original operation data stream information generated by the target device, and based on the device information, the target identification interface corresponding to the target device is called, and based on the target identification interface, the original operation data stream information is subjected to data segmentation processing to obtain multiple operation data units, and based on the target identification interface, the operation type identification processing is performed on the multiple operation data units to obtain the operation type of each operation data unit, thereby improving the accuracy of the operation type identification.

[0058] In some embodiments, the terminal or server can implement the data processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0059] In some embodiments, multiple servers may form a blockchain, and server 200 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the data processing method provided in the embodiment of the present application (such as an operation data unit) may be stored on the blockchain.

[0060] The structure of the electronic device provided by the embodiment of the present application is described below. Figure 2 , Figure 2 is a schematic diagram of the structure of the data processing terminal 200 provided in an embodiment of the present application, Figure 2 The terminal 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the terminal 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0061] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0064] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0066] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0067] A network communication module 252, used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB), etc.;

[0068] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripherals and displaying content and information);

[0069] The input processing module 254 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0070] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data processing device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 2551, a call module 2552, a cutting module 2553 and an identification module 2554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0071] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0072] The data processing method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. Figure 3A , Figure 3A is a flow chart of a data processing method provided in an embodiment of the present application, combined with Figure 3A The steps shown are explained, Figure 3A The step execution subject is an electronic device.

[0073] In step 101, device information of a target device and original operation data stream information generated by the target device are obtained.

[0074] Specifically, the electronic device obtains the device information of the target device through an application programming interface (API, Application Programming Interface) related to the device information.

[0075] Each UI operation performed by the user on the target device will generate corresponding operation data, which can be obtained through the getevent instruction and generate the original operation data stream shown in Table 1. The string shown in each line of the original operation data stream shown in Table 1 includes relative time, event identifier and identifier data.

[0076] The electronic device starts the getevent command of the target device through the debug bridge (adb, Android Debug Bridge) command. The original operation data stream obtained by getevent will be redirected to the electronic device for caching, so that the electronic device can obtain all the operation data generated by the user performing multiple consecutive UI operations on the terminal within the target time period.

[0077] Table 1 Original operation data flow

[0078] Relative time Event Identifier Identifier data [1685635.684049] ABS_MT_TRACKING_ID 000012d7 [1685635.684049] BTN_TOUCH DOWN [1685635.684049] BTN_TOOL_FINGER DOWN [1685635.684049] ABS_MT_POSITION_X 000000a0 [1685635.684049] ABS_MT_POSITION_Y 00000453 [1685635.684049] ABS_MT_PRESSURE 00000552 [1685635.684049] ABS_MT_TOUCH_MAJOR 00000001 [1685635.684049] SYN_REPORT 00000000 [1685635.807936] ABS_MT_POSITION_X 00000114 [1685635.807936] ABS_MT_POSITION_Y 00000454 [1685635.807936] SYN_REPORT 00000000 [1685635.851154] ABS_MT_POSITION_X 00000151 [1685635.851154] ABS_MT_POSITION_Y 0000045b [1685635.851154] ABS_MT_TOUCH_MAJOR 00000007 [1685635.851154] SYN_REPORT 00000000 [1685635.851154] ABS_MT_TRACKING_ID ffffffff

[0079] In step 102, a target identification interface corresponding to the target device is called based on the device information.

[0080] In the embodiment of the present application, different system types are distinguished by configuring a read-only memory (ROM). Specifically, the electronic device determines the system type corresponding to the target device based on the acquired device information, and the electronic device calls the corresponding target identification interface based on the system type of the target device.

[0081] For example, when the device information obtained is XX brand, it can be determined that the system corresponding to the current device is MIUI, and then the target identification interface corresponding to MIUI is called in the identification interface set.

[0082] In the embodiment of the present application, by matching different identification interfaces to different systems, the versatility of the data processing system in identifying the operation data type can be improved.

[0083] In step 103, based on the target recognition interface, data segmentation processing is performed on the original operation data stream information to obtain multiple operation data units.

[0084] Here, after the original operation data stream is cut and processed, a plurality of operation data units are obtained, and each operation data unit corresponds to a UI operation.

[0085] In some embodiments, see Figure 3B , Figure 3A The illustrated step 103 can be implemented by following the steps 31 to 33, which are described in detail below.

[0086] In step 31, a plurality of start identifiers and a plurality of end identifiers in the original operation data stream are determined based on the target identification interface.

[0087] As shown in Table 1, the original operation data stream is generally output in the form of a string array, wherein the original operation data stream contains multiple event data related to UI operations, and each event data contains a start identifier and an end identifier.

[0088] Since different manufacturers have customized their implementations of the Android system, there are certain differences between the systems of different types of devices, and the differences in the kernel versions of different systems exacerbate this difference. Therefore, different systems of different types of devices have different formats and sequence standards for the raw operation data streams obtained through the getevent instruction, for example, different formats of UI operation identifiers.

[0089] Through a large number of system comparisons and data analysis, the applicant found that each UI operation-related event data generated under different systems contains two special identifiers. Although the composition of the special identifier strings in different systems is different, they all mark the beginning and end of the UI operation respectively.

[0090] Since the original operation data stream generated by different target devices includes different preset start identifiers and end identifiers, the identification interface corresponding to the target device is set so that the identification interface can identify the corresponding preset start identifier and end identifier, thereby identifying all the start identifiers and end identifier positions contained in the original operation data stream. As shown in Table 1, the start identifier can be an ABS_MT_TRACKING_ID hexadecimal string, and the end identifier can be ABS_MT_TRACKING_ID ffffffff.

[0091] In step 32, the position indicated by each start identifier is used as the cutting start position, and the position indicated by the end identifier adjacent to the start identifier is used as the cutting end position.

[0092] Specifically, the original operation data stream is traversed to identify all the start identifier positions and end identifier positions contained therein, and then each start identifier position is set as the starting position of the cut (i.e., the cutting start position), and the end identifier position adjacent to the start identifier is set as the end position of the cut (i.e., the cutting end position).

[0093] In step 33, the original operation data stream information is cut based on the cutting start position and the cutting end position to obtain the multiple operation data units.

[0094] For example, the original operation data stream is cut according to all the cutting start positions and cutting end positions, and the electronic device cuts the original operation data stream into the following Figure 7 Multiple independent operation data units are shown, where each independent operation data unit represents all event data corresponding to a UI operation.

[0095] like Figure 7 As shown, each independent operation data unit includes a start identifier, multiple event data and an end identifier. In the original operation data stream, any start identifier is used as the cutting start position, and the position of the next adjacent end identifier of the start identifier is used as the cutting end position, and the cutting process is performed to obtain an independent operation data unit.

[0096] In the embodiment of the present application, the original operation data stream is cut into multiple independent operation data units, thereby improving the accuracy of subsequent UI operation type identification.

[0097] In step 104, based on the target identification interface, operation type identification processing is performed on the multiple operation data units to obtain the operation type of each operation data unit.

[0098] Here, by performing operation type identification processing on each operation data unit, all UI operation types included in the original operation data stream can be obtained.

[0099] In some embodiments, see Figure 3C , Figure 3A The illustrated step 104 can be implemented by following the steps 41 to 42, which are described in detail below.

[0100] In step 41, based on the target recognition interface, click type recognition processing is performed on multiple operation data units to obtain operation data units belonging to the click type.

[0101] Specifically, for the recognition of operation data units belonging to the click type, the core operation metadata mainly includes the click position, the size of the click area and the click operation time, among which the most core is the coordinate data of the click position, that is, the horizontal coordinate x and the vertical coordinate y of the click operation, but when the actual operation data unit is recognized, there are often multiple coordinate data. For example, the operation data generated when operating a click in the game interface includes multiple coordinate data.

[0102] In some embodiments, see Figure 3D , Figure 3C The illustrated step 41 can be implemented by following the steps 411 to 413, which are described in detail below.

[0103] In step 411, the coordinate pairs in each operation data unit are obtained.

[0104] Specifically, all character strings in each operation data unit are traversed, and all original coordinate data contained in the operation data unit are obtained according to the coordinate identifier, wherein the original coordinate data is saved in the form of hexadecimal data. Then, the original coordinate data is converted into corresponding coordinate data through the coordinate system to obtain the coordinate pair in each operation data unit, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate.

[0105] For example, in each operation data unit, all original data of horizontal coordinates are obtained through ABS_MT_POSITION_X, and all original data of vertical coordinates are obtained through ABS_MT_POSITION_Y, and all original data of coordinates are converted into decimal coordinate data to obtain all coordinate pairs of each operation data unit.

[0106] In step 412 , a coordinate bounding range is determined based on at least two coordinate pairs.

[0107] In some embodiments, see Figure 3E , Figure 3D The illustrated step 412 may be implemented by following steps 4121 to 4124, which are described in detail below.

[0108] In step 4121, the maximum horizontal coordinate, the minimum horizontal coordinate, the maximum vertical coordinate, and the minimum vertical coordinate are obtained.

[0109] Specifically, the maximum horizontal coordinate and the minimum horizontal coordinate are determined from the horizontal coordinates of all coordinate pairs contained in the current operation data unit; the maximum vertical coordinate and the minimum vertical coordinate are determined from the vertical coordinates of all coordinate pairs contained in the current operation data unit.

[0110] In step 4122, the maximum horizontal coordinate and the minimum horizontal coordinate are subtracted to obtain the horizontal coordinate limit range.

[0111] For example, when the maximum horizontal coordinate is 5 and the minimum horizontal coordinate is 3, the horizontal coordinate limit range is 2 pixels.

[0112] In step 4123, the maximum vertical coordinate and the minimum vertical coordinate are subtracted to obtain the vertical coordinate limit range.

[0113] For example, when the maximum vertical coordinate is 5 and the minimum vertical coordinate is 3, the vertical coordinate limit range is 2 pixels.

[0114] In step 4124, the horizontal axis limiting range and the vertical axis limiting range are determined as the coordinate limiting range.

[0115] In step 413 , in response to the coordinate limit range being smaller than the coordinate range threshold, the operation data unit is marked as an operation data unit belonging to a click type.

[0116] Here, all the character strings in each operation data unit are traversed, and the original data of the operation area size contained in the operation data unit is obtained according to the operation area identifier, wherein the operation area size is the reference data of the coordinate range threshold. Specifically, the operation area size can be set as the coordinate range threshold, or data smaller than the operation area size can be set as the coordinate range threshold. For example, when the operation area size is 5 pixels, the coordinate range threshold can be set to 5 pixels or 4 pixels.

[0117] Since the original data of the operation area size is saved in the form of hexadecimal data, it is necessary to convert the original data of the operation area size into the corresponding operation area size through the coordinate system to obtain the operation area size in each operation data unit.

[0118] For example, in each operation data unit, original data of the operation area size is obtained through ABS_MT_PRESSURE, and the original data of the operation area size is converted into decimal operation area size data to obtain the operation area size of each operation data unit.

[0119] The size of the operation area is set as a coordinate range threshold, and when both the horizontal axis limit range and the vertical axis limit range are smaller than the coordinate range threshold, the current operation data unit is marked as an operation data unit belonging to a click type.

[0120] For example, when the horizontal axis limit range is 2 pixels, the vertical axis limit range is 2 pixels, and the size of the converted operation area is 5 pixels, the current operation data unit is marked as an operation data unit belonging to the click type.

[0121] In some embodiments, the horizontal axis range threshold and the vertical axis range threshold can also be set according to the size of the operation area. When the horizontal axis limit range is smaller than the horizontal axis range threshold and the vertical axis limit range is smaller than the vertical axis range threshold, the current operation data unit is marked as a click type operation data unit.

[0122] In some embodiments, a coordinate pair in each operation data unit is obtained, and when the number of coordinate pairs is one pair, the operation data unit is directly marked as an operation data unit belonging to a click type, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate.

[0123] In step 42, sub-type identification processing is performed on the operation data units belonging to the click type to obtain the sub-operation type of each operation data unit belonging to the click type.

[0124] The sub-operation type is one of a single-click type, a double-click type, and a long-press type.

[0125] In some embodiments, see Figure 3F , Figure 3C The illustrated step 42 can be implemented by following the steps 421 to 422, which are described in detail below.

[0126] In step 421, feature analysis based on operation time is performed on the operation data unit belonging to the click type to obtain the operation data unit belonging to the candidate single-click type and the operation data unit belonging to the long-press type.

[0127] In some embodiments, see Figure 3G , Figure 3F The illustrated step 421 may be implemented by following steps 4211 to 4214, which are described in detail below.

[0128] In step 4211, multiple timestamps corresponding to the coordinate data in the operation data unit belonging to the click type are obtained.

[0129] Specifically, all character strings in the operation data unit of each click type are traversed to obtain each timestamp information corresponding to the coordinate data, where the timestamp can be an absolute clock value, that is, a time value consistent with the system clock; or a relative clock value, that is, the duration of the system from startup to the current operation time point.

[0130] In step 4212, the difference between the minimum timestamp and the maximum timestamp is determined as the operation time difference.

[0131] Specifically, the maximum timestamp and the minimum timestamp are obtained from all timestamps, the difference between the minimum timestamp and the maximum timestamp is calculated, and the obtained difference is set as the operation time difference.

[0132] In step 4213, in response to the operation time difference being greater than or equal to the time difference threshold, the operation data unit belonging to the click type is marked as an operation data unit belonging to the long press type.

[0133] Specifically, when the operation time difference is greater than or equal to the time difference threshold, a long press type identifier is added to the current operation data unit, and the current click type operation data unit is marked as a long press type operation data unit. The time difference threshold can be set according to actual needs.

[0134] In step 4214, in response to the operation time difference being less than the time difference threshold, the operation data unit belonging to the click type is marked as an operation data unit belonging to the candidate single-click type.

[0135] In an embodiment of the present application, after the original operation data stream information is cut and processed, multiple operation data units are obtained, and the type of the operation data units belonging to the click operation type in the multiple operation data units is identified. Based on the characteristics of the long press type operation, among the operation data units belonging to the click type, all the operation data units belonging to the long press type are determined. By identifying the cut operation data units, the accuracy of identifying the long press operation data is improved.

[0136] In step 422, feature analysis based on operation intervals is performed on the operation data units belonging to the candidate single-click type to obtain operation data units belonging to the single-click type and operation data units belonging to the double-click type.

[0137] In some embodiments, see Figure 3H , Figure 3F The illustrated step 422 may be implemented by following steps 4221 to 4226, which are described in detail below.

[0138] In step 4221, the following processing is performed for the first operation data unit among the operation data units belonging to the candidate single-click type: when the operation type of the next operation data unit of the first operation data unit is the candidate single-click operation type, the first representative coordinates and the first click time of the first operation data unit are obtained.

[0139] Among them, the representative coordinates are used to represent the click operation position of the UI operation corresponding to the current operation data unit, for example, the first representative coordinates are used to represent the click operation position of the UI operation corresponding to the first operation data unit; the click time is used to represent the click operation time of the UI operation corresponding to the current operation data unit, for example, the first click time is used to represent the click operation time of the UI operation corresponding to the first operation data unit.

[0140] Specifically, two consecutive operation data units are obtained from the operation data units belonging to the candidate single-click type, the first operation data unit of the two consecutive operation data units is taken as the first operation data unit, and the first representative coordinates and the first click time of the first operation data unit are obtained.

[0141] In some embodiments, see Fig. 3I , Figure 3H The shown step 4221 can be implemented by following the steps 42211 to 42213, which are described in detail below.

[0142] In step 42211, at least one coordinate pair in the first operation data unit is obtained.

[0143] The coordinate pair includes a horizontal coordinate and a vertical coordinate.

[0144] In step 42212, a coordinate pair in the first operation data unit is determined as a first representative coordinate.

[0145] Among them, the method for determining the first representative coordinates can be to randomly select a coordinate pair in the first operation data unit as the first representative coordinates, or to use a center position coordinate pair as the first representative coordinates, wherein the center position coordinates represent the coordinates of the center position of the polygon formed by all coordinate pairs in the first operation data unit. The specific method for determining the first representative coordinates is not limited in the embodiment of the present application.

[0146] In step 42213, the timestamp corresponding to the first representative coordinate is determined as the first click time.

[0147] Specifically, in the first operation data unit, the timestamps corresponding to the horizontal coordinate and the vertical coordinate in the first representative coordinate are obtained, and the obtained timestamps are determined as the first click time.

[0148] In step 4222, the second representative coordinates and the second click time of the next operation data unit are obtained.

[0149] The method for obtaining the second representative coordinates and the second click time is the same as the method for obtaining the first representative coordinates and the first click time, which will not be repeated here.

[0150] In step 4223, the coordinate difference between the first representative coordinates and the second representative coordinates is determined.

[0151] Specifically, the abscissa of the first representative coordinate and the abscissa of the second representative coordinate are obtained, and the coordinate difference between the two abscissas is calculated; the ordinate of the first representative coordinate and the ordinate of the second representative coordinate are obtained, and the coordinate difference between the two ordinates is calculated.

[0152] In step 4224, the time interval between the first click time and the second click time is determined.

[0153] Specifically, the difference between the second click time and the first click time is calculated to obtain the time interval.

[0154] In step 4225, when the coordinate difference is greater than the coordinate difference threshold, or when the coordinate difference is less than or equal to the coordinate difference threshold and the time interval is greater than the time interval threshold, the first operation data unit is marked as an operation data unit belonging to a single-click type.

[0155] As an example, when the coordinate difference is greater than the coordinate difference threshold, it indicates that there is an obvious deviation between the first representative coordinate and the second representative coordinate. At this time, it can be directly obtained that the first operation data unit belongs to a single-click type operation data unit.

[0156] As another example, when the coordinate difference is less than or equal to the coordinate difference threshold, since the double-click operation includes two click operations, the operation type of the first operation data unit can be determined by determining the time interval between the two operations. Specifically, when the time interval is greater than the time interval threshold, the first operation data unit is marked as an operation data unit belonging to the single-click type; when the time interval is less than or equal to the time interval threshold, the first operation data unit is marked as an operation data unit belonging to the single-click type.

[0157] In step 4226, when the coordinate transformation distance is less than or equal to the coordinate difference threshold and the time interval is less than or equal to the time interval threshold, the first operation data unit and the next operation data unit are merged into an operation data unit belonging to the double-click type.

[0158] In some embodiments, see Figure 3J , Figure 3H The shown step 4226 can be implemented by following the steps 42261 to 42262, which are described in detail below.

[0159] In step 42261, first operation metadata of the first operation data unit and second operation metadata of the next operation data unit are obtained.

[0160] The first operation metadata is data used to describe the first operation data unit, and the second operation metadata is data used to describe the second operation data unit. For example, the operation metadata of the long press operation includes the long press type identifier, the click position, the click area size, and the click duration.

[0161] In step 42262, a double-click operation identifier is added to the first operation metadata and the second operation metadata.

[0162] The double-click operation identifier is used to identify the first operation data unit, and the next operation data unit is combined to obtain a data unit that belongs to the double-click type operation data unit. Figure 8 A schematic diagram of a double-click operation data unit provided in an embodiment of the present application. Figure 8 As shown, each double-click operation data unit is composed of two candidate single-click type event data.

[0163] In an embodiment of the present application, multiple operation data units are obtained after the original operation data stream information is cut and processed, and the type of the operation data units belonging to the candidate click operation type in the multiple operation data units is identified. Based on the characteristics of the double-click type operation, the operation data units belonging to the double-click type and the single-click type are determined in the operation data units belonging to the candidate click type. By identifying the cut operation data units, the accuracy of identifying the double-click and single-click operation data is improved.

[0164] In an embodiment of the present application, a data processing method is proposed (implemented by a method for identifying UI operation types based on the original operation data stream generated by the target device), which realizes universal type identification of data streams of different systems, and improves the accuracy of UI operation type identification by cutting the original operation data and then performing UI operation type identification.

[0165] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0166] The existing data processing model cannot use a unified method to identify the original operation data generated in different systems, and cannot accurately identify all the operation types contained in the original operation data. The embodiment of the present application proposes a data processing method, which obtains the target device information to call the identification interface corresponding to the target device, then cuts and processes the original operation data stream information to obtain the operation data unit corresponding to each UI operation, and finally performs type processing on each operation data unit to obtain the operation type to which each operation data unit belongs, thereby improving the accuracy of operation type identification.

[0167] In the embodiment of the present application, a combination of app-process application process and getevent is combined, and based on the image recognition capability of the opencv open source library, a general recording and playback solution for a mobile real machine environment is implemented, and a back-end recording and playback engine set is started on the electronic device through app-process.

[0168] Specifically, based on getevent, all operation data of the Android system in a specific time period are detected in real time, and these operation metadata are automatically recorded after algorithm analysis and logical processing conversion. In the process, all screen image data are also recorded in real time, and the UI image area covered by the operation is automatically identified through the opencv open source library, and the universal UI recording and playback technology is realized by simulating the regular operations of the real machine.

[0169] Since it involves the application of app-process, the embodiment of the present application also briefly introduces and explains the executable program on the Android system. App-process is actually a system-level executable program pre-embedded in the Android system, located in the / system / bin / directory of the Android file system, where the original application process zygote is also constructed by this executable program. However, although ordinary Apps are forked from the zygote process, various permissions are generally restricted in the initialization logic related to the fork. Therefore, applications started by conventional means (such as clicking on the application icon) in the Android system are generally processes with restrictive permissions. However, although the executable program of app-process cannot be started using the ordinary App process, it can be driven and opened by the adb executable program of a personal computer (PC, Personal Computer), so the Linux process can be started by executing the adb executable program on the PC side, wherein the execution of the adb executable program enables the PC side to have system-level execution permissions.

[0170] like Figure 5 As shown in the figure, when the process is started through app-process, the startup logic of app-process is different from that of other apps. The process started in this way belongs to the native Java process of the Linux system. Since app-process inherits the init process with pid 1 by default, the process started in this way naturally has the shell permission capability corresponding to the init process. This is the core dependency of getevent in the recording function.

[0171] When a process with shell permissions is started through app-process, the process can execute system-level shell commands through the Java process component. Another core capability of the process started by app-process is to detect and collect data of UI operation events based on getevent.

[0172] Getevent is an executable program built into Android. By executing this program, all kernel event data (i.e., raw operation data stream) in a specific format serialized in the Android event system can be detected in real time. In this process, all data that can be collected by UI operations under this framework, such as UI element images, UI operation metadata, UI operation types, etc., are parsed based on the raw operation data stream collected by getevent. The raw operation data stream is the character stream output by the event subsystem in the Android kernel.

[0173] The embodiment of the present application first performs some targeted analysis and identification on click, double-click, long press and slide. For the original operation data stream shown in Table 1, the embodiment of the present application identifies the number of UI operations in the original operation data stream and the type of operation data unit.

[0174] In addition to identifying the number of UI operations, the embodiments of the present application also use data features to determine the type of operation data unit and its corresponding core operation metadata. It is necessary to explain the content of the operation metadata here. The so-called operation metadata refers to some necessary descriptive data during the operation. For example, the operation metadata of the click operation includes the plane coordinate data of the click position, the click time point, and the area size during the click operation; the operation metadata of the long press operation includes the plane coordinates of the click position, the long press duration, etc.; the operation metadata of the double-click operation includes the plane coordinate data of the click position, the interval time between the two clicks, etc.; the operation metadata of the sliding operation is the set of coordinate data points during the sliding process, the screen trajectory of the sliding operation, the sliding duration, the sliding operation frequency, etc. The above metadata are also identified by the embodiments of the present application through data features and specific analysis algorithms, as the core capability foundation of UI recording and playback technology.

[0175] The original operation data stream obtained based on getevent has the problems of poor readability, high complexity and large format differences of system ROM of different models. The embodiments of the present application make corresponding optimizations from various aspects to solve the above problems respectively.

[0176] Regarding the readability of the original operation data stream, the embodiment of the present application automatically distinguishes based on data features and UI operation characteristics, and can basically accurately identify complex UI compound operations in a certain process, and divide the UI operations into corresponding operation data units.

[0177] The problem of high complexity of the original operation data stream is mainly because when continuous UI operations are performed, especially when the types of UI operations are different and there is no fixed operation rule, the complexity of the original operation data stream obtained from getevent is usually much higher than that of a single operation. In addition, due to continuous operations, the original operation data stream may display a combination of event data corresponding to multiple continuous UI operations. The embodiment of the present application specifically cuts out the continuous operation data unit according to the continuous data features and time series.

[0178] The problem of large format differences between ROM types of original operation data streams is mainly due to the differences in key identification characters in some specific data items in the data format output by getevent. For example, in some x86 architecture systems, the identifier ABS_MT_PRESSURE may exist in the output data items, but in arm architecture systems, this identifier may not exist. This difference leads to increased difficulty and complexity in parsing. The embodiments of the present application set different identification interfaces for different ROM types to specifically identify the operation data units in the original operation data stream.

[0179] Figure 6 The hierarchical logical structure of a cutting recognition algorithm provided in the embodiment of the present application is described in detail below. Figure 6 As shown, the overall logic of the entire solution can be divided into four layers according to the hierarchical structure.

[0180] The first layer in the hierarchical logical structure is the system of a specific Android ROM. Four Android systems are given as examples here. The getevent instructions in the above four ROM systems basically have the problem of different original operation data stream identifiers. Figure 6 The systems in this article are not specified by model, but are differentiated based on broad categories.

[0181] The second layer in the hierarchical logical structure is used to detect the original data source of getevent. The getevent command is used in the recording and playback technical solution. In a Java process, the shell capability is used to continuously obtain the data stream obtained through redirection in the process cache. These data streams are string array collections similar to those in Table 1.

[0182] The third layer in the hierarchical logical structure is the cornerstone of this solution. Different types of Android ROMs are distinguished by configuring the ROM. The specific information of the current device can be obtained through the relevant Android API interface. This layer is actually to abstract the common capabilities, and then set different identification interfaces corresponding to different devices through configuration. Specifically, this level contains an abstract interface set, which contains multiple abstract interfaces. Each abstract interface corresponds to an algorithm set for a UI operation type. The algorithm set corresponds to the adapted algorithms in different ROMs according to the configuration information.

[0183] The fourth layer (i.e. the top layer) in the hierarchical logical structure is the supported various event data UI operation type analysis algorithms, which are used to identify click, double-click and long press types. The core idea of ​​each algorithm is customized according to the characteristics of the event original event data. On different ROMs, the main difference lies in the adaptation processing of special string array identifiers.

[0184] See also Figure 4A , Figure 4A This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 4A The steps shown in are explained.

[0185] In step 501, the electronic device traverses the operation data unit.

[0186] The operation data unit is obtained by cutting the original operation data stream, and each operation data unit corresponds to a UI operation.

[0187] The embodiment of the present application can obtain the original operation data stream in a specific time period through getevent, and the original operation data stream is generally displayed in the form of a string array. Through a large number of ROM comparisons and regularity analysis, it is found that each UI operation operation data unit has two special character flags. Although the character flags may be inconsistent in different ROMs, there are basically these two special character flags that mark the start and end of the UI operation event respectively.

[0188] The original operation data stream obtained by getevent is a string array, which can be simply understood as a string array for each line. Through the analysis and comparison of a large amount of data, it is found that for each UI operation, there are always two character marks in these continuous string arrays, namely:

[0189] (1) Start identifier: ABS_MT_TRACKING_ID followed by a string of random hexadecimal numbers, which indicates the start position of a UI operation event;

[0190] (2) End identifier: ABS_MT_TRACKING_ID ffffffff, which indicates the end position of a UI operation event;

[0191] Through the two marks introduced above, the embodiment of the present application can perform a filtering analysis on the entire original operation data stream, and divide and cut out all the operation data units according to the positions of the start identifier and the end identifier, and obtain the following Figure 7 There are multiple independent operation data units shown, and each operation data unit contains all event data corresponding to a UI operation.

[0192] In step 502, all string arrays of the operation data unit are traversed.

[0193] The electronic device obtains all string arrays contained in the current operation data unit, identifies the content of the string array, and obtains the event data contained therein.

[0194] For click operations, the core operation metadata mainly includes the click position, click area size, and click operation time. The most core is the coordinate data of the click position, that is, the x data and y data of the click operation. However, in actual complex UI scenarios, such as when operating clicks in game interfaces, there are often multiple x and y data, which is related to the specific ROM and game scene characteristics.

[0195] In step 503 , it is determined whether the string array contains x-coordinate information. When the string array contains x-coordinate information, step 504 is executed; otherwise, step 505 is executed.

[0196] By traversing all string arrays one by one, each string array will determine whether there is x-coordinate information based on its content. The original operation data stream generally contains ABS_MT_POSITION_X information. Specifically, it is determined whether the ABS_MT_POSITION_X identifier is recognized in the string array. When the ABS_MT_POSITION_X identifier is recognized, it indicates that the current string array contains x-coordinate information.

[0197] In step 504, the original data is acquired and converted to obtain the x-coordinate.

[0198] The original data of the ABS_MT_POSITION_X information is cached and converted to the corresponding x coordinate through the coordinate system. The original data format is saved in hexadecimal data, so special conversion is required.

[0199] In step 505 , it is determined whether the string array contains y-coordinate information. When the string array contains y-coordinate information, step 506 is executed; otherwise, step 507 is executed.

[0200] Specifically, determine whether the ABS_MT_POSITION_Y flag is recognized in the string array. If the ABS_MT_POSITION_Y flag is recognized, it indicates that the y coordinate information exists in the current string array.

[0201] In step 506, the original data is acquired and converted to obtain the y coordinate.

[0202] The original data of the ABS_MT_POSITION_Y information is cached and converted to the corresponding y coordinate through the coordinate system. The original data format is saved in hexadecimal data, so special conversion is required.

[0203] In step 507 , it is determined whether the character string array contains the operation area identifier. If the character string array contains the operation area identifier, step 508 is executed; otherwise, step 509 is executed.

[0204] Specifically, it is determined whether the ABS_MT_PRESSURE identifier is recognized in the string array. When the ABS_MT_PRESSURE identifier is recognized, it indicates that the operation area information exists in the current string array.

[0205] In step 508, a coordinate range threshold is obtained.

[0206] The original data of ABS_MT_PRESSURE information is cached and converted into the corresponding operation area size through the coordinate system, and the operation area size is set as the coordinate range threshold. Among them, the format of the original data is saved in hexadecimal data, so special conversion is required.

[0207] In step 509 , it is determined whether the traversal is completed. If the traversal is completed, step 510 is executed; otherwise, step 502 is executed.

[0208] Each time the string array is traversed, the above three judgment processes will be cycled in turn until all the string array data in the operation data unit is parsed, and the entire algorithm completes the preliminary analysis. After completing all the above loop traversal processes and obtaining all the data information in the operation data unit, the UI type recognition begins.

[0209] In step 510 , it is determined whether multiple coordinate pairs are obtained. If multiple coordinate pairs exist, step 511 is executed; otherwise, step 512 is executed.

[0210] In step 511 , it is determined whether the coordinate limiting range is smaller than the coordinate limiting range threshold. When the coordinate limiting range is smaller than the coordinate limiting range threshold, step 513 is executed; otherwise, step 501 is executed.

[0211] Specifically, the coordinate limiting range is determined according to the size of the operation area, wherein the coordinate limiting range includes a horizontal coordinate limiting range and a vertical coordinate limiting range. For example, when the size of the operation area is 5 pixels, the horizontal coordinate limiting range and the vertical coordinate limiting range in the coordinate limiting range are both 5 pixels.

[0212] Because there may be multiple x and y coordinate data during the acquisition process, the minimum horizontal coordinate value Xmin and the minimum vertical coordinate value Ymin, as well as the maximum horizontal coordinate value Xmax and the maximum vertical coordinate value Ymax are selected from all the acquired coordinate data. The difference between Xmax and Xmin is used as the horizontal coordinate limit range, and the difference between Ymax and Ymin is used as the vertical coordinate limit range.

[0213] When the horizontal axis limit range is smaller than the size of the operation area, and the vertical axis limit range is smaller than the size of the operation area, step 513 is executed.

[0214] In step 512 , it is determined whether a single coordinate pair is obtained. If a single coordinate pair is obtained, step 513 is executed; otherwise, step 501 is executed.

[0215] In step 513, the current operation data unit is marked as a click type operation data unit.

[0216] In step 514, the timestamp information corresponding to the representative coordinates is obtained.

[0217] When there are multiple coordinate pairs, any one coordinate pair is selected as the representative coordinate of the current operation data unit. When there is only one coordinate pair, the obtained coordinate pair is used as the representative coordinate of the current operation data unit.

[0218] In each string array filtered above, each data contains a timestamp information of the corresponding event data, such as the data on the far left in Table 1. It is worth noting that these times are not absolute clock values, but relative clock values. Generally speaking, the absolute clock value is the time that is relatively consistent with the current system clock, which is equivalent to the current time commonly said; while the relative clock value is the time from the Linux system booting to the current time point, recording the duration of the current system from startup to operation.

[0219] In step 515, the operation time difference is calculated.

[0220] Here, the difference between the maximum timestamp value and the minimum timestamp value obtained is calculated, and the calculated difference is used as the operation time difference.

[0221] In step 516, it is determined whether the operation time difference is greater than or equal to the time difference threshold. When the operation time difference is greater than or equal to the time difference threshold, step 517 is executed; otherwise, step 518 is executed.

[0222] In step 517, the current operation data unit is marked as a long press type operation data unit.

[0223] In step 518, the current operation data unit is marked as a candidate single-click type operation data unit.

[0224] In step 519, the operation data unit is serialized and cached.

[0225] Among them, operation serialization is to convert the operation data unit into a standardized format so that the operation data unit can keep its original content unchanged when used across platforms and programs.

[0226] In the embodiment of the present application, after obtaining the operation data units belonging to the candidate single-click type, a second type identification is performed on all the operation data units belonging to the candidate single-click type. Figure 4B , Figure 4B This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 4B The steps shown in are explained.

[0227] A double-click operation can be understood as two consecutive single-click operations. The core operation metadata of the double-click operation mainly includes the click position, click area size, and click operation time of the two consecutive operations. The double-click operation is identified by judging the time interval between the two operations in two consecutive operation data units belonging to the candidate single-click type.

[0228] In step 601, the electronic device traverses operation data units belonging to candidate single-click types.

[0229] Here, the embodiment of the present application will divide the entire original event data stream into multiple separate operation data units, which will be managed using a linked list data structure. When the traversal analysis finds that the current operation data unit is an operation data unit of a candidate single-click type, it will continue to determine whether the subsequent operation data unit in the linked list is also a candidate single-click type.

[0230] In step 602 , it is determined whether the operation data unit currently belonging to the candidate single-click type has a subsequent operation data unit. If there is a subsequent operation data unit, step 603 is executed; otherwise, step 609 is executed.

[0231] In step 603 , it is determined whether the subsequent operation data unit belongs to a candidate single-click type operation data unit. If it belongs to the candidate single-click type, step 604 is executed; otherwise, step 609 is executed.

[0232] If the subsequent operation data unit is not a candidate single-click type, the type identification of the current operation data unit is terminated directly, and the judgment of the next operation data unit is continued; if the subsequent node is also a click-type data unit, its core operation metadata is further compared.

[0233] Among them, the core operation metadata that can be compared mainly has two parts. One is the operation position of the candidate single-click operation event, that is, the representative coordinates. When the difference between the representative coordinates of the two operations is less than the coordinate difference threshold, the operation time is further compared; when the difference between the representative coordinates of the two operations is greater than or equal to the coordinate difference threshold, it is considered that the two consecutive candidate single-click events cannot be combined into a double-click event.

[0234] Another data to be compared is the operation time of two consecutive candidate single-click events. From the previous introduction, we can know that after the data of each click type event is analyzed, the operation time will be parsed from the original operation data stream. By comparing the operation time interval of two consecutive click events, it can be determined whether there is a double-click type operation. Generally speaking, if the operation time interval of two click events is less than the time difference threshold, it is considered that they can be combined into a double-click operation event; otherwise, it is considered that the current operation data unit corresponds to a single-click operation event.

[0235] In step 604, representative coordinates of two operation data units are obtained.

[0236] Specifically, a coordinate pair is arbitrarily selected in the current operation data unit as the first representative coordinate, and a coordinate pair is arbitrarily selected in the subsequent operation data unit of the current operation data unit as the second representative coordinate.

[0237] In step 605 , it is determined whether the coordinate difference value is less than the coordinate difference threshold value. When the coordinate difference value is less than the coordinate difference threshold value, step 606 is executed; otherwise, step 609 is executed.

[0238] The horizontal coordinate difference and the vertical coordinate difference of the first representative coordinate and the second representative coordinate are calculated respectively, and the horizontal coordinate difference and the vertical coordinate difference are compared with the coordinate difference threshold respectively. When the horizontal coordinate difference is less than the coordinate difference threshold, and the vertical coordinate difference is less than the coordinate difference threshold, it is determined that the coordinate difference is less than the coordinate difference threshold, and step 606 is executed.

[0239] In step 606, the time between two click operations is obtained.

[0240] First timestamp information of the first representative coordinate and second timestamp information of the second representative coordinate are obtained.

[0241] In step 607 , it is determined whether the operation time difference is less than a time difference threshold. When the operation time difference is less than the time difference threshold, step 608 is executed; otherwise, step 609 is executed.

[0242] The difference between the first timestamp and the second timestamp is calculated to obtain the operation time difference between the two click operations.

[0243] In step 608, the current operation data unit is marked as a double-click type operation data unit.

[0244] Specifically, a double-click identifier is added to the current operation data unit, and the current operation data unit and the subsequent data operation unit are merged to obtain the following: Figure 8 The data unit shown belongs to the double-click type operation.

[0245] In step 609, the current operation data unit is marked as a single-click type operation data unit.

[0246] After marking the current operation data unit as a single-click type operation data unit, continue to perform pairwise comparative analysis of the operation data units until all UI operation unit data are parsed, and the entire algorithm completes the double-click type analysis.

[0247] In step 610 , it is determined whether the traversal is completed. If the traversal is completed, step 611 is executed; otherwise, step 601 is executed.

[0248] In step 611, the operation data unit is serialized and cached.

[0249] Among them, operation serialization is to convert the operation data unit into a standardized format so that the operation data unit can keep its original content unchanged when used across platforms and programs.

[0250] The principle of the double-click type recognition algorithm is to analyze and compare the data list of the candidate single-click type operation data unit one by one. If there are two consecutive candidate single-click operations, and the location, area, and operation time interval of the two operations meet the specified conditions, the algorithm will merge the two consecutive click operations in this case into the same double-click type UI operation. The merging process includes: adding a double-click operation identifier to the metadata of the candidate single-click type operation data unit, and the double-click operation identifier is used as the main identifier to distinguish it from the single-click operation.

[0251] In the embodiment of the present application, by performing type identification on the operation data unit, firstly, all the operation data units belonging to the click type are identified in the operation data unit; then, according to the operation time, all the operation data units belonging to the long press type are obtained; finally, the operation data units belonging to the candidate single-click type are subjected to a second type identification, and based on the operation time and representative coordinate positions between adjacent operations, the operation data units belonging to the single-click type and the operation data units belonging to the double-click type are determined. The above steps can realize accurate identification of the long press operation, the single-click operation and the double-click operation.

[0252] According to the above introduction, the basic working principle of the embodiment of the present application includes:

[0253] 1) First, taking the Android system as an example, we rely on the data collection capability of Android's getevent to collect event system data to obtain the original operation data stream, which includes event data corresponding to multiple continuous UI operations, such as a data sequence of continuous operations such as a click, a double-click, and a long press;

[0254] 2) You need to execute the getevent-lt terminal command through adb shell or directly through the shell capability in any Android device that supports getevent. This command is actually equivalent to executing it in the Linux shell environment. After executing this command, the electronic device will output relevant Android input information, including information related to the touch screen. Since each Android ROM may be different, it is not shown here.

[0255] 3) After executing step 2, the user can start to perform continuous UI operations on the Android device, such as performing three different UI operations of a click, a double click, and a long press. After completing the UI operation, the electronic device started in step 2 will continuously output the corresponding original operation data stream.

[0256] 4) The core architecture of this solution is as follows Figure 6 As shown, the processing logic mainly starts from the getevent layer. After obtaining the relevant string array from the original operation data stream, the original operation data stream is used as input data to call the algorithm interface provided by the embodiment of the present application.

[0257] 5) When the algorithm starts, the data cutting algorithm will be used for initial analysis. Each operation data unit will be cut into multiple independent operation data units according to the corresponding position of the start identifier and the corresponding position of the end identifier. Figure 7 In the data units shown in , each data unit starts with a fixed start identifier and ends at an end identifier. All string arrays between the start identifier and the end identifier belong to the content of the same operation data unit.

[0258] 6) Continue to perform type analysis on all operation data units. The entire analysis process is actually divided into two loops, each loop will analyze all operation data units once. The first loop will analyze the Figure 4AThe identification algorithm shown in the figure performs type identification and analysis on all operation data units. The first cycle can filter out all operation data units belonging to click type operations, and determine the operation data units belonging to candidate single click types and the operation data units belonging to long press types among the operation data units belonging to click type operations.

[0259] 7) After the first recognition, a second cycle recognition will be carried out. The second cycle recognition will be based on Figure 4B The recognition algorithm introduced performs feature analysis on all operation data units belonging to the candidate single-click type analyzed in the first cycle. If there are two consecutive operation data units belonging to the candidate single-machine type that meet the double-click condition, the two operation data units belonging to the candidate single-click are combined into a double-click event to identify whether a double-click operation exists.

[0260] 8) After completing the above two loop processing, through the above process, it is possible to support the original operation data stream collected by getevent on any Android device, and cut and identify the collected original operation data stream. After identification, the operation data unit corresponding to each UI operation is parsed into the data format required by most UI automation frameworks through serialization.

[0261] After using this solution, on any Android electronic device, after starting the getevent command that the present invention relies on from the adb command on the PC terminal, the user can perform UI operations on the corresponding Android mobile phone device. When performing UI operations, the original event data stream obtained by getevent will be redirected to the terminal cache. After calling the recognition algorithm provided by this solution, all operations in the entire continuous UI operation process can be accurately identified, and the corresponding operation type can be serialized into relevant data.

[0262] At the same time, the recognition algorithm of the embodiment of the present application can basically adapt to various Android ROMs, and well solve the algorithm robustness problem caused by the format difference of getevent in various ROMs. Therefore, the UI automation technology integrated with this solution, such as UI recording and playback, can well support all UI automation on Android devices and make full use of the data in the process and further structure it for analysis and use by the automation framework.

[0263] Since this solution is compatible with various Android models, it has relatively good versatility. In addition, it further identifies some common UI operation types such as long press and double click through type analysis. Therefore, this solution can provide a highly robust data collection capability for Android UI automation.

[0264] The following further describes an exemplary structure of the data processing device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules stored in the data processing device 255 of the memory 250 may include:

[0265] The acquisition module 2551 is used to acquire device information of the target device and original operation data stream information generated by the target device.

[0266] The calling module 2552 is used to call the target identification interface corresponding to the target device based on the device information.

[0267] The cutting module 2553 is used to perform data cutting processing on the original operation data stream information based on the target identification interface to obtain multiple operation data units.

[0268] The identification module 2554 is used to perform operation type identification processing on the multiple operation data units based on the target identification interface to obtain the operation type of each of the operation data units.

[0269] In some embodiments, the cutting module 2553 is also used to determine multiple start identifiers and multiple end identifiers in the original operation data stream based on the target identification interface; use the position indicated by each of the start identifiers as the cutting start position, and use the end identifier adjacent to the start identifier as the cutting end position; based on the cutting start position and the cutting end position, the original operation data stream information is cut and processed to obtain the multiple operation data units.

[0270] In some embodiments, the identification module 2554 is also used to perform the following processing based on the target identification interface: performing click type identification processing on the multiple operation data units to obtain operation data units belonging to the click type; performing sub-type identification processing on the operation data units belonging to the click type to obtain the sub-operation type of each operation data unit belonging to the click type, wherein the sub-operation type is one of a single-click type, a double-click type, and a long press type.

[0271] In some embodiments, the identification module 2554 is also used to obtain a coordinate pair in each of the operation data units, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; when the number of the coordinate pairs is at least two pairs, a coordinate limiting range is determined based on at least two pairs of the coordinate pairs; in response to the coordinate limiting range being less than a coordinate range threshold, the operation data unit is marked as an operation data unit belonging to a click type.

[0272] In some embodiments, the identification module 2554 is also used to obtain the maximum horizontal coordinate, the minimum horizontal coordinate, the maximum vertical coordinate and the minimum vertical coordinate from at least two pairs of the coordinate pairs; to perform a subtraction process on the maximum horizontal coordinate and the minimum horizontal coordinate to obtain a horizontal coordinate limited range; to perform a subtraction process on the maximum vertical coordinate and the minimum vertical coordinate to obtain a vertical coordinate limited range; and to determine the horizontal coordinate limited range and the vertical coordinate limited range as the coordinate limited range.

[0273] In some embodiments, the identification module 2554 is also used to obtain a coordinate pair in each of the operation data units, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; when the number of the coordinate pairs is one pair, the operation data unit is marked as the operation data unit belonging to the click type.

[0274] In some embodiments, the identification module 2554 is further used to perform a feature analysis based on the operation time on the operation data unit belonging to the click type, and obtain an operation data unit belonging to the candidate single-click type and an operation data unit belonging to the long press type; and to perform a feature analysis based on the operation interval on the operation data unit belonging to the candidate single-click type, and obtain an operation data unit belonging to the single-click type and an operation data unit belonging to the double-click type.

[0275] In some embodiments, the identification module 2554 is also used to obtain multiple timestamps corresponding to the coordinate data in the operation data unit belonging to the click type; determine the difference between the minimum timestamp and the maximum timestamp as the operation time difference; in response to the operation time difference being greater than or equal to a time difference threshold, mark the operation data unit belonging to the click type as an operation data unit belonging to the long press type; in response to the operation time difference being less than the time difference threshold, mark the operation data unit belonging to the click type as an operation data unit belonging to the candidate single-click type.

[0276] In some embodiments, the identification module 2554 is also used to perform the following processing on the first operation data unit among the operation data units belonging to the candidate single-click type: when the operation type of the next operation data unit of the first operation data unit is the candidate single-click operation type, obtain the first representative coordinates and the first click time of the first operation data unit, and obtain the second representative coordinates and the second click time of the next operation data unit; determine the coordinate difference between the first representative coordinates and the second representative coordinates, and determine the time interval between the first click time and the second click time; when the coordinate difference is greater than the coordinate difference threshold, or when the coordinate difference is less than or equal to the coordinate difference threshold, and the time interval is greater than the time interval threshold, mark the first operation data unit as an operation data unit belonging to the single-click type; when the coordinate transformation distance is less than or equal to the coordinate difference threshold, and the time interval is less than or equal to the time interval threshold, merge the first operation data unit and the next operation data unit into an operation data unit belonging to the double-click type.

[0277] In some embodiments, the identification module 2554 is also used to obtain the first operation metadata of the first operation data unit and the second operation metadata of the next operation data unit; add a double-click operation identifier to the first operation metadata and the second operation metadata, wherein the double-click operation identifier is used to identify that the combination of the first operation data unit and the next operation data unit belongs to an operation data unit of the double-click type.

[0278] In some embodiments, the identification module 2554 is also used to obtain at least one pair of coordinate pairs in the first operation data unit, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; determine one of the coordinate pairs in the first operation data unit as the first representative coordinate; and determine the timestamp corresponding to the first representative coordinate as the first click time.

[0279] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data processing method described above in the embodiment of the present application.

[0280] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the data processing method provided by the present application embodiment, for example, Figure 3AThe data processing method is shown.

[0281] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0282] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and the computer executable instructions may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0283] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0284] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0285] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0286] In summary, the embodiment of the present application obtains all operation data units after cutting the acquired original operation data stream, and identifies the UI operation type corresponding to each operation data unit. Through the data processing method of the embodiment of the present application, the data generated by user terminal devices of different systems can be identified, which improves the versatility of UI operation type identification. Moreover, the embodiment of the present application further identifies the type of analysis for some common UI operation types such as long press and double click, which can provide a more robust data collection capability for the Android UI automatic recognition system.

[0287] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, It is characterized in that The method comprises: Acquire device information of a target device and original operation data stream information generated by the target device; Calling a target identification interface corresponding to the target device based on the device information; Based on the target identification interface, data segmentation processing is performed on the original operation data stream information to obtain multiple operation data units; Based on the target identification interface, operation type identification processing is performed on the multiple operation data units to obtain the operation type of each of the operation data units.

2. The method according to claim 1, It is characterized in that The data segmentation process is performed on the original operation data stream information based on the target identification interface to obtain a plurality of operation data units, including: Based on the target identification interface, determining a plurality of start identifiers and a plurality of end identifiers in the original operation data stream; Taking the position indicated by each of the start identifiers as the cutting start position, and taking the position indicated by the end identifier adjacent to the start identifier as the cutting end position; Based on the cutting start position and the cutting end position, the original operation data stream information is cut and processed to obtain the multiple operation data units.

3. The method according to claim 1, It is characterized in that The performing operation type identification processing on the multiple operation data units based on the target identification interface to obtain the operation type of each of the operation data units includes: The following processing is performed based on the target recognition interface: Performing click type identification processing on the multiple operation data units to obtain operation data units belonging to the click type; Sub-type identification processing is performed on the operation data unit belonging to the click type to obtain the sub-operation type of each operation data unit belonging to the click type, wherein the sub-operation type is one of a single-click type, a double-click type and a long-press type.

4. The method according to claim 3, It is characterized in that The step of performing click type identification processing on the plurality of operation data units to obtain operation data units belonging to the click type includes: Acquire a coordinate pair in each of the operation data units, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; When the number of the coordinate pairs is at least two, determining a coordinate limiting range based on the at least two coordinate pairs; In response to the coordinate limited range being smaller than a coordinate range threshold, marking the operation data unit as the operation data unit belonging to the click type.

5. The method according to claim 4, It is characterized in that The determining of the coordinate limiting range based on at least two of the coordinate pairs comprises: From at least two pairs of the coordinate pairs, obtain a maximum abscissa, a minimum abscissa, a maximum ordinate, and a minimum ordinate; Performing a subtraction process on the maximum horizontal coordinate and the minimum horizontal coordinate to obtain a horizontal coordinate limited range; Performing a difference process on the maximum ordinate and the minimum ordinate to obtain a ordinate limit range; The horizontal axis limiting range and the vertical axis limiting range are determined as the coordinate limiting range.

6. The method according to claim 3, It is characterized in that The step of performing click type identification processing on the plurality of operation data units to obtain operation data units belonging to the click type includes: Acquire a coordinate pair in each of the operation data units, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; When the number of the coordinate pairs is one pair, the operation data unit is marked as the operation data unit belonging to the click type.

7. The method according to claim 3, It is characterized in that The performing sub-type identification processing on the operation data unit belonging to the click type to obtain the sub-operation type of each operation data unit belonging to the click type includes: Performing feature analysis based on operation time on the operation data unit belonging to the click type to obtain operation data units belonging to candidate single-click types and operation data units belonging to long-press types; The operation data units belonging to the candidate single-click type are subjected to feature analysis based on the operation interval to obtain operation data units belonging to the single-click type and operation data units belonging to the double-click type.

8. The method according to claim 7, It is characterized in that The performing feature analysis based on the operation time on the operation data unit belonging to the click type to obtain the operation data unit belonging to the candidate single-click type and the operation data unit belonging to the long-press type includes: Acquire multiple timestamps corresponding to the coordinate data in the operation data unit belonging to the click type; The difference between the minimum timestamp and the maximum timestamp is determined as the operation time difference; In response to the operation time difference being greater than or equal to a time difference threshold, marking the click type operation data unit as a long press type operation data unit; In response to the operation time difference being less than a time difference threshold, the operation data unit belonging to the click type is marked as an operation data unit belonging to a candidate single-click type.

9. The method according to claim 7, It is characterized in that The step of performing feature analysis based on operation intervals on the operation data units belonging to the candidate single-click type to obtain operation data units belonging to the single-click type and operation data units belonging to the double-click type includes: The following processing is performed on the first operation data unit among the operation data units belonging to the candidate single-click type: When the operation type of the next operation data unit of the first operation data unit is a candidate single-click operation type, obtaining the first representative coordinates and the first click time of the first operation data unit, and obtaining the second representative coordinates and the second click time of the next operation data unit; Determine a coordinate difference between the first representative coordinate and the second representative coordinate, and determine a time interval between the first click time and the second click time; When the coordinate difference is greater than a coordinate difference threshold, or when the coordinate difference is less than or equal to the coordinate difference threshold and the time interval is greater than the time interval threshold, marking the first operation data unit as an operation data unit belonging to a single-click type; When the coordinate transformation distance is less than or equal to the coordinate difference threshold, and the time interval is less than or equal to the time interval threshold, the first operation data unit and the next operation data unit are merged into an operation data unit belonging to the double-click type.

10. The method according to claim 9, It is characterized in that The step of merging the first operation data unit and the next operation data unit into an operation data unit belonging to a double-click type includes: Acquire first operation metadata of the first operation data unit and second operation metadata of the next operation data unit; Add a double-click operation identifier to the first operation metadata and the second operation metadata, The double-click operation identifier is used to identify that the combination of the first operation data unit and the next operation data unit belongs to a double-click type operation data unit.

11. The method according to claim 9, It is characterized in that The obtaining of the first representative coordinates and the first click time of the first operation data unit includes: Acquire at least one coordinate pair in the first operation data unit, wherein the coordinate pair includes a horizontal coordinate and a vertical coordinate; determining one of the coordinate pairs in the first operation data unit as the first representative coordinates; The timestamp corresponding to the first representative coordinates is determined as the first click time.

12. A data processing device, It is characterized in that The device comprises: An acquisition module, used to acquire device information of a target device and original operation data stream information generated by the target device; A calling module, used for calling a target identification interface corresponding to the target device based on the device information; A cutting module, used for performing data cutting processing on the original operation data stream information based on the target identification interface to obtain multiple operation data units; The identification module is used to perform operation type identification processing on the multiple operation data units based on the target identification interface to obtain the operation type of each of the operation data units.

13. An electronic device, It is characterized in that The electronic device comprises: Memory for storing computer programs or computer executable instructions; A processor, configured to implement the data processing method according to any one of claims 1 to 11 when executing the computer program or computer executable instructions stored in the memory.

14. A computer-readable storage medium, It is characterized in that A computer program or a computer executable instruction is stored, and when the computer program or the computer executable instruction is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

15. A computer program product comprising a computer program or computer executable instructions, It is characterized in that When the computer program or computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images