Mobile application intelligent detection method and device based on image content recognition technology

Abstract

The application discloses a kind of mobile application intelligent detection method and device based on image content recognition technology, comprising: obtaining the interface image of the self-defined control of mobile application to be detected;Image classification model and image recognition model are constructed and trained, and the trained image classification model and the trained image recognition model are obtained;Interface image is input to the trained image classification model for classification, and the corresponding self-defined control type is obtained, and interface image is distributed to different self-defined control image sets according to different self-defined control types;Interface image in each self-defined control image set is input to the trained image recognition model, and control information is obtained by identification;Using automated test tool, native control in the mobile application to be detected is identified, and instruction coordinate set is obtained, and native control and self-defined control are automatically clicked according to instruction coordinate set and control information respectively, and detection data is obtained, which can improve success rate.

Description

Technical Field

[0001] This invention relates to the field of application testing, and more specifically to a method and apparatus for intelligent detection of mobile applications based on image content recognition technology. Background Technology

[0002] Currently, there are many detection technologies for mobile applications (APPs). Common detection technologies mainly include code feature-based detection and dynamic behavior feature-based detection. Code feature-based detection mainly uses decompilation tools to extract static features from the program, such as syntax, semantics, and signature information. However, this detection technology has limitations, especially against malicious APPs that employ stealth techniques, such as encryption, code transformation, and hardening, which cannot be decompiled and their malicious behavior cannot be extracted. Research on dynamic behavior feature-based detection mainly focuses on using simulators or real device sandboxes to monitor the real-time dynamic behavior data of the APP during operation, including its calls to sensitive APIs and network data requests.

[0003] Current research on mobile application detection technology mainly focuses on how to locate malicious code detection models, how to build malicious application detection models, and the classification of malicious families, while there is relatively little research on automated detection of mobile applications. Summary of the Invention

[0004] The purpose of this application is to propose a method and apparatus for intelligent detection of mobile applications based on image content recognition technology to address the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a mobile application intelligent detection method based on image content recognition technology, comprising the following steps:

[0006] Obtain the UI image of the custom controls of the mobile application to be tested during runtime;

[0007] Construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network.

[0008] The interface image of the custom control is input into a trained image classification model for classification to obtain the corresponding custom control type. The interface image is then assigned to a different custom control image set according to the different custom control types. The interface image in each custom control image set is input into a trained image recognition model to identify the control information corresponding to the interface image in each custom control image set. The control information includes text elements and position information.

[0009] An automated testing tool is used to identify native controls in the mobile application to be tested, and obtain a set of command coordinates. Based on the command coordinate set and the control information corresponding to the interface image in each set of custom control images, automated click operations are performed on native controls and custom controls respectively to obtain test data.

[0010] Preferably, different custom control types include swipe, skip, login, registration, customer service, and payment channel; different custom control image sets include swipe set, skip set, login set, registration set, customer service set, and payment channel set; and text elements include the names of the operation bits of the custom control.

[0011] As a preferred choice, the convolutional neural network uses the ResNet50 network.

[0012] As a preferred option, the image recognition model employs OCR recognition technology.

[0013] As a preferred option, the automated testing tool includes UiAutomator2. UiAutomator2 identifies the names of the operation bits of the native control, and the element positioning tool Weditor is used to obtain the position information of the operation bits of the native control. The names of the operation bits of the native control and their position information constitute the instruction coordinate set.

[0014] As a preferred method, automated click operations are implemented using automated click scripts. These scripts utilize instruction coordinate sets and control information to perform automated clicks and collect test data generated by the mobile application to be tested after the automated click operations are completed.

[0015] Secondly, the present invention provides a mobile application intelligent detection device based on image content recognition technology, comprising:

[0016] The data acquisition module is configured to acquire the interface images of the custom controls of the mobile application to be tested during runtime;

[0017] The model building module is configured to build and train an image classification model and an image recognition model, resulting in a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network.

[0018] The classification and recognition module is configured to input the interface image of the custom control into a trained image classification model to classify the operation components and obtain the corresponding custom control type. Based on the different custom control types, the interface image is assigned to different custom control image sets. The interface image in each custom control image set is input into a trained image recognition model to identify the control information corresponding to the interface image in each custom control image set. The control information includes text elements and position information.

[0019] The click module is configured to use automated testing tools to identify native controls in the mobile application to be tested, obtain a set of instruction coordinates, and perform automated click operations on native controls and custom controls respectively based on the instruction coordinate set and the control information corresponding to the interface image in each set of custom control images to obtain detection data.

[0020] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0021] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0022] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) The mobile application intelligent detection method based on image content recognition technology proposed in this invention, based on conventional automated click technology, uses image classification technology and OCR recognition technology to realize automated clicks on custom controls such as web interfaces and images in the APP, which improves the automated detection capability of the APP to a certain extent and alleviates the heavy manual detection work.

[0025] (2) The mobile application intelligent detection method based on image content recognition technology proposed in this invention can automatically extract the control information of illegal APPs and automatically label the corresponding operation behaviors to form a key behavior feature database, which provides convenience for relevant departments to accurately warn, crack down and analyze.

[0026] (3) The mobile application intelligent detection method based on image content recognition technology proposed in this invention has high compatibility and can detect apps on both Android and iOS systems; the detection environment is compatible with real devices and emulators. However, real devices have higher compatibility than emulators. The real device testing environment can provide a real running environment for apps, including a real environment with real CPU, memory, making phone calls, sending and receiving text messages, etc. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a mobile application intelligent detection method based on image content recognition technology, as an embodiment of this application.

[0029] Figure 2 This is a framework diagram of a mobile application intelligent detection method based on image content recognition technology, as an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of the UiAutomator2 automated testing principle of the mobile application intelligent detection method based on image content recognition technology, which is an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of a mobile application intelligent detection device based on image content recognition technology, as an embodiment of this application.

[0032] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] Figure 1 An embodiment of this application illustrates a mobile application intelligent detection method based on image content recognition technology, comprising the following steps:

[0035] S1, obtain the interface image of the custom control of the mobile application to be tested during operation.

[0036] For details, please refer to Figure 2The mobile application intelligent detection method based on image content recognition technology proposed in the embodiments of this application is based on detection on real devices or virtual machines. The mobile application intelligent detection platform built with the help of image classification technology and OCR recognition technology covers Android and iOS, and can realize the ability to automatically detect the dynamic and static behavior of APP. It can simultaneously record dynamic and static behavior data to form a feature database, providing data support for subsequent rapid strikes, analysis and early warning and other work.

[0037] The detection environment of this application's embodiments is compatible with both real devices and emulators, and can deploy multiple customized real devices or emulators in parallel using multi-threaded deployment. Parallel scheduling can be implemented according to detection needs to improve detection efficiency. Real devices include Android and iOS devices. The detection principle for Android devices mainly employs ROM customization technology, deeply customizing the operating system by modifying source code, compiling, and flashing. API interface monitoring technology is used to monitor key API calls made by the app during operation, such as reading user privacy information like IMEI, SMS messages, and contacts; sensitive behaviors of the app such as recording, taking photos, and reading / writing SD card; and background network data access, enabling real-time monitoring of the app's dynamic behavioral characteristics. The detection principle for iOS devices is mainly based on a jailbroken operating system, using plugins to track the network request data and behavioral data of a specified app. Network request data is mainly obtained through interfaces provided by utility classes, such as obtaining ad space identifiers, UUIDs, MAC addresses, IP addresses, and network addresses. Behavioral data is mainly monitored by tracking API calls to monitor the app's access to sensitive information such as contacts and SMS messages during operation.

[0038] The detection principle of emulators is similar to that of real devices, but in practical applications, real device detection has advantages over Android emulators or systems using Hook technology in the following two main aspects:

[0039] Advantage 1: It has better compatibility with apps. Real device detection can provide apps with a real running environment, while Android emulators, although they provide apps with an Android system running environment, cannot truly reproduce the real environment of the phone, such as receiving and sending text messages, making and receiving calls. Therefore, some anti-emulator or anti-Hook technology applications can only run on a real device operating system.

[0040] Advantage 2: High deployment cost-effectiveness. If the detection system deploys multiple parallel simulators, in addition to APP compatibility issues, it places higher demands on the server's CPU, memory, hard drive, etc., and the simulator's running speed is also relatively slow.

[0041] Therefore, the real device detection module has certain advantages in terms of compatibility, running speed, and cost, and can be selectively deployed according to different needs in practical applications.

[0042] Based on the above detection environment deployment, the interface image of the custom control of the mobile application during operation can also be obtained. This interface image can be obtained by taking a screenshot or by other methods, without any restrictions.

[0043] S2, construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network.

[0044] In a specific embodiment, the convolutional neural network uses the ResNet50 network.

[0045] In a specific embodiment, the image recognition model employs OCR recognition technology.

[0046] Specifically, the embodiments of this application introduce image classification technology and OCR recognition technology. Image classification technology mainly uses image classification models to classify interface images, identifying and classifying key components and icons such as input boxes, buttons, and sliders on the interface images to form different sets of custom control images, such as login sets, registration sets, customer service sets, payment channel sets, and slider sets. Different sets of custom control images can be defined as A. n For example, the login set is A1, the registration set is A2, the customer service set is A3, and so on. In the embodiments of this application, the image classification model uses a ResNet50 network. In other embodiments, other neural network structures can also be used for image classification models.

[0047] Regarding the aforementioned A n The interface images in the collection are recognized using OCR technology, which comprises multiple techniques including text detection, text correction, text recognition, and text layout analysis. In one embodiment, the text detection algorithm employs a lightweight, single-stage object detection model to extract text regions from the input interface image and provide the location and pixel information of these regions. The algorithm process mainly includes: building a lightweight fully convolutional network, collecting and processing samples, generating training and validation sets, training the fully convolutional network to convergence using the training and validation sets, and verifying the network's performance. The total loss function used by the text detection algorithm during training is:

[0048] L=αL det +βL box ;

[0049] Where L represents the total loss function, L det The loss function for determining text regions is the cross-entropy loss function. L box The text box regression loss function is denoted by MSE (mean squared error) loss function, where α (0 < α < 1) and β (0 < β < 1) are the weights of the text region judgment loss function and the text box regression loss function, respectively.

[0050] The embodiments of this application utilize image classification and OCR recognition technologies to train on samples from nearly 50,000 apps, resulting in trained image classification and recognition models. The app samples mentioned herein must cover the key research sets of the embodiments of this application, such as swiping sets, skip sets, login sets, registration sets, and customer service sets.

[0051] S3. Input the interface image of the custom control into the trained image classification model for classification to obtain the corresponding custom control type. Based on the different custom control types, assign the interface image to different custom control image sets. Input the interface image in each custom control image set into the trained image recognition model to identify the control information corresponding to the interface image in each custom control image set. The control information includes text elements and position information.

[0052] In specific embodiments, different custom control types include swipe, skip, login, registration, customer service, and payment channel; different custom control image sets include swipe set, skip set, login set, registration set, customer service set, and payment channel set; and text elements include the names of the operation bits of the custom control.

[0053] Specifically, embodiments of this application use a trained image classification model to identify and classify key components and icons on interface images, forming different custom control image sets A. n ; Using OCR recognition technology to identify A n The system locates and recognizes text in the interface images within the collection, detecting and locating key app components such as input boxes and buttons to obtain relevant control information. This control information includes text elements and their corresponding position information. The text elements include the names of the operation positions of custom controls, such as the text for operation positions of custom controls for app types like login, registration, and customer service. Correspondingly, the position information is the location information of the operation positions of the custom controls. In one example, the login type will identify the names and positions of operation positions such as username, password, and login.

[0054] The interface image of the custom control is detected and recognized by the trained image classification model and the trained image recognition model, and relevant control information, such as the name, coordinates and type of the control's operation position, is output to further guide the implementation of the automated click technology in step S4.

[0055] S4. An automated testing tool is used to identify native controls in the mobile application to be tested, and obtain a set of command coordinates. Based on the command coordinate set and the control information corresponding to the interface image in each set of custom control images, automated click operations are performed on native controls and custom controls respectively to obtain test data.

[0056] In a specific embodiment, the automated testing tool includes UiAutomator2. UiAutomator2 identifies the name of the operation bit of the native control, and the element positioning tool Weditor is used to obtain the position information of the operation bit of the native control. The name of the operation bit of the native control and its position information constitute the instruction coordinate set.

[0057] In a specific embodiment, the automated click operation is implemented using an automated click script. The automated click script uses a set of instruction coordinates and control information to perform automated clicks and collects test data generated by the mobile application to be tested after the automated click operation is completed.

[0058] Specifically, the embodiments of this application are compatible with the Android APP automated testing tool UiAutomator2. UiAutomator2 supports test scripts written in Python, allowing scripts to run on a computer to control automated testing of the mobile phone. The basic process of using the UiAutomator2 automated testing tool involves connecting the mobile phone to the automated intelligent detection system via Wi-Fi or USB, using the element location tool Weditor to acquire the coordinates of controls such as elements on the mobile APP interface to form a set of instructions, and then writing UI automation scripts to automatically identify controls on the mobile APP. Its schematic diagram is shown below. Figure 3 As shown, the PC side mainly installs the automation script and sends HTTP requests to the Android device. The Android device side mainly runs an HTTP service that encapsulates UiAutomator2, parses the received requests, and converts them into UiAutomator2 code.

[0059] The automated testing tool is primarily designed for recognizing native controls in an app to automate clicks. For automated recognition of custom controls in web interfaces and images, it relies on the interaction of the trained image classification and recognition models from step S3. This primarily involves classifying and recognizing interface images and outputting the parsed results as a JSON file, generating control information for custom controls such as swipe, login, and registration. Automated click operations are then performed based on this control information, thus collecting detection data. The trained image classification and recognition models are compatible with both Android and iOS app interface image classification and recognition. The automated click script is designed to execute automated click operations using the names and positions of the operation bits of native controls in the instruction coordinate set, as well as the names and positions of the operation bits of custom controls in their control information.

[0060] To verify the impact of the intelligent mobile application detection method based on image content recognition technology proposed in the embodiments of this application on the automated detection effect of mobile applications, a comparative experiment is conducted between detection system A, which integrates the intelligent mobile application detection method based on image content recognition technology proposed in the embodiments of this application, and detection system B, which does not integrate the intelligent mobile application detection method based on image content recognition technology proposed in the embodiments of this application. The comparative experiment compares the automatic click success rate of 700 APP samples, of which 260 APPs contain swipe or skip controls, 604 APPs contain login controls, and 604 APPs contain registration controls. The test results are shown in Table 1.

[0061] Table 1

[0062]

[0063] As can be seen from Table 1, the automatic click success rate of detection system A is about 63% higher than that of detection system B. The success rate of clicks on image-based sliding and skip controls is particularly improved. Therefore, image classification technology and OCR recognition technology can improve the success rate of automatic clicks in APP to a certain extent.

[0064] Further reference Figure 4 As an implementation of the methods shown in the above figures, this application provides an embodiment of a mobile application intelligent detection device based on image content recognition technology. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0065] This application provides a mobile application intelligent detection device based on image content recognition technology, including:

[0066] Data acquisition module 1 is configured to acquire the interface images of the custom controls of the mobile application to be tested during operation;

[0067] Model building module 2 is configured to build and train an image classification model and an image recognition model, resulting in a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network.

[0068] The classification and recognition module 3 is configured to input the interface image of the custom control into a trained image classification model for classification to obtain the corresponding custom control type, and to assign the interface image to different custom control image sets according to different custom control types; input the interface image in each custom control image set into a trained image recognition model to identify the control information corresponding to the interface image in each custom control image set, including text elements and position information;

[0069] Click module 4 is configured to use automated testing tools to identify native controls in the mobile application to be tested, obtain instruction coordinate sets, and perform automated click operations on native controls and custom controls respectively based on the instruction coordinate sets and the control information corresponding to the interface images in each custom control image set to obtain detection data.

[0070] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device of this embodiment includes a processor 501 and a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0071] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0072] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0073] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 501, implement the above method.

[0074] This invention also provides a computer program product, including a computer program that, when executed by a processor 501, implements the above-described method.

[0075] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0076] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0077] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0078] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 501 to execute some steps of the methods of the various embodiments of this application.

[0079] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor 501. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 501 executing the steps, or as a combination of hardware and software modules within the processor 501 executing the steps.

[0080] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0081] Bus 503 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0082] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0083] An exemplary storage medium is coupled to processor 501, enabling processor 501 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 501. Processor 501 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 501 and storage medium can exist as discrete components in an electronic device or host device.

[0084] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile application intelligent detection method based on image content recognition technology, characterized in that, Includes the following steps: Obtain the UI image of the custom controls of the mobile application to be tested during runtime; An image classification model and an image recognition model are constructed and trained to obtain a trained image classification model and a trained image recognition model, wherein the image classification model includes a convolutional neural network; The interface image of the custom control is input into a trained image classification model for classification to obtain the corresponding custom control type. The interface image is then assigned to different custom control image sets according to the different custom control types. The interface image in each custom control image set is input into the trained image recognition model to identify the control information corresponding to the interface image in each custom control image set. The control information includes text elements and position information. An automated testing tool, including UiAutomator2, is used to identify the native controls in the mobile application under test, obtaining a set of instruction coordinates. UiAutomator2 identifies the names of the operation positions of the native controls, and the Webitor element positioning tool obtains the position information of the operation positions. The names and position information of the operation positions of the native controls constitute the instruction coordinate set. Based on the instruction coordinate set and the control information corresponding to the interface images in each set of custom control images, automated click operations are performed on the native controls and custom controls respectively to obtain test data. The automated click operation is implemented using an automated click script. The automated click script uses the instruction coordinate set or the control information to perform automated clicks and collects the test data generated by the mobile application to be tested after the automated click operation is completed.

2. The intelligent detection method for mobile applications based on image content recognition technology according to claim 1, characterized in that, The different custom control types include swipe, skip, login, registration, customer service, and payment channel; the different custom control image sets include swipe set, skip set, login set, registration set, customer service set, and payment channel set; and the text element includes the name of the operation position of the custom control.

3. The intelligent detection method for mobile applications based on image content recognition technology according to claim 1, characterized in that, The convolutional neural network used is the ResNet50 network.

4. The intelligent detection method for mobile applications based on image content recognition technology according to claim 1, characterized in that, The image recognition model uses OCR recognition technology.

5. A mobile application intelligent detection device based on image content recognition technology, characterized in that, include: The data acquisition module is configured to acquire the interface images of the custom controls of the mobile application to be tested during runtime; The model building module is configured to build and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model, wherein the image classification model includes a convolutional neural network; The classification and recognition module is configured to input the interface image of the custom control into a trained image classification model for classification to obtain the corresponding custom control type, and to assign the interface image to different custom control image sets according to different custom control types; input the interface image in each custom control image set into the trained image recognition model to identify the control information corresponding to the interface image in each custom control image set, wherein the control information includes text elements and position information; The click module is configured to use an automated testing tool to identify native controls in the mobile application under test, and obtain a set of instruction coordinates. The automated testing tool includes UiAutomator2, which identifies the names of the operation positions of the native controls and obtains the position information of the operation positions of the native controls using the element positioning tool Weditor. The names and position information of the operation positions of the native controls constitute the set of instruction coordinates. Based on the set of instruction coordinates and the control information corresponding to the interface images in each set of custom control images, automated click operations are performed on the native controls and custom controls respectively to obtain test data. The automated click operation is implemented using an automated click script. The automated click script uses the instruction coordinate set or the control information to perform automated clicks and collects the test data generated by the mobile application to be tested after the automated click operation is completed.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images