Mobile application intelligent detection method and device based on image content recognition technology

Abstract

The invention discloses a mobile application intelligent detection method and device based on an image content recognition technology. The method comprises the steps of obtaining an interface image of a user-defined control of a to-be-detected mobile application; constructing and training an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model; inputting the interface images into a trained image classification model for classification to obtain corresponding user-defined control types, and allocating the interface images to different user-defined control image sets according to different user-defined control types; inputting an interface image in each user-defined control image set into a trained image recognition model, and recognizing to obtain control information; according to the method, the native control in the to-be-detected mobile application is identified by adopting the automatic test tool to obtain the instruction coordinate set, and the native control and the self-defined control are automatically clicked according to the instruction coordinate set and the control information to obtain the detection data, so that the success rate can be improved.

Description

Technical Field

[0001] The present invention relates to the field of application testing, and particularly to an intelligent detection method and device for mobile applications based on image content recognition technology. Background Art

[0002] Currently, there are many detection technologies for mobile applications (APPs). The common detection technologies mainly include detection technologies based on code features and detection technologies based on dynamic behavior features. The detection technology based on code features mainly extracts static features in the program through decompilation tools, such as syntax semantics, signature and other feature information. However, this detection technology has limitations. Especially for malicious APPs using stealth technologies, such as encryption, code conversion, reinforcement and other technologies, it is impossible to decompile them and extract malicious behaviors. The research on detection technologies based on dynamic behavior features mainly focuses on real-time monitoring of the real-time dynamic behavior data of the calls of sensitive APIs in the system and network data requests during the running process of the APP based on simulators or real device sandboxes.

[0003] Currently, the research on mobile application detection technologies mainly focuses on how to locate the detection model of malicious code, how to establish the detection model of malicious applications, malicious family classification, etc., and there is less research on the automated detection of mobile applications. Summary of the Invention

[0004] The purpose of this application is to propose an intelligent detection method and device for mobile applications based on image content recognition technology for the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides an intelligent detection method for mobile applications based on image content recognition technology, including the following steps:

[0006] Obtain the interface image of the custom control of the mobile application to be detected during the running process;

[0007] Construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network;

[0008] Input the interface image of the custom control into the trained image classification model for classification to obtain the corresponding custom control type, and allocate the interface image to different custom control image sets according to different custom control types; input the interface images in each custom control image set into the trained image recognition model to identify the control information corresponding to the interface images in each custom control image set. The control information includes text elements and position information;

[0009] An automated testing tool is used to identify native controls in the mobile application to be tested, obtaining an instruction coordinate set. Based on the instruction coordinate set and the control information corresponding to the interface images in each custom control image set, automated click operations are performed on the native controls and custom controls respectively to obtain detection data.

[0010] Preferably, different custom control types include sliding, skipping, logging in, registering, customer service, and payment channels. Different custom control image sets include a sliding set, a skipping set, a logging-in set, a registering set, a customer service set, and a payment channel set. The text elements include the names of the operation positions of the custom controls.

[0011] Preferably, the convolutional neural network adopts the Resnet50 network.

[0012] Preferably, the image recognition model adopts OCR recognition technology.

[0013] Preferably, the automated testing tool includes UiAutomator2. The names of the operation positions of the native controls are identified through UiAutomator2, and the position information of the operation positions of the native controls is obtained with the help of the element positioning tool Weditor. The names and position information of the operation positions of the native controls constitute the instruction coordinate set.

[0014] Preferably, the automated click operation is implemented using an automated click script. The automated click script uses the instruction coordinate set and the control information to perform automated clicks and collects the test data generated by the mobile application to be tested after the automated click operation is completed.

[0015] In a second aspect, the present invention provides a mobile application intelligent detection device based on image content recognition technology, including:

[0016] A data acquisition module configured to acquire interface images of custom controls in the mobile application to be tested during operation;

[0017] A model construction module configured to construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network;

[0018] A classification and recognition module configured to input the interface images of the custom controls into the trained image classification model for classification of operation components to obtain the corresponding custom control types, and allocate the interface images to different custom control image sets according to different custom control types; input the interface images in each custom control image set into the trained image recognition model to identify the control information corresponding to the interface images in each custom control image set. The control information includes text elements and position information;

[0019] The click module is configured to use an automated testing tool to identify native controls in a mobile application to be detected, obtain an instruction coordinate set, and perform automated click operations on the native controls and custom controls respectively according to the instruction coordinate set and the control information corresponding to the interface images in each set of custom control images, so as to obtain detection data.

[0020] In a third aspect, the present invention provides an electronic device, including one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method described in any implementation manner of the first aspect.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0022] In a fifth aspect, the present invention provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) Based on the conventional automated click technology, the intelligent detection method for mobile applications based on image content recognition technology proposed by the present invention realizes the automated click of custom controls such as Web-based interfaces and pictures in an APP by means of image classification technology and OCR recognition technology, which improves the automated detection ability of the APP to a certain extent and alleviates the heavy manual detection work.

[0025] (2) The intelligent detection method for mobile applications based on image content recognition technology proposed by the present invention can automatically extract the control information of illegal APPs, automatically label the corresponding operation behaviors, and form a key behavior feature database, which provides convenience for the accurate early warning, crackdown and series analysis of relevant departments.

[0026] (3) The intelligent detection method for mobile applications based on image content recognition technology proposed by the present invention has high compatibility and can be compatible with the detection of APPs on Android and iOS systems; the detection environment is compatible with real machines and emulators. However, the compatibility of real machines is higher than that of emulators. The real machine test environment can provide a real running environment for the APP, including providing a real CPU, memory, making calls, sending and receiving text messages, etc. Description of the Drawings

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0028] Figure 1 Schematic flowchart of the mobile application intelligent detection method based on image content recognition technology for the embodiments of the present application;

[0029] Figure 2 Framework diagram of the mobile application intelligent detection method based on image content recognition technology for the embodiments of the present application;

[0030] Figure 3 Schematic diagram of the UiAutomator2 automated test principle for the mobile application intelligent detection method based on image content recognition technology for the embodiments of the present application;

[0031] Figure 4 Schematic diagram of the mobile application intelligent detection device based on image content recognition technology for the embodiments of the present application;

[0032] Figure 5 Schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present invention. Specific embodiments

[0033] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the present invention in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0034] Figure 1 Shows a mobile application intelligent detection method provided by the embodiments of the present application, including the following steps:

[0035] S1, obtain the interface image of the custom control of the mobile application to be detected during operation.

[0036] Specifically, refer to Figure 2, the mobile application intelligent detection method based on image content recognition technology proposed in the embodiments of this application is a detection based on real machines or virtual machines. The mobile application intelligent detection platform constructed by means of image classification technology and OCR recognition technology covers Android and iOS, and can realize the ability to automatically detect the dynamic and static behaviors of APPs, and can synchronously record the dynamic and static behavior data to form a feature database, providing data support for later work such as rapid crackdown, research and judgment warning, etc.

[0037] The detection environment of the embodiments of this application is compatible with both real machines and emulators. Multiple customized real machines or emulators can be deployed in parallel with multiple threads, and can be scheduled in parallel according to detection requirements to improve detection efficiency. Among them, the real machines include Android real machines and iOS real machines. The detection principle of Android real machines mainly uses the customized technology of ROM to deeply customize the source code of the real machine's operating system, compile and flash the machine, etc., and uses the API interface monitoring technology to monitor the key APIs called by the APP during operation, such as reading user privacy information such as IMEI, text messages, and address books on the mobile phone, sensitive behaviors such as the APP operating the mobile phone's recording, taking pictures, reading and writing the SD card, and key behaviors such as network data accessed in the background, so as to realize the real-time monitoring of the dynamic behavior characteristics of the APP. The detection principle of iOS real machines is mainly based on the jailbroken operating system, and the network request data and behavior data of the specified APP are tracked through plug-ins. The network request data is mainly obtained through the interface for obtaining network information provided by the tool class, such as obtaining data such as advertising identifier, UUID, MAC address, IP address, network address, etc.; the behavior data is mainly to monitor the behavior of the APP accessing sensitive information such as the address book and text messages on the mobile phone by monitoring the API interface call method during operation.

[0038] The detection principle of the emulator is similar to that of the real machine. However, in actual applications, the real machine detection has the following two advantages compared with Android emulators or systems using Hook technology:

[0039] Advantage 1: It has better compatibility with the APP. Real machine detection can provide a real running environment for the APP. Although the Android emulator provides an Android system running environment for the APP, it cannot truly restore the real environment of the mobile phone, such as receiving and sending text messages, making and answering calls, etc. Therefore, for some applications against emulator or anti-Hook technology, they can only run on the real machine operating system.

[0040] Advantage 2: High deployment cost performance. If multiple parallel emulators are deployed in the detection system, in addition to the APP compatibility problem, it puts higher requirements on the CPU, memory, hard disk, etc. of the server, and the running speed of the emulator is also relatively slow.

[0041] Therefore, the real machine detection module has certain advantages in terms of compatibility, running speed, and cost, and is selectively deployed according to different requirements in actual applications.

[0042] Based on the deployment of the above detection environment, the interface images of the custom controls of the mobile application during operation can be obtained simultaneously. The interface images can be obtained by taking screenshots or other methods, and there is no limitation here.

[0043] S2. Construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model. The image classification model includes a convolutional neural network.

[0044] In a specific embodiment, the convolutional neural network adopts the Resnet50 network.

[0045] In a specific embodiment, the image recognition model adopts the OCR recognition technology.

[0046] Specifically, the embodiments of the present application introduce image classification technology and OCR recognition technology. The image classification technology mainly uses an image classification model to classify the interface images, identify and classify key components and key icons such as input boxes, buttons, and slides on the interface images, and form different custom control image sets, such as key interface image sets like a login set, a registration set, a customer service set, a payment channel set, a slide set, etc. Different custom control image sets can be defined as A n , such as the login set being A 1 , the registration set being A 2 , the customer service set being A 3 , and so on. In the embodiments of the present application, the image classification model adopts the Resnet50 network. In other embodiments, an image classification model with other neural network structures can also be used.

[0047] For the interface images in the above-mentioned A n set, the OCR recognition technology is used for recognition. The OCR recognition technology consists of multiple technologies such as a text detection algorithm, a text correction algorithm, a text recognition algorithm, and a text layout analysis. In one embodiment, the text detection algorithm uses a lightweight single-stage object detection model for object detection, extracts the text regions in the input interface images, and provides the position information and pixel information of the text regions. The algorithm process mainly includes: building a lightweight fully convolutional network, collecting samples, processing samples, generating a training set and a validation set, and training the fully convolutional network with the training set and the validation set until convergence and verifying the performance of the network, etc. The total loss function used by the text detection algorithm during training is:

[0048] L = αL det + βLbox ;

[0049] Among them, L represents the total loss function, and L det represents the text region judgment loss function, and the cross-entropy loss function is adopted; L box is the text box regression loss function, and the MSE mean square error loss function is adopted. α(0 < α < 1) and β(0 < β < 1) are the weights of the text region judgment loss function and the text box regression loss function respectively.

[0050] Embodiments of the present application rely on image classification technology and OCR recognition technology to train samples of nearly 50,000 APPs, obtaining a trained image classification model and a trained image recognition model. The APP samples mentioned here need to cover the key research sets of the embodiments of the present application, such as the sliding set, skip set, login set, registration set, customer service set, etc.

[0051] S3. Input the interface image of the custom control into the trained image classification model for classification to obtain the corresponding custom control type, and allocate the interface image to different custom control image sets according to different custom control types; input the interface images in each custom control image set into the trained image recognition model to identify the control information corresponding to the interface images in each custom control image set. The control information includes text elements and position information.

[0052] In specific embodiments, different custom control types include sliding, skipping, logging in, registering, customer service, and payment channels. Different custom control image sets include a sliding set, a skip set, a login set, a registration set, a customer service set, and a payment channel set. The text elements include the names of the operation positions of the custom control.

[0053] Specifically, embodiments of the present application identify and classify key components and key icons on the interface image through the trained image classification model to form different custom control image sets A n ; Rely on OCR recognition technology to locate and recognize the text of the interface images in set A n Detect and locate key APP components such as input boxes and buttons, and obtain relevant control information on the interface image. The control information includes text elements and their corresponding position information. The text elements include the names of the operation positions of the custom control, such as the text of the relevant operation positions of custom controls of types such as login, registration, and customer service of the App. Correspondingly, the position information is the position information of the operation position of the custom control. In one example, in the login type, the names and position information of operation positions such as username, password, and login will be recognized.

[0054] Detect and recognize the interface image of the custom control through a trained image classification model and a trained image recognition model, and output relevant control information, such as the name, coordinates, type, etc. of the operation position of the control, to further guide the implementation of the automated click technology in step S4.

[0055] S4. Use an automated testing tool to recognize the native controls in the mobile application to be tested, obtain an instruction coordinate set, and perform automated click operations on the native controls and custom controls respectively according to the instruction coordinate set and the control information corresponding to the interface images in each custom control image set to obtain detection data.

[0056] In a specific embodiment, the automated testing tool includes UiAutomator2. The name of the operation position of the native control is recognized through UiAutomator2, and the position information of the operation position of the native control is obtained with the help of the element positioning tool Weditor. The name of the operation position of the native control and its position information form an instruction coordinate set.

[0057] In a specific embodiment, the automated click operation is implemented by an automated click script. The automated click script uses the instruction coordinate set and control information to perform automated clicks and collect the test data generated by the mobile application to be tested after the automated click operation is completed.

[0058] Specifically, the embodiment of the present application is compatible with the automated testing tool UiAutomator2 for Android APP. The UiAutomator2 version supports writing test scripts in Python scripts and supports running the scripts on a computer to control the automated testing of the mobile phone. The basic process of using the UiAutomator2 automated testing tool is to connect the mobile phone to the automated intelligent detection system through Wifi or USB, complete the acquisition of controls such as the interface elements of the mobile phone APP through the element positioning tool Weditor to form an instruction coordinate set, and then achieve the purpose of automatically recognizing the controls on the mobile phone APP by writing a UI automation script. Its schematic diagram is as Figure 3 shown. The PC side mainly installs an automated operation script and sends an HTTP request to the Android device. The Android device side mainly runs an HTTP service encapsulating UiAutomator2, parses the received request, and converts it into UiAutomator2 code.

[0059] The automated testing tool is mainly applicable to the recognition of native controls of APPs to complete automated clicks. For the automated recognition of custom controls such as Web-based interfaces and images, it is necessary to interact with the trained image classification model and the trained image recognition model in step S3. It mainly realizes the classification and recognition of interface images, and outputs the parsing results in the form of a Json file to form control information of custom controls such as swiping, logging in, and registering. Automated click operations are performed according to the control information, and the collection of detection data is realized. The trained image classification model and the trained image recognition model are compatible with the classification and recognition of Android APP and iOS APP interface images. The automated click script is designed to perform automated click operations using the name and position information of the operation bits of native controls in the instruction coordinate set and the name and position information of the operation bits of custom controls in the control information of custom controls.

[0060] To verify the influence of the mobile application intelligent detection method based on image content recognition technology proposed in the embodiments of the present application on the automated detection effect of mobile applications, a comparative experiment is now carried out between detection system A integrating the mobile application intelligent detection method based on image content recognition technology proposed in the embodiments of the present application and detection system B not integrating the mobile application intelligent detection method based on image content recognition technology proposed in the embodiments of the present application. This comparative experiment conducts a comparative test on the automatic click success rate of 700 APP samples, among which 260 APPs contain swiping or skipping controls, 604 APPs contain login controls, and 604 APPs contain registration controls. The test results are shown in Table 1.

[0061] Table 1

[0062]

[0063] As can be seen from Table 1, the automatic click success rate of detection system A has increased by about 63% compared with that of detection system B. Especially for the click success rate of swiping and skipping controls based on image functions, it has increased significantly. Therefore, the use of image classification technology and OCR recognition technology can improve the success rate of APP automated clicks to a certain extent.

[0064] Further referring to Figure 4 , as an implementation of the methods shown in the above figures, an embodiment of a mobile application intelligent detection device based on image content recognition technology is provided in the present application. This device embodiment corresponds to the method embodiment shown in Figure 1 , and this device can be specifically applied to various electronic devices.

[0065] An embodiment of the present application provides a mobile application intelligent detection device based on image content recognition technology, including:

[0066] A data acquisition module 1, configured to acquire an interface image of a custom control of a mobile application to be detected during operation;

[0067] A model construction module 2, configured to construct and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model, where the image classification model includes a convolutional neural network;

[0068] A classification and recognition module 3, configured to input the interface image of the custom control into the trained image classification model for classification to obtain the corresponding custom control type, and allocate the interface images to different custom control image sets according to different custom control types; input the interface images in each custom control image set into the trained image recognition model to recognize the control information corresponding to the interface images in each custom control image set, where the control information includes text elements and position information;

[0069] A click module 4, configured to use an automated testing tool to recognize native controls in the mobile application to be detected to obtain an instruction coordinate set, and perform automated click operations on the native controls and custom controls respectively according to the instruction coordinate set and the control information corresponding to the interface images in each custom control image set to obtain detection data.

[0070] Figure 5 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present invention. As Figure 5 shown, the electronic device of this embodiment includes: a processor 501 and a memory 502; where the memory 502 is used to store computer execution instructions; the processor 501 is used to execute the computer execution instructions stored in the memory to implement each step executed by the electronic device in the above embodiment. For details, reference can be made to the relevant descriptions in the foregoing method embodiments.

[0071] Optionally, the memory 502 can be either independent or integrated with the processor 501.

[0072] When the memory 502 is independently provided, the electronic device further includes a bus 503 for connecting the memory 502 and the processor 501.

[0073] An embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored, and when the processor 501 executes the computer execution instructions, the above method is implemented.

[0074] An embodiment of the present invention further provides a computer program product, including a computer program, and when the computer program is executed by the processor 501, the above method is implemented.

[0075] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be in electrical, mechanical, or other forms.

[0076] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to implement the solution of this embodiment.

[0077] In addition, each functional module in various embodiments of the present invention can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in one unit. The units formed by the above modules can be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0078] The integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 501 to execute some steps of the methods in various embodiments of the present application.

[0079] It should be understood that the above processor 501 can be a central processing unit (Central Processing Unit, abbreviated as CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, abbreviated as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, abbreviated as ASIC), etc. The general-purpose processor can be a microprocessor or the processor 501 can also be any conventional processor 501, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed by the hardware processor 501, or executed by a combination of hardware and software modules in the processor 501.

[0080] The memory 502 may include high-speed RAM memory and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a portable hard drive, a read-only memory, a magnetic disk, or an optical disc, etc.

[0081] The bus 503 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 503 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0082] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0083] An exemplary storage medium is coupled to the processor 501, enabling the processor 501 to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 501. The processor 501 and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor 501 and the storage medium can also exist as discrete components in an electronic device or a master control device.

[0084] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: ROM, RAM, a magnetic disk, or an optical disc, etc., all kinds of media that can store program codes.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile application intelligent detection method based on image content recognition technology, characterized in that: The following steps are involved: Obtaining an interface image of a custom control of a mobile application to be detected during operation; Constructing and training an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model, wherein the image classification model includes a convolutional neural network; Input the interface image of the custom control into the trained image classification model for classification to obtain the corresponding custom control type, and assign the interface image to different custom control image sets according to different custom control types; input the interface image in each custom control image set into the trained image recognition model to identify and obtain the control information corresponding to the interface image in each custom control image set, wherein the control information includes text elements and position information; An automated testing tool is used to identify the native controls in the mobile application to be tested, and an instruction coordinate set is obtained. According to the instruction coordinate set and the control information corresponding to the interface image in each custom control image set, automatic click operations are performed on the native controls and custom controls to obtain test data.

2. The mobile application intelligent detection method based on image content recognition technology according to claim 1 is characterized in that: The different custom control types include sliding, skipping, login, registration, customer service and payment channels, the different custom control image sets include sliding set, skipping set, login set, registration set, customer service set and payment channel set, and the text elements include the names of the operation positions of the custom controls.

3. The mobile application intelligent detection method based on image content recognition technology according to claim 1 is characterized in that: The convolutional neural network adopts the Resnet50 network.

4. The mobile application intelligent detection method based on image content recognition technology according to claim 1 is characterized in that: The image recognition model adopts OCR recognition technology.

5. The mobile application intelligent detection method based on image content recognition technology according to claim 1 is characterized in that: The automated testing tool includes UiAutomator2, which identifies the name of the operation bit of the native control and obtains the position information of the operation bit of the native control with the help of the element positioning tool Weditor. The name of the operation bit of the native control and its position information constitute the instruction coordinate set.

6. The mobile application intelligent detection method based on image content recognition technology according to claim 1 is characterized in that: The automated click operation is implemented using an automated click script, which uses the instruction coordinate set or the control information to perform automated clicks and collects test data generated by the mobile application to be tested after the automated click operation is completed.

7. A mobile application intelligent detection device based on image content recognition technology, characterized in that: include: A data acquisition module is configured to acquire an interface image of a custom control of a mobile application to be detected during operation; A model building module is configured to build and train an image classification model and an image recognition model to obtain a trained image classification model and a trained image recognition model, wherein the image classification model includes a convolutional neural network; The classification and recognition module is configured to input the interface image of the custom control into a trained image classification model for classification, obtain the corresponding custom control type, and assign the interface image to different custom control image sets according to different custom control types; input the interface image in each custom control image set into the trained image recognition model, and recognize and obtain the control information corresponding to the interface image in each custom control image set, wherein the control information includes text elements and position information; The click module is configured to use an automated testing tool to identify the native controls in the mobile application to be tested, obtain an instruction coordinate set, and perform automated click operations on the native controls and custom controls according to the instruction coordinate set and the control information corresponding to the interface image in each custom control image set to obtain detection data.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images