A method and system for identifying the source of mobile application screenshots based on deep learning

Abstract

The present disclosure provides a method and system for identifying the source of mobile application screenshots based on deep learning, which relates to the field of artificial intelligence. The method includes two steps: data collection and model construction and training: the data collection step constructs a mobile application sample library; the model construction and training step includes four steps: step 1, improving the MobileNetV3 network and constructing the corresponding loss function to form an improved MobileNetV3‑Large network; step 2, using the set contrast loss function to train the kernel parameters to obtain a trained mobile application matching model; step 3, processing the pictures in the mobile application sample library; step 4, using the trained mobile application matching model, calculating the Euclidean distance between the specific feature and each feature in the mobile application feature library, and obtaining the minimum value of all Euclidean distances. If it is lower than the threshold, the corresponding mobile application name is returned, which can continuously realize the accurate recognition capability of the latest mobile applications.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a method and system for identifying the source of mobile application screenshots based on deep learning. Background Art

[0002] The number of apps on the market has exploded, and different apps have their own characteristics and styles in terms of functions, user interface (UI) design, icon style, etc. When using an app, users often need to share screenshots to discuss the app's functions, express their views on the design, or share the information content carried by the app. These screenshots contain source information, which plays a vital role in monitoring and analyzing the changes and trends of various apps on the market and tracing and tracking the source of information. Existing solutions for screenshot source identification mainly rely on manual search or text-based metadata identification, which is inefficient and easily limited by factors such as the absence of identifiable text in the screenshot or incomplete metadata.

[0003] With the development of artificial intelligence technology, it has become possible to automatically identify APP screenshots through deep learning technology. The image classification technology used can achieve this goal, but the types of APPs have been changing, that is, the categories of images have been changing. Traditional image classification technology needs to constantly update the model to support new mobile application categories.

[0004] The present disclosure aims to solve the technical problem of the shortcomings of the existing technology for identifying the source of mobile application screenshots. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method and system for identifying the source of mobile application screenshots based on deep learning, which automatically identifies and determines the APP to which the screenshot belongs by analyzing visual elements such as icon features and layout features in the APP screenshot, and supports dynamically changing APP data.

[0006] In general, a method for identifying the source of mobile application screenshots based on deep learning is provided, which includes two steps: data collection and model construction and training:

[0007] The data collection step collects system updated data through a data collection module to collect application description information and screenshots of related main pages, and performs data cleaning and data completion to build a mobile application sample library;

[0008] The model building and training steps include four steps:

[0009] Step 1, improve the MobileNetV3 network and construct the corresponding loss function to form an improved MobileNetV3-Large network, pre-process the image data of the screenshots of the main pages related to the application in the mobile application sample library and extract features through the MobileNetV3-Large network; then construct a MobileNetV3-Sim network that extracts feature maps such as layout, icons, and color matching in the picture, and construct a contrast loss function;

[0010] Step 2, using the feature-extracted mobile application image data as the input of the MobileNetV3-Sim network, using the set contrast loss function to train the kernel parameters, and obtaining a trained mobile application matching model;

[0011] Step 3, using the trained mobile application matching model, processing the images in the mobile application sample library to obtain a mobile application feature library of <mobile application name, mobile application feature>;

[0012] Step 4: Use the trained mobile application matching model to extract specific features from the mobile application screenshots that need to be identified, calculate the Euclidean distance between the specific features and each feature in the mobile application feature library, and obtain the minimum value of all Euclidean distances. If it is lower than the threshold, return the corresponding mobile application name.

[0013] The specific method of constructing the mobile application sample library is as follows: first, the software is used to monitor the large mainstream channel distribution platforms on the market, collect application metadata information such as application name, publisher, application description, application size, application version, application screenshots, and download volume, and regularly refresh the data updated by the collection system;

[0014] The collected application metadata information is saved to the PostgreSQL database, with one row for each application, including the following column names: application ID, application name, publisher, application description, application size, application version, download volume, application screenshot storage path, storage date, update date, cleaning flag, and remarks;

[0015] The application screenshot file is saved to the MinIO object storage system;

[0016] For the collected application information, duplicate elimination operations are performed according to the application name, publishing manufacturer, application description, and application size; for different versions of the same mobile application, a record is saved in the application metadata information, and the application screenshots corresponding to the recorded version are saved; for mobile applications that lack application screenshots, corresponding application screenshots are collected from search engines and forums based on the mobile name.

[0017] The preprocessing includes: one or a combination of image graying, image sharpening, image cropping, image mirroring, image rotation, image blurring, image smearing, and adding text to the image.

[0018] The image grayscale converts a 256-color image into a grayscale image, and uses the RGB three-primary color system to convert to the lightness approximation constructor of the Munsell color system, and the formula is f(x,y)=0.3×R(x,y)+0.59×G(x,y)+0.11×B(x,y).

[0019] The image sharpening uses a high-pass filtering method to remove low-frequency components of the image and perform sharpening processing on the image.

[0020] The image cropping is to randomly crop the image, and the cropped area is no more than 20%.

[0021] The image mirroring flips an image in the horizontal or vertical direction.

[0022] The image rotation rotates the image randomly by 1 to 10 degrees.

[0023] The image blurring is accomplished by adding Gaussian noise, and the signal-to-noise ratio is maintained within 10%.

[0024] The image smearing is to randomly add lines and color blocks in different directions to the image.

[0025] The adding of text to the image is to randomly add text information to the image.

[0026] The improved MobileNetV3-Large network structure is: Input triple<x,x+,x-> , where x is the sample iterated during training, x+ is a randomly selected screenshot of the same APP as x, and x- is a randomly selected screenshot of a different APP from x;

[0027] The samples iterated during training, randomly selected screenshots of the same APP as the samples iterated during training, and randomly selected screenshots of APPs different from x are respectively input into a MobileNetV3-Sim network. The weights are shared among the three MobileNetV3-Sim networks, and the output results of the three MobileNetV3-Sim networks are trained by the loss function.

[0028] The MobileNetV3-Sim includes an input layer, 15 improved E-BNeck structures and an output layer, and constructs a new loss function. The model receives a 3-channel image as input. The input layer is 448*448 pixels. The input image first passes through a convolution layer to increase the number of channels from 3 to 16, and is downsampled by a convolution kernel with a step size of 4, and then passes through batch normalization and h-swish activation function to obtain the input layer result. After being processed by 15 improved E-BNeck structures, it enters the output layer. The output layer uses a 1x1 convolution to mix features of different scales, and is further processed by batch normalization and h-swish activation function; then a 7*7 global average pooling layer is used to downsample each feature map to a single output value of 1*1*960 dimensions; finally, a 1*1 convolution layer, batch normalization and h-swish activation function are used to transform the pooled output to a 1*1*1280 high-dimensional space, and a Dropout with a parameter of 0.2 is used for regularization.

[0029] The improved E-BNeck structure includes a convolution operation with a convolution kernel of 1*1, batch normalization, and h-swish activation function, a depth convolution operation with a convolution kernel of N*N and a step size of 1 or 2, batch normalization, and h-swish activation function, and a point-by-point convolution operation with a convolution kernel of 1*1, batch normalization and h-swish activation function, and also includes a CA attention mechanism module that adds a spatial attention weight mechanism.

[0030] The parameters of the 15 E-BNecks are as follows: the input of the first E-BNeck is 112*112*16, the convolution kernel size is 3, the expansion size is 16, the number of output channels is 16, the step size is 1, and the activation function is ReLU; the input of the second E-BNeck is 112*112*16, the convolution kernel size is 3, the expansion size is 64, the number of output channels is 24, the step size is 2, and the activation function is ReLU; the input of the third E-BNeck is 56*56*24, the convolution kernel size is 3, the expansion size is 72, the number of output channels is 24, the step size is 1, and the activation function is ReLU; the input of the fourth E-BNeck is 56*56*24, the convolution kernel size is 3, the expansion size is 72, the number of output channels is 24, the step size is 1, and the activation function is ReLU. The size is 5, the expansion size is 72, the number of output channels is 40, the step size is 2, and the activation function is ReLU; the input of the fifth E-BNeck is 28*28*40, the convolution kernel size is 5, the expansion size is 120, the number of output channels is 40, the step size is 1, and the activation function is ReLU; the input of the sixth E-BNeck is 28*28*40, the convolution kernel size is 5, the expansion size is 120, the number of output channels is 40, the step size is 1, and the activation function is ReLU; the input of the seventh E-BNeck is 28*28*40, the convolution kernel size is 3, the expansion size is 240, the number of output channels is 80, the step size is 2, and the activation function is H-Swish; the eighth E-BNeck is 28*28*40, the convolution kernel size is 3, the expansion size is 240, the number of output channels is 80, the step size is 2, and the activation function is H-Swish; -BNeck's input is 14*14*80, the convolution kernel size is 3, the expansion size is 200, the number of output channels is 80, the step size is 1, and the activation function is H-Swish; the ninth E-BNeck's input is 14*14*80, the convolution kernel size is 3, the expansion size is 184, the number of output channels is 80, the step size is 1, and the activation function is H-Swish; the tenth E-BNeck's input is 14*14*80, the convolution kernel size is 3, the expansion size is 184, the number of output channels is 80, the step size is 1, and the activation function is H-Swish; the eleventh E-BNeck's input is 14*14*80, the convolution kernel size is 3, and the expansion size is 480 , the number of output channels is 112, the step size is 1, and the activation function is H-Swish; the input of the twelfth E-BNeck is 14*14*112, the convolution kernel size is 3, the expansion size is 672, the number of output channels is 112, the step size is 1, and the activation function is H-Swish; the input of the thirteenth E-BNeck is 14*14*112, the convolution kernel size is 5, the expansion size is 672, the number of output channels is 160, the step size is 2, and the activation function is H-Swish; the input of the fourteenth E-BNeck is 7*7*160, the convolution kernel size is 5, the expansion size is 960, the number of output channels is 160, the step size is 1, and the activation function is H-Swish;The input of the fifteenth E-BNeck is 7*7*160, the convolution kernel size is 5, the expansion size is 960, the number of output channels is 160, the step size is 1, and the activation function uses H-Swish. ;

[0031] The CA attention mechanism module is divided into two steps: step 1: coordinate information embedding, step 2: coordinate attention generation;

[0032] The coordinate information embedding method is: for an input image containing three-dimensional information of width, height, and color, first use pooling kernels of size (H, 1) and (1, W) to encode each channel along the horizontal coordinate direction and the vertical coordinate direction, then the output of the Cth channel with a height of H and a width of W is expressed as:

[0033] ;

[0034] ;

[0035] The two feature maps generated by the previous module are cascaded and transformed using a shared 1x1 convolution F1, expressed as: ,in , is the intermediate feature map of spatial information in the horizontal and vertical directions, and r represents the downsampling ratio;

[0036] Next, along the spatial dimension Split into two separate tensors and , and then use two 1x1 convolutions Fh and Fw to transform the feature maps fh and fw to the same number of channels as the input:

[0037] ;

[0038] The attention generation module and Expanding on this, the final output of the CA attention module can be expressed as follows:

[0039] ;

[0040] The final module output of E-BNeck is the sum of the residual result s, the attention structure y, and the dilated convolution result c.

[0041] The new loss function of the improved MobileNetV3-large model is an improved triplet loss function: its input is a triplet<x,x+,x-> The parameters in are the output values ​​after passing through the improved MobileNetV3-Sim network, where y is the output value of the iterated sample x during training, and y i+ is a randomly selected APP screenshot x that is the same as x i + The output value, y i - Then randomly select screenshot x of an APP different from x i - The output value is specifically expressed as:

[0042] L ];

[0043] Its goal is to continuously optimize and reduce the distance between x and x+ samples, while expanding the distance between x and x- samples, where y is the output of x after the MobileNetV3-Sim network, dis is the Euclidean distance between the two ys, and t is the temperature coefficient used to control the distribution shape of L. The larger the t value, the smoother the distribution of L.

[0044] The training process of the improved MobileNetV3-large model is as follows: for each batch of input data, all screenshots of an APP are randomly selected, and then other parts are randomly selected from all screenshots of all other APPs as negative examples; the data are input into the model after data enhancement methods such as cropping, blurring, and grayscale; the Adam algorithm of the back-propagation optimization algorithm is used, and the training process uses multiple iterations until the training is completed when the loss no longer decreases.

[0045] In another general aspect, a mobile application screenshot source identification system based on deep learning comprises steps 1-3:

[0046] Step 1, information collection: The collection module continuously collects newly released mobile application screenshots from the application market. After cleaning, the mobile application screenshots are inferred and calculated through the trained feature extraction model to obtain a 1280-dimensional feature vector f app ;

[0047] The collected content includes the application name, publisher, application description, application size, application version, application screenshots, download volume, and the data updated by the collection system is refreshed regularly;

[0048] Step 2, vector library construction: record feature vector f app The corresponding relationship between the feature vector and the mobile application name; specifically, the feature vector is stored in the Milvus vector library, and when the number of mobile applications is greater than a preset threshold, the retrieval speed is accelerated, which specifically includes the following steps:

[0049] 1) Use the Milvus Python SDK to establish a connection with the Milvus service, initialize the collection, and create an HNSW index;

[0050] 2) Obtain the mobile application screenshots from MinIO, perform a resize operation, and perform normalization processing to obtain image data with dimensions [448, 448, 3];

[0051] 3) Perform forward propagation calculation of the model on the image data to obtain the feature vector of the image, with dimensions [1280, 1];

[0052] 4) Batch import the extracted feature vector data into the pre-created collection in Milvus through the interface of the Milvus Python SDK;

[0053] 5) Record the correspondence between the Milvus vector id and the mobile application name in PostgreSQL;

[0054] Step 3: Identification of the source of mobile application screenshots, which specifically includes the following steps:

[0055] 1) Perform a resize operation on the image to be recognized and perform pre-normalization processing to obtain RGB image data with dimensions [448, 448, 3];

[0056] 2) Perform forward propagation calculation of the model on the RGB image data to obtain a feature vector f with dimensions [1280, 1] img ;

[0057] 3) Retrieve the vector ID with the top 1 Euclidean distance in the Milvus feature vector library for the obtained feature vector f img Calculate the Euclidean distance d for each feature f i Calculate the Euclidean distance d;

[0058] 4) When making a result determination, a threshold needs to be set in advance. The setting of this threshold is based on the model training results and experience. Determine whether d is less than the threshold D. The threshold D is calculated according to statistical values during the training process; if d > D, it is determined that the picture does not belong to any mobile application. If d < D, query the corresponding mobile application name from PostgreSQL according to the vector ID.

[0059] The innovation of the embodiment of the present invention lies in:

[0060] 1. The method disclosed in the present invention collects description information of mobile applications and related screenshots of main pages to build a mobile application sample library. MobileNetV3 is used as the backbone network, and its network architecture and loss function are modified. Based on the image data in the mobile application sample library, a MobileNetV3-Sim network (i.e., a MobileNetV3-Sim deep neural network model) is constructed to extract feature maps such as layout, icons, and color schemes in the image; the processed image data is used as the input of the end-to-end matching framework, and the kernel loss function is set to obtain a trained feature extraction model; the trained feature extraction model is used to process the mobile application sample library to build a library of <mobile application name, mobile application features>. A new mobile application screenshot to be identified is input, and the system uses the trained deep neural network model to extract features from the image, compares it with the pre-trained database, and outputs the probability that the screenshot belongs to each application. Features such as mobile application layout and color scheme are extracted through the modified MobileNetV3 network, and the contrast loss function is used to make the model have strong resolution in mobile applications with high similarity.

[0061] 2. The method disclosed in the present invention adopts a dynamic update mechanism to regularly obtain the latest application information and screenshot data from major channels so as to timely update the training data set of the deep neural network model, so that the system can continue to maintain the ability to accurately identify the latest mobile applications.

[0062] 3. The mobile application screenshot source identification system based on similarity retrieval provided by the present invention automatically builds a mobile application sample library through acquisition technology, and will be automatically updated over time; the recognition of mobile application screenshots is achieved by calculating the distance with the mobile application sample library. When the mobile application sample library increases, there is no need to update the weight parameters of MobileNetV3 and can still maintain a good recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0064] Figure 1 is a schematic diagram showing a method for identifying the source of mobile application screenshots based on deep learning and a system architecture diagram according to an embodiment of the present disclosure;

[0065] Figure 2 is a schematic diagram showing a twin network architecture diagram based on an improved MobileNetV3-Large according to an embodiment of the present disclosure;

[0066] Figure 3 is a schematic diagram showing the architecture of the backbone network MobileNetV3-Sim according to an embodiment of the present disclosure;

[0067] Figure 4 is a schematic diagram showing an E-BNeck architecture diagram based on MobileNetV3 according to an embodiment of the present disclosure;

[0068] Figure 5 It is a schematic diagram showing an architecture diagram of a mobile application screenshot source identification system based on deep learning according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0070] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0071] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0072] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0073] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0074] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0075] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0076] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0077] Example 1

[0078] Figure 1 It is a schematic diagram showing a method for identifying the source of mobile application screenshots based on deep learning according to Example 1 of the present disclosure.

[0079] A method for identifying the source of mobile application screenshots based on deep learning includes the following specific implementation steps: including two steps: data collection and model construction and training:

[0080] Data collection step: Through the data collection module, the system updates the data to collect the application description information and screenshots of the main pages, and then clean and complete the data to build a mobile application sample library. Specifically:

[0081] Through the data collection module, the monitoring covers large mainstream channel distribution platforms on the market, including: Huawei AppGallery, OPPO Software Store, vivo App Store, Xiaomi App Store, Meizu Flyme App Store, App Store, Baidu Mobile Assistant, 360 Mobile Assistant, Wandoujia, Samsung Market, Apple APP Store, etc. The collected content includes application name, publishing manufacturer, application description, application size, application version, application screenshots, download volume, and regularly refreshes the data updated by the collection system.

[0082] The collected application metadata information is saved in the PostgreSQL database, with one row for each application. The column names include: application ID, application name, publisher, application description, application size, application version, download volume, application screenshot storage path, storage date, update date, cleaning flag, and remarks.

[0083] Application screenshot files will be saved to the MinIO object storage system.

[0084] Data cleaning: Duplicate applications are removed from the collected application information by application name, publisher, application description, and application size. Specifically, applications with a size less than 1M are deleted; applications with a download size less than 100 bytes are deleted; applications with descriptions less than 10 words and incoherent or colloquial text are deleted; and applications with unselected software categories are deleted.

[0085] For different versions of the same mobile application, one metadata entry is saved and the corresponding application screenshots are saved at the same time.

[0086] Data completion: For mobile applications that lack application screenshots, corresponding images are collected from search engines, forums, etc. according to the mobile name; specifically, https: / / image.baidu.com / search / index?word=mobile application name+Android+app screenshots is used to search for the top 10 images, which are then manually reviewed and used as application screenshots.

[0087] Model building and training steps build a specific screenshot-based application recognition method, which specifically includes:

[0088] 1) Before identifying the application screenshot, the application screenshot may be preprocessed. The purpose of preprocessing is to improve the quality of the application screenshot and increase the generalization of the model. Preprocessing may specifically include at least a combination of image grayscale, image sharpening, image cropping, image mirroring, image rotation, image blurring, image smearing, and adding text to the image.

[0089] Image grayscale is mainly used to convert 256-color images into grayscale images, removing color differences and retaining only brightness differences. The lightness approximation constructor that uses the RGB three-primary color system to convert to the Munsell color system is f(x,y)=0.3×R(x,y)+0.59×G(x,y)+0.11×B(x,y).

[0090] Image sharpening is mainly to make the blurred image clear, and high-pass filtering is used to remove the low-frequency components of the image to sharpen the image. Image cropping, in simple terms, is to select a specific area from the original image and delete the rest. In this embodiment, random image cropping will be performed, and the cropped area is no more than 20%.

[0091] Image mirroring means flipping an image to make it symmetrical horizontally or vertically.

[0092] Image rotation refers to rotating the image by a random angle of 1 to 10 degrees to change its display direction.

[0093] Image blurring is accomplished by adding Gaussian noise, and the signal-to-noise ratio is kept within 10%.

[0094] Image smearing refers to adding some colors or textures to the image. In this embodiment, lines and color blocks in different directions are randomly added, and text information is randomly added to the image.

[0095] 2) Determine the network structure.

[0096] In this embodiment, if Figure 2 As shown in the figure, it is a twin network based on the improved MobileNetV3-Large. It extracts app screenshot features based on contrastive learning. The network structure is shown in Figure 2 shown.

[0097] The input data are labeled x+ and x-, indicating positive and negative sample sets. These input data will be used to train the model. The input data first passes through the "MobileNetV3-Sim" module for feature extraction. This module contains convolutional layers (such as Conv2d) and pooling layer structures to learn useful feature representations from raw data. Positive and negative samples pass through multiple branches, but the weight parameters are shared. The extracted features are calculated and optimized through the Information Noise Contrastive Estimation Loss (InfoNCE Loss) module. InfoNCE Loss is a method to measure the ability of a model to distinguish between positive and negative samples. It does this by maximizing the similarity (i.e., inner product) between pairs of positive samples and minimizing the similarity between pairs of negative samples. Finally, after multiple iterations and optimizations, the model will generate a feature representation that can effectively distinguish between positive and negative samples.

[0098] Among them, the backbone network MobileNetV3-Sim is based on the transformation of MobileNetV3-large. Its network structure is as follows Figure 3 shown.

[0099] MobileNetV3-Sim increases the input layer to 448*448 pixels. The network structure includes an input convolution layer, 15 improved E-BNeck structures and an output layer. The parameters of the 15 E-BNecks are as follows:

[0100]

[0101] process:

[0102] 1. The model receives a 3-channel image as input. The input image first passes through a convolutional layer to increase the number of channels from 3 to 16, and is downsampled by a convolution kernel with a stride of 4, so that the size of the feature map is reduced to 1 / 4. This is followed by a batch normalization layer (BN) and an activation function H-swish. The result is input to E-BNeck.

[0103] 2. The structure of E-BNeck is based on the BNeck structure of MobileNetV3. The attention SE module is replaced by the CA attention mechanism. The improved module adds a spatial attention weight mechanism relative to the SE module, making full use of the position information of each element of the APP screenshot. The structure of E-BNeck is as follows: Figure 4 shown.

[0104] For a deep network, in order to pass more details to the next layer, more residual structures are added based on the residual algorithm of BNeck. When the step size is 1, the input and output sizes do not change. Only one convolution layer is added to change the number of channels. When the step size is 2, two convolution layers are added. The first convolution layer changes the image size, and the second convolution layer changes the number of image channels. In either case, the convolution layer does not add a nonlinear layer, retaining the biggest features of the original image.

[0105] The improved BNeck SE attention mechanism is the CA attention mechanism, which encodes channel relationships and long-range dependencies through precise location information. It is divided into two steps: coordinate information embedding and coordinate attention generation. Specifically, for the input image (W, H, C), the pooling kernels of size (H, 1) and (1, W) are first used to encode each channel along the horizontal and vertical coordinate directions, namely horizontal pooling (X Avg-Pooling) and vertical pooling (Y Avg-Pooling). Therefore, the output of the Cth channel with a height of H and a width of W is expressed as follows

[0106] ;

[0107] .

[0108] Then concatenate (concat(X,Y)) the two feature maps generated by the previous module, and then use a shared 1x1 convolution to transform F1, which is expressed as follows, generating It is the intermediate feature map of the spatial information in the horizontal and vertical directions, where r represents the downsampling ratio, which is 1 in this embodiment:

[0109] .

[0110] Next, split f into two separate tensors along the spatial dimension and , and then use two 1x1 convolutions Fh and Fw to transform the feature maps fh and fw to the same number of channels as the input, and get the following result.

[0111] .

[0112] Then and Expanding on this, as the attention weight, the final output of the CA attention module can be expressed as follows.

[0113] .

[0114] The final module output of E-BNeck is the sum of the residual result s, the attention structure y, and the dilated convolution result c.

[0115] The output layer uses a 1x1 convolution to mix features of different scales, and is further processed by a batch normalization layer and an activation function h-swish. Then, a 7*7 global average pooling layer (Avg-Pooling) is used, which downsamples each feature map to a single 1*1*960 output value, and then a 1*1 convolution layer, batch normalization and h-swish activation function, and a Dropout layer are used to transform the pooled output into a 1*1*1280 vector.

[0116] 3) Determine the model loss function. The loss function used in the model training is Triplet loss. Its input is a triplet<x,x+,x-> , where y is the output value of the iterated sample x during training, y i + is a randomly selected APP screenshot x that is the same as x i + The output value, y i - Then randomly select screenshot x of an APP different from x i - The goal of this loss function is to continuously optimize and reduce the distance between x and x+ samples, while expanding the distance between x and x- samples, where y is the output of x after MobileNetV3-Sim, and dis is the Euclidean distance between y.

[0117] L ];

[0118] Where t is the temperature coefficient used to control the distribution shape of L. The larger the t value, the smoother the distribution of L.

[0119] 4) Determine the training method. For each batch of input data, randomly select all screenshots of an APP, and then randomly select other parts from all screenshots of all other APPs as negative examples. After data enhancement methods such as cropping, blurring, and grayscale, input the model. The network structure and loss function are determined by steps one and two. The back propagation optimization algorithm uses the Adam algorithm. The training process uses multiple epochs to iterate repeatedly until the loss stops decreasing and the training is completed.

[0120] Example 2

[0121] This embodiment discloses a mobile application screenshot source identification system based on deep learning, including the following specific implementation steps.

[0122] Step 1: Information collection. The collection module continuously collects newly released mobile application screenshots from the application market. After cleaning, the mobile application screenshots are inferred and calculated through the trained feature extraction model to obtain a 1280-dimensional feature vector fapp, which represents the image feature that can best distinguish different mobile applications.

[0123] Through the data collection module, the monitoring covers large mainstream channel distribution platforms on the market, including: Huawei AppGallery, OPPO Software Store, vivo App Store, Xiaomi App Store, Meizu Flyme App Store, App Store, Baidu Mobile Assistant, 360 Mobile Assistant, Wandoujia, Samsung Market, Apple APP Store, etc. The collected content includes application name, publishing manufacturer, application description, application size, application version, application screenshots, download volume, and regularly refreshes the data updated by the collection system.

[0124] Step 2: vector library construction. Record the correspondence between the feature vector fapp and the mobile application name. Optionally, the feature vector can be stored in the Milvus vector library to speed up the search when the number of mobile applications is large (for example, greater than a preset threshold).

[0125] 1) Use Milvus' Python SDK to establish a connection with the Milvus service, initialize the collection, and create an HNSW index.

[0126] 2) Get the mobile application screenshot from MinIO, resize it, and normalize it to get the image data with the dimension of [448,448,3].

[0127] 3) Use the feature extraction model trained above to perform forward propagation calculations on the image data to obtain the feature vector of the image with a dimension of [1280, 1].

[0128] 4) Import the extracted feature vector data into the pre-created collection in Milvus in batches through the Milvus Python SDK interface.

[0129] 5) Record the correspondence between Milvus vector ID and mobile application name in PostgreSQL.

[0130] Step 3: Identify the source of mobile app screenshots.

[0131] 1) Resize the image with recognition and perform normalization preprocessing to obtain image RGB data with a dimension of [448, 448,3].

[0132] 2) Use the feature extraction model trained above to perform forward propagation calculation on the RGB data of the image to obtain a feature vector fimg with a dimension of [1280, 1].

[0133] 3) Retrieve the vector ID with the top 1 Euclidean distance in the feature vector library of Milvus for the obtained feature vector fimg, and calculate the Euclidean distance d for each feature fi.

[0134] 4) When making the result determination, a threshold needs to be set in advance. The setting of this threshold is based on the model training results and experience. Determine whether d is less than the threshold D, and the threshold D is calculated according to statistical values during the training process. If d > D, it is determined that the picture does not belong to any mobile application. If d < D, query the corresponding mobile application name from PostgreSQL according to the vector ID.

[0135] Although some embodiments of the present disclosure have been shown and described, those skilled in the art should understand that these embodiments can be modified without departing from the principles and spirit of the present disclosure as defined by the claims and their equivalents.

Claims

1. A method for identifying the source of mobile application screenshots based on deep learning, characterized in that: It includes two steps: data collection and model building and training: The data collection step collects application description information and screenshots of related main pages through a data collection module, and performs data cleaning and data completion to build a mobile application sample library; The model building and training steps include four steps: Step 1, improving the MobileNetV3 network and constructing a corresponding loss function to form an improved MobileNetV3-Large network, preprocessing the image data of the screenshots of the main pages related to the application in the mobile application sample library and extracting features through the MobileNetV3-Large network; then constructing a MobileNetV3-Sim network for extracting feature maps in the image, and constructing a contrast loss function, wherein the feature maps include layout, icon, and color matching; Step 2, using the feature-extracted mobile application image data as the input of the MobileNetV3-Sim network, using the set contrast loss function to train the kernel parameters, and obtaining a trained mobile application matching model; Step 3, using the trained mobile application matching model, processing the images in the mobile application sample library to obtain a mobile application feature library of <mobile application name, mobile application feature>; Step 4: using the trained mobile application matching model, extract specific features from the mobile application screenshots to be identified, calculate the Euclidean distance between the specific features and each feature in the mobile application feature library, and obtain the minimum value of all Euclidean distances. If the minimum value is lower than the threshold, return the corresponding mobile application name. The improved MobileNetV3-Large network structure is as follows: Input triplet<x,x+,x-> , where x is the sample iterated during training, x+ is a randomly selected screenshot of the same APP as x, and x- is a randomly selected screenshot of a different APP from x; The samples iterated during training, randomly selected screenshots of the same APP as the samples iterated during training, and randomly selected screenshots of APPs different from x are respectively input into a MobileNetV3-Sim network. The weights are shared among the three MobileNetV3-Sim networks, and the output results of the three MobileNetV3-Sim networks are trained by the loss function.

2. A method for identifying the source of mobile application screenshots based on deep learning as claimed in claim 1, characterized in that: The specific method of constructing the mobile application sample library is as follows: first, the software is used to monitor the large mainstream channel distribution platforms on the market, collect application metadata information such as application name, publisher, application description, application size, application version, application screenshots, and download volume, and regularly refresh the data updated by the collection system; The collected application metadata information is saved to the PostgreSQL database, with one row for each application, including the following column names: application ID, application name, publisher, application description, application size, application version, download volume, application screenshot storage path, storage date, update date, cleaning flag, and remarks; The application screenshot file is saved to the MinIO object storage system; For the collected application information, duplicate elimination operations are performed according to the application name, publishing manufacturer, application description, and application size; for different versions of the same mobile application, a record is saved in the application metadata information, and the application screenshots corresponding to the recorded version are saved; for mobile applications that lack application screenshots, corresponding application screenshots are collected from search engines and forums based on the mobile name.

3. A method for identifying the source of mobile application screenshots based on deep learning as claimed in claim 2, characterized in that: The preprocessing includes: one or a combination of image graying, image sharpening, image cropping, image mirroring, image rotation, image blurring, image smearing, and adding text to the image.

4. The method for identifying the source of mobile application screenshots based on deep learning as claimed in claim 1, characterized in that: The MobileNetV3-Sim includes an input layer, 15 improved E-BNeck structures and an output layer, and constructs a new loss function. The model receives a 3-channel image as input, and the input layer is 448*448 pixels. The input image first passes through a convolution layer to increase the number of channels from 3 to 16, and is downsampled by a convolution kernel with a step size of 4, and after batch normalization and h-swish activation function, the input layer result is obtained; after being processed by 15 improved E-BNeck structures, it enters the output layer; the output layer uses a 1x1 convolution to mix features of different scales, and is further processed by batch normalization and h-swish activation function; then a 7*7 global average pooling layer is used to downsample each feature map to a single output value of 1*1*960 dimensions; finally, a 1*1 convolution layer, batch normalization and h-swish activation function are used to transform the pooled output to a 1*1*1280 high-dimensional space, and a Dropout with a parameter of 0.2 is used for regularization.

5. A method for identifying the source of mobile application screenshots based on deep learning as claimed in claim 4, characterized in that: The improved E-BNeck structure includes a convolution operation with a convolution kernel of 1*1, batch normalization, and h-swish activation function, a depth convolution operation with a convolution kernel size of N and a step size of 1 or 2, batch normalization, and h-swish activation function, and a point-by-point convolution operation with a convolution kernel of 1*1, batch normalization and h-swish activation function, and also includes a CA attention mechanism module that adds a spatial attention weight mechanism.

6. A method for identifying the source of mobile application screenshots based on deep learning as claimed in claim 5, characterized in that: The parameters of the 15 improved E-BNecks are as follows: the input of the first E-BNeck is 112*112*16, the convolution kernel size is 3, the expansion size is 16, the number of output channels is 16, the step size is 1, and the activation function is ReLU; the input of the second E-BNeck is 112*112*16, the convolution kernel size is 3, the expansion size is 64, the number of output channels is 24, the step size is 2, and the activation function is ReLU; the input of the third E-BNeck is 56*56*24, the convolution kernel size is 3, the expansion size is 72, the number of output channels is 24, the step size is 1, and the activation function is ReLU; the fourth E-BN The input of eck is 56*56*24, the convolution kernel size is 5, the expansion size is 72, the number of output channels is 40, the step size is 2, and the activation function is ReLU; the input of the fifth E-BNeck is 28*28*40, the convolution kernel size is 5, the expansion size is 120, the number of output channels is 40, the step size is 1, and the activation function is ReLU; the input of the sixth E-BNeck is 28*28*40, the convolution kernel size is 5, the expansion size is 120, the number of output channels is 40, the step size is 1, and the activation function is ReLU; the input of the seventh E-BNeck is 28*28*40, the convolution kernel size is 3, and the expansion size is 2 40, the number of output channels is: 80, the step size is: 2, and the activation function is H-Swish; the input of the eighth E-BNeck is 14*14*80, the convolution kernel size is: 3, the expansion size is: 200, the number of output channels is: 80, the step size is: 1, and the activation function is H-Swish; the input of the ninth E-BNeck is 14*14*80, the convolution kernel size is: 3, the expansion size is: 184, the number of output channels is: 80, the step size is: 1, and the activation function is H-Swish; the input of the tenth E-BNeck is 14*14*80, the convolution kernel size is: 3, the expansion size is: 184, the number of output channels is: 80, the step size is: 1, and the activation function is The activation function uses H-Swish; the input of the eleventh E-BNeck is 14*14*80, the convolution kernel size is 3, the expansion size is 480, the number of output channels is 112, the step size is 1, and the activation function uses H-Swish; the input of the twelfth E-BNeck is 14*14*112, the convolution kernel size is 3, the expansion size is 672, the number of output channels is 112, the step size is 1, and the activation function uses H-Swish; the input of the thirteenth E-BNeck is 14*14*112, the convolution kernel size is 5, the expansion size is 672, the number of output channels is 160, the step size is 2, and the activation function uses H-Swish;The input of the fourteenth E-BNeck is 7*7*160, the convolution kernel size is 5, the expansion size is 960, the number of output channels is 160, the step size is 1, and the activation function is H-Swish; The input of the fifteenth E-BNeck is 7*7*160, the convolution kernel size is 5, the expansion size is 960, the number of output channels is 160, the step size is 1, and the activation function is H-Swish. ;

Images