Mobile application interface abnormity identification method based on variational auto-encoder

Abstract

The invention discloses a mobile application interface abnormity identification method based on a variational auto-encoder, and the method comprises the following steps: obtaining mobile application interface screenshot data, carrying out the preprocessing, and generating a training data set; performing feature extraction on the preprocessed interface image by using a pre-trained feature extraction network to generate a feature vector; building and training a variational auto-encoder model; and inputting the screenshot of the interface to be detected into the trained variational auto-encoder model for anomaly detection, and outputting a result. The method has the advantages that high accuracy can still be achieved under the condition that sample data is not much, manual intervention and resource requirements are reduced, the anomaly detection efficiency is improved, interface anomaly judgment can be conducted on various mobile phone models, cross-device detection is achieved, and the anomaly detection universality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and recognition, and particularly to a method for abnormal recognition of mobile application interfaces based on variational autoencoders. Background Art

[0002] The abnormal recognition of mobile application interfaces is crucial for user experience and application stability. However, during the use of mobile applications (Apps), interface abnormal problems such as unable to open pages, disordered component layouts, and unknown error prompt windows often occur. These abnormal problems not only affect the user experience but may also lead to application crashes. How to discover the abnormalities of application interfaces in a timely and efficient manner with less human and time investment has become an important problem faced by each mobile application development company.

[0003] Currently, the traditional mobile application testing methods have the following main drawbacks:

[0004] 1. Manual testing is labor-intensive: Manual testing requires a large amount of human resources. Testers need to execute test cases one by one, which is time-consuming and error-prone.

[0005] 2. Limited test coverage: Due to the limitations of test scenarios and boundary conditions, it is difficult for manual testing to achieve comprehensive coverage and potential problems are easily missed.

[0006] 3. Repetitive work: Manual testing usually involves a large amount of repetitive work, especially in regression testing, where testers need to repeatedly execute the same test cases multiple times.

[0007] 4. Device and environment limitations: Mobile application testing needs to be carried out on various devices and operating system versions, and it is difficult for traditional testing methods to cover all device and environment combinations, affecting the test effect.

[0008] Based on the above drawbacks, there is an urgent need for a more intelligent and efficient application interface anomaly detection solution to improve the testing efficiency, expand the test coverage, and discover anomalies in a timely manner. Summary of the Invention

[0009] In order to solve the problems of low efficiency, insufficient coverage, a large amount of repetitive work, and device environment limitations existing in traditional mobile application testing methods, the present invention provides a method for abnormal detection of mobile application interfaces based on variational autoencoders. The present invention realizes feature learning and reconstruction of interface images by training a variational autoencoder model, and uses an anomaly scoring mechanism to automatically determine whether there are anomalies in the interface images to be tested. This method can quickly and accurately detect abnormal situations of application interfaces without manual intervention. Through the present invention, the human and time costs can be significantly reduced, and the intelligence and efficiency of mobile application testing can be improved.

[0010] To achieve the above-mentioned invention object, the present invention provides a method for detecting anomalies in mobile application interfaces based on variational autoencoders, and the method includes the following steps:

[0011] S1: Obtain mobile application interface screenshot data and perform preprocessing to generate a training data set;

[0012] S2: Use a pre-trained feature extraction network to extract features from the preprocessed interface images to generate feature vectors;

[0013] S3: Build and train a variational autoencoder model;

[0014] S4: Input the screenshot of the interface to be tested into the trained variational autoencoder model for anomaly detection and output the results.

[0015] Among them, the step of S1 obtaining mobile application interface screenshot data specifically includes the following steps:

[0016] a) Use Python to run an APP automated test program, record the screenshots of each page of the mobile application and save them;

[0017] b) Select the normal interface screenshots and record them as the image data set Dataset1, where Dataset1 = {x 1 , x 2 ,..., x n}, and x i represents a specific interface screenshot instance.

[0018] The preprocessing in S1 includes the following steps:

[0019] a) Resize the interface image to 224×224 pixels;

[0020] b) Normalize the image pixel values to the range of [-1, 1];

[0021] c) Standardize the image so that it is converted into a standard normal distribution;

[0022] d) Use data augmentation technology to expand the data set Dataset1 to obtain the augmented data set Dataset2 = {x 1 ′, x 2 ′,..., x′ n}.

[0023] The feature extraction steps in S2 include:

[0024] a) Use a pre-trained VGG network to extract features from the preprocessed image data;

[0025] b) Convert the image data into feature vectors {f1 , f 2 , ..., f n}。

[0026] The variational autoencoder model in S3 includes an encoder and a decoder; S3 includes the following steps:

[0027] S301: Use the encoder to map the feature vector to the latent space to obtain the mean vector and variance vector of the latent variable;

[0028] S302: Use the decoder to process the latent variable to generate the corresponding reconstructed features;

[0029] S303: Define the loss function of the VAE;

[0030] S304: Use the training dataset to minimize the loss function through an optimizer to train the variational autoencoder model;

[0031] S305: Set a threshold by analyzing the reconstruction error distribution of the training data to distinguish normal images from abnormal images.

[0032] The encoder maps the input feature vector f i to the latent space, generating the mean vector u i and the variance vector The formula is as follows:

[0033]

[0034] The decoder generates the reconstructed data according to the distribution of the latent variable z i The formula is as follows:

[0035] p(z i ) = N(z i |0, I).

[0036] The loss function of the variational autoencoder model consists of the reconstruction error and the KL divergence. The specific formula is as follows:

[0037]

[0038] Among them, θ and are the parameters of the decoder and encoder respectively.

[0039] Specifically, S305 calculates the mean square error (MSE) between the input sample and the reconstructed sample as the anomaly score, and sets the threshold τ. The formula is as follows:

[0040]

[0041] τ is the set threshold. When MSE > τ, it is determined as an abnormal image; when MSE ≤ τ, it is determined as a normal image.

[0042] Specifically, S4 is as follows:

[0043] a) Input the image of the interface to be tested into the trained variational autoencoder model; among them, the image to be monitored is obtained by taking screenshots of the mobile application interface through a Python automated monitoring program;

[0044] b) Obtain the reconstruction result output by the model;

[0045] c) Determine whether the image of the interface to be tested is normal or abnormal according to the comparison result between the calculated mean square error (MSE) and the set threshold τ.

[0046] The beneficial effects of the present invention are as follows: The present invention can still have a high accuracy rate even when there is not much sample data, and it reduces manual intervention and resource requirements, improves the efficiency of anomaly detection, and can judge interface anomalies on various mobile phone models, realizing cross-device detection and improving the versatility of anomaly detection. Description of the Drawings

[0047] Figure 1 It is a flowchart of Embodiment 3 of the present invention. Detailed Embodiments

[0048] To clearly illustrate the technical features of the present solution, the present solution will be described below through specific embodiments.

[0049] Embodiment 1

[0050] The embodiment of the present invention provides a method for identifying anomalies in a mobile application interface based on a variational autoencoder, including the following steps:

[0051] S1: Obtain the screenshot data of the mobile application interface and perform preprocessing to generate a training data set; among them,

[0052] The steps for obtaining the screenshot data of the mobile application interface include:

[0053] a) Use Python to run the APP automated test program, record the screenshots of each page of the mobile application program and save them;

[0054] b) Select the normal interface screenshots and record them as the image data set Dataset1, where Dataset1 = {x 1 , x 2 ,........., x n}, and x i represents a specific interface screenshot instance;

[0055] The preprocessing includes the following steps:

[0056] a) Adjust the interface image to 224×224 pixels;

[0057] b) Normalize the image pixel values to the range of [-1, 1];

[0058] c) Standardize the image so that it is transformed into a standard normal distribution;

[0059] d) Use data augmentation techniques to expand the dataset Dataset1 to obtain the augmented dataset Dataset2 = {x 1 ′, x 2 ′,....., x′ n}.

[0060] S2: Use a pre-trained feature extraction network to extract features from the pre-processed interface image to generate feature vectors; among them, the feature extraction steps include:

[0061] a) Use the pre-trained VGG network to extract features from the pre-processed image data;

[0062] b) Convert the image data into feature vectors {f 1 , f 2 ,..., f n}.

[0063] S3: Build and train a variational autoencoder model; the variational autoencoder model includes an encoder and a decoder; specifically, it includes the following steps:

[0064] S301: Use the encoder to map the feature vectors to the latent space to obtain the mean vector and variance vector of the latent variables;

[0065] S302: Use the decoder to process the latent variables to generate corresponding reconstructed features;

[0066] S303: Define the loss function of the VAE;

[0067] S304: Use the training dataset to minimize the loss function through an optimizer to train the variational autoencoder model;

[0068] S305: Set a threshold by analyzing the reconstruction error distribution of the training data to distinguish normal images from abnormal images.

[0069] Among them, the encoder maps the input feature vector f i to the latent space to generate the mean vector u i and the variance vector The formula is as follows:

[0070]

[0071] The decoder generates the reconstructed data according to the distribution of the latent variable z i as follows:

[0072] p(z i ) = N(z i | 0, I).

[0073] The loss function of the variational autoencoder model consists of the reconstruction error and the KL divergence, and the specific formula is as follows:

[0074]

[0075] where θ and are the parameters of the decoder and the encoder respectively.

[0076] Specifically, S305 calculates the mean square error (MSE) between the input sample and the reconstructed sample as the anomaly score, and sets a threshold τ, and the formula is as follows:

[0077]

[0078] τ is the set threshold. When MSE > τ, it is determined as an abnormal image; when MSE ≤ τ, it is determined as a normal image.

[0079] S4: Input the screenshot of the interface to be tested into the trained variational autoencoder model for anomaly detection and output the result; specifically, it includes the following steps:

[0080] a) Input the image of the interface to be tested into the trained variational autoencoder model; among them, the image to be monitored is obtained by taking screenshots of the mobile application interface through a Python automated monitoring program;

[0081] b) Obtain the reconstructed result output by the model;

[0082] c) Determine whether the image of the interface to be tested is normal or abnormal according to the comparison result between the calculated mean square error (MSE) and the set threshold τ.

[0083] Embodiment 2

[0084] The embodiment of the present invention provides a method for identifying anomalies in a mobile application interface based on a variational autoencoder (VAE), including:

[0085] S1: Obtain a large number of normal interfaces of the mobile application and take screenshots, perform annotation processing on the collected image data to generate a training data set, and use data augmentation technology to expand the data set to improve the generalization ability of the model.

[0086] S2: Use the pre-trained VGG network to extract features from the mobile application interface images, and convert the complex image data into low-dimensional feature vectors.

[0087] S3: Build a VAE model, including an encoder and a decoder. The encoder maps the input feature vector to the latent space to obtain the mean vector and variance vector; the decoder generates the reconstructed data according to the distribution of the latent variables. The VAE model is optimized by minimizing the reconstruction error and KL divergence.

[0088] S4: Calculate the difference between the input sample and the reconstructed sample, and use the reconstruction error as the anomaly scoring metric. The larger the reconstruction error, the more likely the sample is to be an anomaly. Set a threshold, and when the anomaly score exceeds this threshold, it is determined as an abnormal sample.

[0089] S5: Use the training set to train the VAE model and optimize the model parameters. After training, use the application interface screenshot data to evaluate the model and verify its anomaly detection performance.

[0090] Among them, the VAE model is trained through the following steps:

[0091] First, preprocess the data set, including converting the images into tensors and normalizing them within the range of [-1:1];

[0092] Among them, the VAE model consists of two parts: an encoder and a decoder. The encoder encodes the input data into the mean and variance of the latent variables, and the decoder reconstructs the input data from the latent variables.

[0093] Define the loss function of the VAE. The loss function consists of two parts: the reconstruction error and the KL divergence. The total loss is the sum of the reconstruction error and the KL divergence.

[0094] Use the optimizer and the defined loss function to train the VAE model. In each epoch, iterate over the data set, calculate the loss, and update the parameters in the model through backpropagation.

[0095] Example 3

[0096] Reference Figure 1 , this embodiment of the present invention provides a method for identifying anomalies in mobile application interfaces based on a variational autoencoder (VAE), including:

[0097] 1. Use Python to run the APP automated test program, record the screenshots of each page of the mobile application and save them, and select the interface screenshots that are normally displayed and record them as the image data set Dataset1, where Dataset1 = {x 1 , x 2 ,..., x n}, x iRepresents a specific interface screenshot example.

[0098] 2. Preprocess the collected image dataset Dataset1, which includes the following sub-steps:

[0099] 2a) Resize the interface image size to 224x224 pixels to fit the input image specifications of the VGG network;

[0100] 2b) Normalize the image pixels to be between [-1, 1];

[0101] 2c) Standardize the interface image and convert it to a standard normal distribution (Gaussian distribution) to make the model converge more easily;

[0102] 2d) Use data augmentation techniques to expand the data in Dataset1 to obtain the augmented dataset Dataset2 = {x′ 1 , x′ 2 ,..., x′ n}.

[0103] 3. Extract image features using the VGG network and convert the image data into feature vectors {f 1 , f 2 ,..., f n}.

[0104] 4. Use the encoder of the VAE model to map the feature vector f i to the latent space to obtain the mean vector u i and the variance vector The specific formula is as follows:

[0105]

[0106] The decoder generates reconstructed data according to the distribution of the latent variables:

[0107] p(z i ) = N(z i |0, I).

[0108] 5. The VAE loss function consists of the reconstruction error and the KL divergence. The specific formula is as follows:

[0109]

[0110] Among them, θ and are the parameters of the decoder and the encoder respectively.

[0111] 6. Abnormal score calculation:

[0112] Calculate the mean squared error (MSE) between the input sample and the reconstructed sample as the abnormal score:

[0113]

[0114] Set a threshold τ. When MSE > τ, it is determined as an abnormal image; when MSE ≤ τ, it is determined as a normal image.

[0115] Application example:

[0116] When monitoring the mobile application of government services in a certain province, by using the present invention, all pages of the application can be quickly detected periodically. The abnormal pages identified will be automatically stored in a folder, and the abnormal pages will be promptly fed back to the relevant departments and construction manufacturers, so as to discover and solve problems early. Compared with manual abnormal testing of mobile application pages, as shown in the following table, the present invention obviously has the advantages of a wide test range and a short test time.

[0117]

[0118]

[0119] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying abnormalities in a mobile application interface based on a variational autoencoder, the method comprising the following steps: S1: Obtain mobile application interface screenshot data and preprocess it to generate a training data set; S2: Use the pre-trained feature extraction network to extract features from the pre-processed interface image and generate a feature vector; S3: Build and train a variational autoencoder model; S4: Input the screenshot of the interface to be tested into the trained variational autoencoder model for anomaly detection and output the result.

2. The method according to claim 1, characterized in that The step S1 of obtaining the screenshot data of the mobile application interface specifically includes the following steps: a) Use Python to run the APP automated testing program, record and save screenshots of each page of the mobile application; b) Select normal interface screenshots and record them as image dataset Dataset1, where Dataset1 = {x1, x2, ..., x n }, x i Indicates a specific interface screenshot instance.

3. The method according to claim 1, characterized in that The preprocessing in S1 includes the following steps: a) Adjust the interface image to 224×224 pixels; b) Normalize the image pixel values ​​to the range of [-1,1]; c) Standardizing the image to convert it into a standard normal distribution; d) Use data enhancement technology to expand the dataset Dataset1 and obtain the enhanced dataset Dataset2 = {x′1, x′2, ..., x′ n }.

4. The method according to claim 1, characterized in that: The feature extraction step of S2 includes: a) Use the pre-trained VGG network to extract features from the pre-processed image data; b) Convert the image data into a feature vector {f1, f2, ..., f n }.

5. The method according to claim 1, characterized in that The variational autoencoder model in S3 includes an encoder and a decoder; S3 includes the following steps: S301: Use an encoder to map the feature vector to a latent space to obtain a mean vector and a variance vector of the latent variable; S302: using a decoder to process the latent variables to generate corresponding reconstruction features; S303: Define the loss function of VAE; S304: Using the training data set to minimize the loss function through the optimizer, the variational autoencoder model is trained; S305: By analyzing the reconstruction error distribution of the training data, a threshold is set to distinguish between normal images and abnormal images.

6. The method according to claim 5, characterized in that The encoder takes as input the feature vector f i Mapped to the latent space, generating the mean vector u i and variance vector The formula is as follows: The decoder is based on the latent variable z i The distribution of generates reconstructed data, the formula is as follows: p(z i )=N(z i |0,I)。 7. The method according to claim 5, characterized in that The loss function of the variational autoencoder model consists of reconstruction error and KL divergence, and the specific formula is as follows: Among them, θ and are the parameters of the decoder and encoder respectively.

8. The method according to claim 5, characterized in that Specifically, S305 calculates the mean square error (MSE) between the input sample and the reconstructed sample as an abnormality score and sets a threshold τ, and the formula is as follows: τ is the set threshold. When MSE>τ, it is judged as an abnormal image; when MSE≤τ, it is judged as a normal image.

9. The method according to any one of claims 1 to 8, characterized in that: The S4 is specifically: a) Inputting the image of the interface to be tested into the trained variational autoencoder model; b) obtaining the reconstruction result of the model output; c) According to the comparison result of the calculated mean square error (MSE) and the set threshold value τ, it is determined whether the image of the interface to be tested is normal or abnormal.

Images