An automated testing method based on multimodal fusion of text and images

Abstract

The present invention discloses an automated testing method based on multimodal fusion of text and image, comprising the following steps: acquiring image data of the entered interface through a camera; acquiring text data through a text detection and text recognition model; sending the image data and text data together to a multimodal model for processing, wherein the multimodal model includes a convolution layer and a maximum pooling layer for image data processing, and the image data is subjected to a convolution layer and a maximum pooling layer, and image modal features are extracted by Resnet after passing through the convolution layer and the maximum pooling layer, and the multimodal model also includes a convolutional neural network for text data processing to obtain text modal features; obtaining a label corresponding to the current image through the multimodal model, and judging whether the interface is correct. The present invention performs multimodal fusion of text modal features and image modal features through the multimodal model, so that the accuracy of judging whether the correct interface has been entered is higher during automated testing.

Description

Technical Field

[0001] The present invention relates to the technical field of automated testing, and in particular to an automated testing method based on multimodal fusion of text and images. Background Art

[0002] Testing is an essential component of a comprehensive R&D system. A project will eventually transition to a maintenance-focused state through rapid iteration. Introducing automated testing at the right time and in the right manner can effectively reduce manual maintenance costs. Automated testing is divided into internal and external testing. External testing primarily determines whether the interface achieves the expected results. When running automated tests, the front-end interface is used to determine whether the test steps are running as specified. If there are any interface errors, the test will automatically be considered a failure, and the problem can be identified and corrected.

[0003] In existing technologies, single-modality visual technology only uses images to determine the label corresponding to the interface. Due to the large number of similar interfaces in the interface images, the accuracy of single-modality visual technology is low;

[0004] To this end, we propose an automated testing method based on multimodal fusion of text and images to solve the above problems. Summary of the Invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0006] In view of the above problems existing in the existing automated testing method based on multimodal fusion of text and image, the present invention is proposed.

[0007] Therefore, the purpose of the present invention is to provide an automated testing method based on multimodal fusion of text and image, in order to improve the accuracy of interface automated testing.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] An automated testing method based on multimodal fusion of text and image, comprising the following steps:

[0010] Step 1: Obtain image data of the entered interface through the camera;

[0011] Step 2: Obtain text data through text detection and text recognition models;

[0012] Step 3: The image data and text data are fed into a multimodal model for processing. The multimodal model includes a convolutional layer and a maximum pooling layer for image data processing. After the image data passes through one convolution layer and one maximum pooling layer, image modality features are extracted by Resnet. The multimodal model also includes a convolutional neural network for text data processing to obtain text modality features.

[0013] Step 4: Obtain the label corresponding to the current image through the multimodal model and determine whether the interface is correct.

[0014] As a preferred solution of the automated testing method based on multimodal fusion of text and image described in the present invention, wherein: in the process of step three, Resnet50 is specifically selected for image feature extraction, and the image modal features extracted by Resnet50 and the text modal features obtained by convolutional neural network processing are sent to the Fusion Block fusion module to obtain the fused feature layer, and finally the classification results predicted by the fully connected layer Dense and Softmax function calculation model are obtained. Softmax converts the output values ​​of multiple classifications into relative probabilities, which are easier to understand and compare.

[0015] As a preferred solution of the automated testing method based on multimodal fusion of text and image described in the present invention, the image modal features extracted by the Resnet are set as Xi, the text modal features obtained by the convolutional network are set as Xt, and Xi and Xt are used as inputs of the Fusion Block fusion module. The features of the two modalities of image and text are then spliced ​​together by the fully connected layer Dense and concat splicing, and the tanh function is introduced. The low-level text modal features are then supplemented with the high-level features of the image using the add operation, thereby ensuring the integrity of the original structural features of the image modality. The calculation formula is as follows:

[0016] X tanh =tanh(concat(W i X i +b i ,W t X t ))

[0017] Output of Fusion Block:

[0018] X output =add(X tan□ *X i ,X i );

[0019] Among them, Wi and Wt are the weights of the image and text modalities after passing through the fully connected layer Dense, bi represents the bias, and tanh is the activation function.

[0020] As a preferred solution of the automated testing method based on multimodal fusion of text and image described in the present invention, the specific operation mode of step 2 is:

[0021] a. Feed the image data obtained in step 1 into the text detection model to obtain the coordinate data of the text in the image;

[0022] b. Crop the text image based on the coordinate data and feed the text image into the text recognition model to predict all the text data in the new image.

[0023] As a preferred solution of the automated testing method based on multimodal fusion of text and images described in the present invention, in order to ensure the accuracy of the model test, the multimodal model is trained, and the specific method is: collecting images and text data of each interface, annotating corresponding labels for each group of image and text data, and dividing them into training set, validation set and test set in an 8:1:1 ratio. The training set data is used for model training; the validation set data is used to verify the performance of the model during training to observe the model training effect; the test set data is used for the result evaluation of the final model.

[0024] As a preferred solution of the automated testing method based on multimodal fusion of text and image described in the present invention, the cross entropy loss function is used as the loss function during multimodal model training. In deep learning, cross entropy can be regarded as the difficulty of expressing probability distribution p(x) by probability distribution q(x), and its expression is:

[0025]

[0026] As a preferred solution of the automated testing method based on multimodal fusion of text and images described in the present invention, the prepared training set is cyclically sent to the multimodal model for training according to the batch size, the training is completed after n iterations, and the trained model structure and weights are saved.

[0027] Beneficial effects of the present invention:

[0028] 1. The present invention integrates text modal features and image modal features through a multimodal model, which makes it more accurate in determining whether the correct interface has been entered during automated testing.

[0029] 2. Resnet is selected as the backbone network for image modality feature extraction, and its residual structure is used to eliminate the problem of network degradation during training;

[0030] 3. The Tanh activation function is introduced in Fusion Block fusion, which not only makes the neural network nonlinear, but also alleviates the problem of gradient vanishing during back propagation to a certain extent;

[0031] 4. The self-attention mechanism of the transformer is used in Fusion Block fusion to help the text modality pay more attention to features that have a greater impact on the result weight;

[0032] 5. The multimodal network designed by the present invention improves the recognition accuracy compared to the method that relies only on a single image modality, and the model size is only increased by 1.6M. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0034] Figure 1 This is a flow chart of an automated testing method based on multimodal fusion of text and images proposed by the present invention. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] Example 1

[0038] Reference Figure 1 , which is a first embodiment of the present invention, provides an automated testing method based on multimodal fusion of text and image, the method comprising the following steps:

[0039] Step 1: Obtain image data of the entered interface through the camera;

[0040] Step 2: Acquire text data through text detection and text recognition models. The specific acquisition method is as follows: feed the image data obtained in step 1 into the text detection model to obtain the coordinate data of the text in the image; crop the text image based on the coordinate data, and feed the text image into the text recognition model to predict all the text data in the new image;

[0041] Step 3: The image data and text data are fed into a multimodal model for processing. The multimodal model includes a convolutional layer and a maximum pooling layer for image data processing. After the image data undergoes one convolution and one maximum pooling layer, a Resnet layer is used to extract image modal features. The multimodal model also includes a convolutional neural network for text data processing to obtain text modal features. Specifically, Resnet50 is used for image feature extraction. The image modal features extracted by Resnet50 and the text modal features obtained by the convolutional neural network are fed into a fusion block to obtain a fused feature layer. Finally, the model predicts the classification result through a fully connected layer and a Softmax function. Softmax converts the output values ​​of multiple classifications into relative probabilities, which is easier to understand and compare. The softmax function is also called a normalized exponential function. Purpose: Convert classification results within the real number range --> to probabilities between 0 and 1; use the properties of exponentials to map real numbers to 0-positive infinity (non-negative); and use normalization methods to convert results of 1. to probabilities between 0 and 1.

[0042] The core operation of the fully connected Dense layer described above is the matrix-vector product, which essentially linearly transforms one feature space into another. Therefore, the purpose of the Dense layer is to transform the previously extracted features nonlinearly in the Dense layer, extract the relationships between these features, and finally map them to the output space.

[0043] Step 4: Obtain the label corresponding to the current image through the multimodal model and determine whether the interface is correct.

[0044] It should be noted that Resnet is a CNN network structure. According to the different network depths, Resnet is divided into Resnet18, Resnet50, Resnet101, etc. This paper selects Resnet50 for image feature extraction, so only the output of the last Resnet Block is taken, that is, the last average pooling layer and the fully connected layer of the original network are discarded.

[0045] In addition, in step three, the image modality features extracted by the Resnet are set to Xi, and the text modality features obtained by the convolutional network are set to Xt, and Xi and Xt are used as the input of the Fusion Block fusion module. The transformer's self-attention (self-attention mechanism) is used to help the text modality pay more attention to the features that have a greater impact on the result weight. Then, the features of the two modalities of image and text are spliced ​​together by the fully connected layer Dense and concat splicing, and the tanh function is introduced. The add operation is then used to supplement the low-level text modality features to the high-level features of the image. For two inputs, if the number of channels is the same and there is a convolution behind it, add is equivalent to the corresponding channels sharing the same convolution kernel after concat splicing, which ensures the integrity of the original structural features of the image modality. The calculation formula is as follows:

[0046] X tanh =tanh(concat(W i X i +b i ,W t X t ))

[0047] Output of Fusion Block

[0048] X output =add(X tan□ *X i ,X i );

[0049] Among them, Wi and Wt are the weights of the image and text modalities after passing through the fully connected layer Dense, bi represents the bias, and tanh is the activation function.

[0050] In order to ensure the accuracy of the multimodal model test, the multimodal model is trained. The specific method is as follows: collect images and text data of each interface, annotate corresponding labels for each set of image and text data, and divide them into training set, validation set and test set according to 8:1:1. The training set data is used for model training; the validation set data is used to verify the performance of the model during training to observe the model training effect; the test set data is used to evaluate the results of the final model, and the cross entropy loss function is used as the loss function for multimodal model training. In deep learning, cross entropy can be regarded as the difficulty of expressing the probability distribution p(x) through the probability distribution q(x). Its expression is:

[0051]

[0052] Specifically, the prepared training set is cyclically fed into the multimodal model for training according to the batch size, the training is completed after n iterations, and the trained model structure and weights are saved.

[0053] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An automated testing method based on multimodal fusion of text and images, characterized by: The following steps are involved: Step 1: Obtain image data of the entered interface through the camera; Step 2: Obtain text data through text detection and text recognition models; Step 3: The image data and text data are fed into a multimodal model for processing. The multimodal model includes a convolutional layer and a maximum pooling layer for image data processing. After the image data passes through one convolution layer and one maximum pooling layer, image modality features are extracted by Resnet. The multimodal model also includes a convolutional neural network for text data processing to obtain text modality features. Step 4: Obtain the label corresponding to the current image through the multimodal model and determine whether the interface is correct; The image modality features extracted by the Resnet are set as Xi, and the text modality features obtained by the convolutional neural network are set as Xt. Xi and Xt are used as the input of the Fusion Block fusion module. The features of the two modalities of image and text are then spliced ​​together by the fully connected layer Dense and concat splicing. The tanh function is introduced, and the add operation is used to supplement the low-level text modality features to the high-level features of the image, thereby ensuring the integrity of the original structural features of the image modality. The calculation formula is as follows: Output of Fusion Block: Among them, Wi and Wt are the weights of the image and text modalities after passing through the fully connected layer Dense, bi represents the bias, and tanh is the activation function.

2. The automated testing method based on multimodal fusion of text and images according to claim 1, characterized in that: In the step three, Resnet50 is specifically selected for image feature extraction, and the image modal features extracted by Resnet50 and the text modal features obtained by convolutional neural network processing are sent to the Fusion Block fusion module to obtain the fused feature layer. Finally, the classification results predicted by the model are calculated through the fully connected layer Dense and the Softmax function. The Softmax function converts the output values ​​of multiple classifications into relative probabilities, which is easier to understand and compare.

3. The automated testing method based on multimodal fusion of text and images according to claim 2, characterized in that: The specific operation method of step 2 is: a. Feed the image data obtained in step 1 into the text detection model to obtain the coordinate data of the text in the image; b. Crop the text image based on the coordinate data and feed the text image into the text recognition model to predict all the text data in the new image.

4. The automated testing method based on multimodal fusion of text and images according to any one of claims 1 to 3, characterized in that: In order to ensure the accuracy of the model test, the multimodal model is trained in the following way: collect images and text data of each interface, annotate each set of image and text data with corresponding labels, and divide them into training set, validation set and test set in the ratio of 8:1:

1. The training set data is used for model training; the validation set data is used to verify the performance of the model during training and observe the model training effect; the test set data is used for the evaluation of the final model results.

5. The automated testing method based on multimodal fusion of text and images according to claim 4, characterized in that: The cross entropy loss function is used as the loss function for multimodal model training. In deep learning, cross entropy can be regarded as the difficulty of expressing the probability distribution p(x) through the probability distribution q(x). Its expression is:

6. The automated testing method based on multimodal fusion of text and images according to claim 5, characterized in that: The prepared training set is fed into the multimodal model for training in a cyclic manner according to the batch size. The training is completed after n iterations, and the trained model structure and weights are saved.

Images