GUI (Graphical User Interface) anomaly detection method based on multi-modal model

By designing a multimodal model architecture to integrate text and image modal features, and using BERT and DyHead models for transfer learning, the problem of UI bug detection relies on a large number of labeled data in the existing technology is solved, and efficient identification on small sample data sets is achieved The ability of GUI exceptions.

CN119065965APending Publication Date: 2024-12-03NANJING UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411064418.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing UI bug detection methods rely on a large amount of labeled data and a high-cost training process, making it difficult to deal with diversified and dynamically changing GUI interfaces, and lack small sample and zero sample capabilities, so they cannot effectively identify GUI exceptions.

Method used

Using the GUI anomaly detection method based on multimodal model, the multimodal model architecture is designed to integrate text and image modal features, and transfer learning is used to combine DSL to generate a small data set for fine-tuning of the model to achieve new Recognition of types of UI exceptions.

Benefits of technology

It reduces the cost of data volume and model training time, improves the effectiveness and robustness of GUI anomaly detection, and can achieve good detection effects on small sample data sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065965A_ABST
    Figure CN119065965A_ABST
Patent Text Reader

Abstract

The invention discloses a GUI anomaly detection method based on a multi-modal model, and the method comprises the steps: S1, designing a multi-modal model architecture which is used for fusing the features of a text mode and an image mode; s2, enabling the model to learn matching of different types of components and characters on a normal UI data set, and image-text matching capability of some common types of UI anomalies, so that the model has a feed shot capability; s3, performing feed-shot model fine tuning on a small amount of data set generated based on the exception defined by a user by using DSL, so that the model has the capability of identifying a new type of UI exception; and S4, performing UI anomaly detection by using character prompt and a multi-modal model. According to the method, how to construct a multi-modal model with zero sample and small sample migration capabilities and how to use the multi-modal model to perform GUI anomaly detection are included, and a good GUI anomaly detection effect can be achieved only by finely adjusting the model on a small sample data set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical fields of software testing, transfer learning and multimodal models, and in particular to a GUI anomaly detection method based on a multimodal model. Background Art

[0002] UI (User Interface) is the medium for information exchange between people and computers or other devices. The quality of the UI interface directly affects the user experience and satisfaction. Therefore, testing and evaluating the UI interface is an indispensable part of the software development process.

[0003] UI Bugs refer to errors or defects in the UI interface, such as overlapping, missing, deformed, or misplaced controls, which can reduce the aesthetics and functionality of the UI interface and even prevent users from using the software normally. In order to effectively detect and fix UI Bugs, a large and diverse set of abnormal UI datasets is required as data resources for training and testing.

[0004] DSL (Domain Specific Language) is a language designed for a specific domain or problem, with the characteristics of simplicity, efficiency, and easy extensibility. DSL can be used to describe the concepts, rules, operations, etc. involved in a certain domain or problem, and the DSL statements can be converted into executable codes or instructions through a parser.

[0005] Most of the current UI Bug detection methods rely on a large amount of labeled data and training processes. Each abnormality identification requires high-cost data and training, which limits their generalization ability and adaptability, and makes it difficult to cope with diverse and dynamically changing GUI interfaces. This is because the current deep learning models are similar to CNN or the target recognition model Faster-RCNN, which are all aimed at learning features in images. The model's capabilities are limited to the learned features. Even for common types and features, training needs to be started from scratch, and there is almost no way to apply the existing model capabilities. Although there are many subdivisions of GUI element abnormalities, there is a strong commonality between the types. For example, although the text overlap and icon overlap types of abnormalities are different in terms of widget types, they have the same visual features.

[0006] A multimodal model is a model that can process information from multiple modalities, such as images, text, and speech, and can interact and fuse between different modalities. The advantage of a multimodal model is that it can utilize the complementarity and redundancy of information from different modalities to improve the expressiveness and robustness of the model. The small sample and zero sample capabilities mean that it can use pre-trained knowledge or additional information to achieve reasoning and learning of new categories or new tasks with only a small amount or no labeled data. Summary of the invention

[0007] Most of the existing UI Bug detection methods rely on a large amount of labeled data and training processes. Each identification of an anomaly requires high-cost data and training. The present invention provides a GUI anomaly detection method based on a multimodal model. The method includes how to build a multimodal model with zero-sample and small-sample migration capabilities and how to use the multimodal model for GUI anomaly detection. The present invention only needs to fine-tune the model on a small sample data set to achieve a good GUI anomaly detection effect. The present invention is implemented through the following technical solutions.

[0008] A GUI anomaly detection method based on a multimodal model, characterized in that the method comprises the following steps:

[0009] S1, design a multimodal model architecture to fuse the features of text and image modalities;

[0010] S2, on the normal UI dataset, allows the model to learn the matching of different types of components and texts, and the image-text matching capabilities of some common types of UI anomalies, so that the model has a few shot capability;

[0011] S3, fine-tuning the model with few-shots based on a small dataset of exceptions defined by users using the DSL, so that the model has the ability to identify new types of UI exceptions;

[0012] S4, detects UI anomalies using textual hints and multimodal models.

[0013] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that the multimodal model architecture in step S1 is a multimodal model architecture based on the GLIP model architecture with strong transfer learning capabilities, and is designed for four modules, including a text encoder, an image encoder, a DeepFusion module and a loss function.

[0014] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that the text encoder uses BERT, specifically a Bert-based model, and the Bert-based model includes 12 layers of BERT Layer; the image encoder uses DyHead, which is responsible for receiving the input of the image and encoding the image into corresponding features; the DeepFusion module is used to fuse the information of the two dimensions of image and text, and BERT Layer and DyHead Module are selected as the modal feature fusion layer; the feature output of the backbone network of the two modalities is obtained. 0 and P 0 At the beginning, the output of each layer will be passed through the cross-modality multi-head attention module to calculate the characteristics of bimodal communication The calculation formula is:

[0015]

[0016] Among them, the subscript t2i represents text to image, the subscript i2t represents image to text, X-MHA is a multi-modal multi-head attention module, and L is the number of DyHead modules in the model; Used to calculate the output feature O of the i+1th layer image modality and text modality backbone network i+1 , P i+1 , the relevant formula is:

[0017]

[0018]

[0019] Among them, the multimodal multi-head attention module X-MHA is calculated according to the following steps:

[0020] Q o =OW (q,I) ,Q p =PW (q,L) ,

[0021] V p =PW (v,L) ,C t2i =SoftMax(Attn)V p W (out,I)

[0022] V o =OW (v,I) ,C i2t =SoftMax(AttnΤ )V o W (out,L)

[0023] Among them, {W (symbol,I) ,W (symbol,L) |symbol∈{q,v,out}} is a trainable parameter. When symbol=q,v,out, it is used as the weight matrix corresponding to query, value and output respectively. O and P are the input vectors of image mode and text mode respectively, that is, O i and P i .

[0024] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that, in the process of Cross-Attention, the image modality input vector O and the text modality input vector P are multiplied by their respective query weight matrices W (q ,I) and W (q,L) Get their respective query matrices, denoted as Q o and Q p , calculate the Attn intermediate matrix by the product of the two queries, and then multiply the input vector of the other modality by the other value weight matrix to obtain the other value matrix V o and V p , then multiply the value matrix and out matrix weights of the other mode by the Attn intermediate matrix after SoftMax to obtain the communication feature vector.

[0025] The above-mentioned GUI anomaly detection method based on the multimodal model is characterized in that the calculation formula of the loss function l is:

[0026] O out =Enc I (Image),P out =Enc L (Prompt),

[0027] l = loss (S ground ; T)

[0028] Among them, Enc I and Enc L Represent image encoder and text encoder respectively, is the final image feature vector obtained by the image encoder, is the final text feature vector obtained by the text encoder; T∈{0,1} N×M , where element t ijWhen it is 1, it means that the image area i matches the text content j; element t ij When it is 0, it means no match; loss(S ground ; T) is a typical cross entropy loss calculation method.

[0029] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that the normal UI data set in step S2 is a set of triples, and each element in the set is composed of an image, a text, and a label.

[0030] The above-mentioned GUI anomaly detection method based on multimodal model is characterized in that the step S2 specifically includes:

[0031] S21, based on the physical image dataset (Object365 and GoldG), adds a normal GUI component dataset and a representative abnormal GUI dataset;

[0032] S22, dividing the GUI data set into a training set, a test set, and a validation set, and performing training after setting parameters.

[0033] The above-mentioned GUI anomaly detection method based on multimodal model is characterized in that step S3 specifically includes:

[0034] S31, select five types of anomalies: text overlap, text occlusion, image missing, text null value, and fuzzy, and design a domain-specific language (DSL) to formally describe them;

[0035] S32, using DSL statements to generate an abnormal data set, and fine-tuning the model based on the generated abnormal data set;

[0036] S33, freeze the weights of the last two layers of the backbone network during fine-tuning;

[0037] S34, in the X-shot process, the data set is randomly sampled so that there are at least X samples in each category;

[0038] S35, change the random seed and perform three independent runs for each X-shot experiment, and select the one with the highest accuracy on the test set as the final result.

[0039] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that the UI anomaly detection process in step S4 requires input of two modes, which include a text mode and a picture mode.

[0040] The above-mentioned GUI anomaly detection method based on a multimodal model is characterized in that, for the input of the text modality, a concatenation of text descriptions of one or more types of anomalies is used as input, and the text modality is passed through a text encoder to obtain corresponding text features; for the input of the picture modality, the picture modality is passed through an image encoder to obtain the image features corresponding to each proposal box, and the dot product matrix of the image features of each region and all the text features is calculated. The picture text greater than a threshold in the matrix can be regarded as a correct match, that is, the classification of each region.

[0041] The present invention adopts the above technical solution and has the following beneficial effects:

[0042] The present invention combines the information of visual modality and textual modality, and has better detection effect on fully learned GUI anomaly types than the target detection model method. At the same time, the present invention can use the learned model capabilities to perform small sample migration, reducing the data volume requirements for a type of anomaly detection and the cost of model training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The figure is an overall architecture diagram of a GUI anomaly detection method based on a multimodal model according to an embodiment of the present invention.

[0044] Figure 2 It is a schematic diagram of the framework of the GLIP model architecture according to an embodiment of the present invention.

[0045] Figure 3 It is a structural schematic diagram of a file encoder module according to an embodiment of the present invention.

[0046] Figure 4 FIG. 4 is a schematic diagram of an image of the final result of anomaly detection according to an embodiment of the present invention.

[0047] Figure 5 A schematic diagram of the result of inputting a multimodal model to perform UI anomaly detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The embodiment of the present invention provides a GUI anomaly detection method based on a multimodal model. Figure 1 The overall structure of the method is as follows: Figure 1 As shown, this method pre-trains a multimodal model with certain migration capabilities based on a public data set, and fine-tunes the model based on the abnormal data set generated by DSL, so that it has the ability to detect and locate GUI abnormalities. The method steps of this embodiment include:

[0050] S1, design a multimodal model architecture to integrate the features of text modality and image modality; the specific analysis is as follows:

[0051] This method refers to the GLIP model architecture, which was jointly proposed by the UCLA and Microsoft Research teams. The GLIP model structure is as follows: Figure 2 As shown, in this embodiment, the method of the present invention mainly designs four modules, and the four modules are a text encoder, an image encoder, a DeepFusion module and a loss function.

[0052] The text encoder module is responsible for receiving the input of the text prompt and abstracting the text into the corresponding TextFeatures. This method uses BERT as the text encoder. Specifically, the Bert-based model is selected, which contains 12 layers of BERT Layer. The BERT architecture is actually a multi-layer Transformer encoder module stacking structure. The Transformer is a seq2seq model, which itself consists of an encoder and a decoder. The encoder consists of a Multi-Head Attention layer and a fully connected feedforward network layer. The encoder structure is as follows Figure 3 As shown in the figure, each Transformer Encoder, namely BERT Layer, consists of two parts, one of which is a residual network composed of a multi-head attention mechanism layer, and the other is a residual network composed of a fully connected feedforward network layer. Unlike other language models, BERT uses three embeddings, namely Token Embedding, Segment Embedding, and Position Embedding. The Token Embeddings convert each input word into a word vector, which is the simplest way to distinguish each word itself and is also the most commonly used embedding method. Segment Embedding converts sentences into sentence vectors by judging the similarity between them. BERT can complete the classification task between sentence pairs, and Segment Embedding itself serves to judge the similarity between sentence pairs. Finally, Position Embedding adds position information to each word, because the same word in different positions between sentences will also affect its semantics.

[0053] The image encoder module is responsible for receiving the input of the image and encoding the image into corresponding features. This method selects DyHead as the image encoder. The DyHead is a framework that uses the attention mechanism to unify the target detection head. It combines self-attention in three different attention aspects: scale perception, spatial perception, and task perception. The DyHead model used in this method adopts a stacked structure of six DyHead blocks. In the subsequent Deep fusion structure, each DyHead module is a DyHead block.

[0054] The DeepFusion module is used to fuse information from two dimensions: image and text. This method uses BERTLayer and DyHead Module as the layers for modal feature fusion. 0 and O 0 To begin with, as shown in Formula 1, the output of each layer will be passed through the cross-modality multi-head attention module (X-MHA) to calculate the characteristics of bimodal communication (Where t2i represents text to image, i2t represents image to text, and L represents the number of DyHead modules in the model. In the actual implementation, L=2). In the feature update process of the next layer, the encoders of the two modalities will carry the communication features of the other modality, that is, as shown in Formula 2 and Formula 3, the output of the communication modality is added to the output value of the previous layer of the modality. This calculation method of deep fusion between modal features will make the features of the two modalities better matched and integrated. It not only improves the overall performance of the phrase grounding task, but also makes the model's image reasoning ability have better text combination ability, which can help transfer learning to perform better.

[0055]

[0056]

[0057]

[0058] In the calculation process of the cross-modal multi-head attention module X-MHA, each Head calculates the context vector output of the modality by adding another modality. The calculation process is shown in Formula 4-Formula 6. (symbol,I) ,W (symbol ,L)|symbol∈{q,v,out}} is a trainable parameter. When symbol=q,v,out, it serves as the weight matrix corresponding to query, value and output respectively. O and P are also the previous image feature and text feature inputs. In Cross-Attention, the two vectors are multiplied by their respective query weight matrices as the Q matrix, and the Attn intermediate matrix is ​​calculated as the product of the two Q matrices. Then the vector of the other modality is multiplied by the corresponding value weight matrix as the V matrix. Finally, the V matrix of the other modality is multiplied by the corresponding out matrix weight, and multiplied by the Attn intermediate matrix after SoftMax as the output of the interactive feature vector.

[0059]

[0060] V p =PW (v,L) ,C t2i =SoftMax(Attn)V p W (out,I) (5)

[0061] V o =OW (v,I) ,C i2t =SoftMax(Attn Τ )V o W (out,L) (6)

[0062] In target detection of non-multimodal models, the loss function is usually defined by the classification loss L cls and the localization loss L loc (Formula 7). In general, the positioning loss is determined by the network of the proposed region (usually RPN), so here we only consider L cls .

[0063] L= L loc + L cls (7)

[0064] O out =Enc I (Image),S cls =OW Τ ,L cls =loss(S cls ;T) (8)

[0065] In formula 8, It is the image region feature after passing through the image encoder, is the weight matrix of the region classifier C, is the number of classification logits after the image features pass through the classifier (hereinafter referred to as logits), T∈{0,1} N×c is the target match between regions and classes computed from classic many-to-1 matching or binary Hungarian matching. loss(S cls ; T) is a typical cross-entropy loss calculation method. In order to make the phrase grounding task unify the detection task and the grounding task, GLIP proposes that each region can be grounded to c phrases in the text prompt instead of being classified into c categories. For example, for animal category classification, all animal categories can be connected into a prompt: "Detection: cat, dog,..., elephant", and the category is determined by calculating which token in the phrase has the closest feature to a region. After changing the detection task to the grounding task, the original loss calculation is changed to the following for S ground Calculation method:

[0066]

[0067] in is the text feature obtained by the text encoder. The region-word matching score is a Matrix. And the target matrix T′∈{0,1} N×M Referring to the definition method of contrastive learning, in contrastive learning, only positive samples and negative samples need to be specified. In the grounding task, when the area matches the text token, it is judged as a positive sample (the training data set is a data set with box annotations, and this target matrix can be directly generated). At this time, the calculated loss function becomes loss (S ground ; T).

[0068] S2, pre-training multimodal models on normal data sets and common abnormal data sets; the specific analysis is as follows:

[0069] After designing the model architecture, in order to enable the model to detect GUI element anomalies through text prompts using a very small number of samples or zero-sample learning, model pre-training is required first.

[0070] The ability of zero-shot or small-shot transfer learning comes from the model’s ability to derive image features from text. If the target label domain contains all GUI element anomaly types W×B, then the model needs to learn at least Only by matching the text and image features can the detection capability of the entire set of element anomalies be migrated.

[0071] The model pre-trained only on the Objects365 and GoldG datasets (both of which are physical image datasets) does not have the ability to recognize GUI components, let alone the ability to recognize GUI component anomalies. In order to increase the GUI-related knowledge of the pre-trained model, this method adds two types of datasets: one is the dataset of normal GUI component types, and the other is a representative anomaly dataset selected from GUI anomaly data.

[0072] This method first divides the GUI dataset into a training set, a test set, and a validation set, and then sets the parameters for training; then selects five types of anomalies, namely text overlap, text occlusion, image missing, text null value (missing), and fuzzy, and uses a DSL-based method to generate the corresponding anomaly dataset, and further trains the model based on the generated anomaly dataset. The specific steps include:

[0073] S21, construct a common data set and train the model.

[0074] The GLIP team proposed to use a teacher model on a manually annotated dataset to predict image text data obtained on the web, thereby obtaining a 27M text-image dataset (3M manually annotated, 24M pseudo-labeled). However, due to limitations in training equipment and time, the present invention did not use the 27M dataset, but used Objects365+GoldG (3M manually annotated data provided by the GLIP team) as a common dataset. Based on this dataset, we trained and obtained model M0.

[0075] S22, construct normal and abnormal UI datasets to further train the model.

[0076] The VINS dataset (a GUI dataset) is selected to construct normal and abnormal UI datasets. When processing the VINS dataset, the superclass attribute of each category is assigned to APPWidget, and the category name is set to the type of the component itself. Here, 21 component types are extracted from the VINS dataset: {CheckBox, Text, Toolbar, Drawer, Image, CheckedTextView, Modal, Card, Map, Bottom_Navigation, Spinner, UpperTaskBar, PageIndicator, EditText, TextButton, Checkbox, Switch, Icon, BackgroundImage, Multi_Tab, Remember}. It is divided into training set, test set and validation set in a ratio of 8:1:1, with 3846, 480 and 480 GUI screenshots respectively, and epoch=12, batch_size=4, and learning_rate of BERT backbone network=1×10 -5 , other parameters learning_rate = 1×10 -4 , train model M0 on Tesla V100-32GB and get M1.

[0077] Use the following DSL to define five exceptions: text overlap, text occlusion, image missing, text null value (missing), and blur:

[0078] text overlap: Text keep full nochange[X1+20,Y 1+20,X2+20,Y2+20]

[0079] text occlusion: Text nokeep full occlusion[X1,Y1,X2,Y2]

[0080] text null: Text nokeep focus null[X1*2 / 3+X2 / 3,Y1,X1*2 / 3+X2 / 3+5*(Y2-Y1) / 4,Y2]

[0081] Blur: UpperTaskBarkeep

[0082] full distortion[X1,Y1,X2*2 / 3+X1 / 3,Y2*2 / 3+Y1 / 3]&

[0083] full distortion[X1+20,Y1+20,X2*2 / 3+X1 / 3+20,Y2*2 / 3+Y1 / 3+20]&

[0084] full distortion[X1+40,Y 1+40,X2*2 / 3+X1 / 3+40,Y2*2 / 3+Y1 / 3+40]&

[0085] full distortion[X1+60,Y 1+60,X2*2 / 3+X1 / 3+60,Y2*2 / 3+Y1 / 3+60]

[0086] missing image:Image nokeep full imageloaderror[X1,Y1,X2,Y2]

[0087] For each type of anomaly, a 1k GUI anomaly image dataset is generated. On average, two anomalies of each type are generated on each GUI image. When generating dataset annotations, the super class of each type of anomaly is set to "APP widget element bug", and the category name is the bug name defined in DSL. This anomaly dataset is used to further train model M1 and obtain M pretrained ; The training parameters are the same as those when training with the normal dataset. At this point, the model should have a certain transfer learning ability on GUI element anomalies.

[0088] S3, using real data sets or small sample data sets automatically generated by DSL, fine-tuning the multimodal model; the specific analysis is as follows:

[0089] When fine-tuning, set epoch = 12, batch_size = 4, and learning_rate of the BERT backbone network = 1×10 -5 , other parameters learning_rate = 1×10 -4 , weight decay is 0.05, freeze the weights of the last two layers of the backbone network. Perform 1-shot, 3-shot, 5-shot, 10-shot, 50-shot, 100-shot and 200-shot training for each type (N-shot means there are N training samples in the fine-tuning training set) and randomly sample the dataset during the X-shot process so that there are at least X samples in each category. Change the random seed (and thus the sampling data) and run each X-shot experiment 3 times independently, and select the one with the highest accuracy on the test set as the final result.

[0090] S4, construct a sample consisting of an image and a text prompt, and input it into the model for detection; the specific analysis is as follows:

[0091] Unlike the target detection model that directly classifies proposal boxes, the multimodal model classification is based on calculating the similarity between text features and regional image features. This is similar to the model training process, so in the detection and classification process, two modal inputs are also required.

[0092] For a type of anomaly, we have two ways to detect anomaly: fully conduct supervised learning on this type, or use a small sample data set to transfer training and then perform detection. In the model inference stage, we will use the connection of superclass and category_name as the text prompt.

[0093] For text modal input, what is needed is the concatenation of texts of all types of interest. For example, for target detection in an animal-type dataset, the input should be text input of the animal type such as "animals:cats;dogs;tigers..." For GUI anomalies, the concatenation of text descriptions of one or more types of anomalies can be used as input, and the corresponding text features can be obtained through the text encoder. In the same detection task, the calculated features are often fixed and do not need to be calculated multiple times. Using manually written prompts will have better reasoning effects. For example, if the superclass is "APPwidget" and the category_name is "element bug:icon overlap", the text prompt is "APP widgetelement bug:icon overlap". After experiments, if "two icon covers eachother" is added after the description and added to the concatenation of all detection targets, the AP of icon overlap category UI anomaly positioning can be increased by 1.2.

[0094] For each image input, it is also passed through the image encoder to obtain the image features corresponding to each proposal box, and the dot product matrix of the image features of each region and all the text features is calculated. The image text greater than the threshold in the matrix can be regarded as a correct match, that is, the classification of each region; Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram of an image of the final result of abnormality detection according to an embodiment of the present invention; Figure 5 As shown, Figure 5 This is a schematic diagram of the result of inputting a multimodal model to perform UI anomaly detection in an embodiment of the present invention. Figure 5In the result matrix on the right, the threshold is 0.5, then there are three proposal boxes whose features match the input type.

[0095] The above is only a preferred embodiment of the present invention, but the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Any simple modification, equivalent changes and modifications made to the above embodiments by any technician familiar with the profession according to the technical essence of the present invention without departing from the principle and spirit of the present invention shall be included in the protection scope of the present invention.

Claims

1. A GUI anomaly detection method based on a multimodal model, characterized in that: The method comprises the following steps: S1, design a multimodal model architecture to fuse the features of text and image modalities; S2, on the normal UI dataset, allows the model to learn the matching of different types of components and texts, and the image-text matching capabilities of some common types of UI anomalies, so that the model has a few shot capability; S3, fine-tuning the model with few-shots based on a small dataset of exceptions defined by users using the DSL, so that the model has the ability to identify new types of UI exceptions; S4, detects UI anomalies using textual hints and multimodal models.

2. The GUI anomaly detection method based on a multimodal model according to claim 1, characterized in that: The multimodal model architecture in step S1 is a multimodal model architecture based on the GLIP model architecture with strong transfer learning capabilities, and is designed for four modules, including a text encoder, an image encoder, a DeepFusion module and a loss function.

3. The GUI anomaly detection method based on a multimodal model according to claim 1 or 2, characterized in that: The text encoder uses BERT, specifically a Bert-based model, which includes 12 BERT layers; the image encoder uses DyHead, which is responsible for receiving image input and encoding the image into corresponding features; the DeepFusion module is used to fuse information in two dimensions, image and text, and uses BERT Layer and DyHead Module as modal feature fusion layers; From the feature output O of the backbone network of the two modalities 0 and P 0 At the beginning, the output of each layer will be passed through the cross-modality multi-head attention module to calculate the characteristics of bimodal communication The calculation formula is: Among them, the subscript t2i represents text to image, the subscript i2t represents image to text, X-MHA is a multi-modal multi-head attention module, and L is the number of DyHead modules in the model; Used to calculate the output feature O of the i+1th layer image modality and text modality backbone network i+1 , P i+1 , the relevant formula is: Among them, the multimodal multi-head attention module X-MHA is calculated according to the following steps: Q o =OW (q,I) ,Q p =PW (q,L) , V p =PW (v,L) ,C t2i =SoftMax(Attn)V p W (out,I) V o =OW (v,I) ,C i2t =SoftMax(Attn Τ )V o W (out,L) Among them, {W (symbol,I) ,W (symbol,L) |symbol∈{q,v,out}} is a trainable parameter. When symbol=q,v,out, it is used as the weight matrix corresponding to query, value and output respectively. O and P are the input vectors of image mode and text mode respectively, that is, O i and P i .

4. The GUI anomaly detection method based on a multimodal model according to claim 3, characterized in that: In the process of Cross-Attention, the image modality input vector O and the text modality input vector P are multiplied by their respective query weight matrices W (q,I) and W (q,L) Get their respective query matrices, denoted as Q o and Q p , calculate the Attn intermediate matrix by the product of the two queries, and then multiply the input vector of the other modality by the other value weight matrix to obtain the other value matrix V o and V p , then multiply the value matrix and out matrix weights of the other mode by the Attn intermediate matrix after SoftMax to obtain the communication feature vector.

5. The GUI anomaly detection method based on a multimodal model according to claim 2, characterized in that: The calculation formula of the loss function l is: O out =Enc I (Image),P out =Enc L (Prompt), l=loss(S ground ;T) Among them, Enc I and Enc L Represent image encoder and text encoder respectively, is the final image feature vector obtained by the image encoder, is the final text feature vector obtained by the text encoder; T∈{0,1} N×M , where element t ij When it is 1, it means that the image area i matches the text content j; element t ij When it is 0, it means no match; loss(S ground ; T) is a typical cross entropy loss calculation method.

6. The GUI anomaly detection method based on a multimodal model according to claim 1, characterized in that: The normal UI data set in step S2 is a set of triplets, each element of which consists of an image, a text and a label.

7. The GUI anomaly detection method based on a multimodal model according to claim 1, characterized in that: The step S2 specifically includes: S21, based on the physical image dataset (Object365 and GoldG), adds a normal GUI component dataset and a representative abnormal GUI dataset; S22, dividing the GUI data set into a training set, a test set, and a validation set, and performing training after setting parameters.

8. The GUI anomaly detection method based on a multimodal model according to claim 1, characterized in that: The step S3 specifically includes: S31, select five types of anomalies: text overlap, text occlusion, image missing, text null value, and fuzzy, and design a domain-specific language (DSL) to formally describe them; S32, using DSL statements to generate an abnormal data set, and fine-tuning the model based on the generated abnormal data set; S33, freeze the weights of the last two layers of the backbone network during fine-tuning; S34, in the X-shot process, the data set is randomly sampled so that there are at least X samples in each category; S35, change the random seed and perform three independent runs for each X-shot experiment, and select the one with the highest accuracy on the test set as the final result.

9. The GUI anomaly detection method based on a multimodal model according to claim 1, characterized in that: The process of detecting UI anomalies in step S4 requires input of two modes, including text mode and picture mode.

10. The GUI anomaly detection method based on a multimodal model according to claim 9, characterized in that: For the input of the text modality, one or more types of abnormal text descriptions are concatenated as input, and the text modality is passed through a text encoder to obtain the corresponding text features. For the input of the picture modality, the picture modality is passed through an image encoder to obtain the image features corresponding to each proposal box, and the dot product matrix of the image features of each region and all the text features is calculated. The picture text greater than the threshold in the matrix can be regarded as a correct match, that is, the classification of each region.

Citation Information

Cited By

  • Self-repairing UI automatic detection method based on multi-modal fusion

    CN120234251A