Part anomaly detection method based on vision and text prompt

Abstract

The invention discloses a part anomaly detection method based on vision and text prompt, and belongs to the technical field of part detection, and the method comprises the steps: S1, obtaining the vision and text prompt data of a part; s2, calculating visual features and text features of the parts by using a visual-text pre-training model according to the visual and text prompt data in the step S1; s3, fusing the visual features and the text features of the parts by adopting a multi-modal fusion module to obtain multi-modal fusion features; s4, calculating an anomaly detection result according to the multi-modal fusion features; s5, distilling the multi-modal fusion feature information to a visual branch; s6, inputting the image of the part into a visual branch to calculate an anomaly detection result; according to the part anomaly detection method based on vision and text prompt, the robustness and generalization of the vision-language pre-training model are utilized, and a good anomaly detection effect can be achieved when limited part image samples and text description serve as training data.

Description

Technical Field

[0001] The present invention relates to the technical field of component detection, and in particular to a component anomaly detection method based on vision and text prompts. Background Art

[0002] Railway transportation occupies a relatively important position in the field of transportation due to its advantages such as large transportation volume, speed, safety and high reliability.

[0003] As the core part of railway transportation, trains may have abnormal body parts during operation, which seriously threatens the safe operation of trains. Therefore, it is very important to detect abnormal train parts to ensure the safe operation of trains.

[0004] At present, the method of detecting abnormal parts based on machine vision mainly trains neural networks to model the characteristic distribution of normal samples. During detection, the difference between the test sample and the normal distribution is used to determine whether the sample is abnormal. However, this method requires a large number of diverse normal images. In actual production, it is costly to collect a large number of diverse normal images. Moreover, normal images and abnormal images can only give examples of visual normality and abnormality, but sometimes the definition of normality and abnormality needs to be defined according to the context. Therefore, using text information describing normality or abnormality can help improve the accuracy of abnormality detection. Summary of the invention

[0005] The purpose of the present invention is to provide a component anomaly detection method based on visual and text prompts to solve the problems existing in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides a method for detecting component anomalies based on visual and text prompts, comprising the following steps:

[0007] S1. Obtain visual and textual prompt data of parts;

[0008] S2, using the visual and textual prompt data in step S1 to calculate the visual features and textual features of the parts using the visual-textual pre-training model;

[0009] S3, using a multimodal fusion module to fuse the visual features and text features of the parts to obtain multimodal fusion features;

[0010] S4, calculating anomaly detection results based on multimodal fusion features;

[0011] S5, distill the multimodal fusion feature information to the visual branch;

[0012] S6. Input the image of the component into the visual branch to calculate the anomaly detection result.

[0013] Preferably, in step S1, the visual data of the component is a color image of the component to be detected; and the text prompt data is a normal or abnormal description corresponding to the color image of the component to be detected.

[0014] Preferably, the visual-text pre-trained model of step S2 is a model pre-trained on a large number of image-text pairs, the input of the model is visual and text prompt data, and the output is visual features and text features, specifically:

[0015] Input the color image and text prompt data of the parts to be detected into the visual-text pre-training model to obtain the visual feature H∈R n×c and text features S∈R m×e , where c is the number of visual feature channels, m is the sequence length of text features, and e is the number of text feature channels. The two-dimensional image features are expanded into one dimension to obtain the visual feature H∈R n×c , where n=wh is the size of the visual feature.

[0016] Preferably, the multimodal fusion module in step S3 fuses the visual features and text features obtained in step S2 through an attention mechanism to obtain a multimodal fusion feature, specifically, fuses the visual features and text features through a bilinear attention mechanism, and the attention weight is calculated as follows:

[0017]

[0018] Among them, U∈R n×h ,V∈R m×h is the weight matrix, b is the bias, 1∈R h is a vector of all ones, h is a hyperparameter, and h≤(min(n,m)), H i ∈R c is the i-th visual feature vector, S j ∈R e is the jth text feature vector, T represents transposition;

[0019] The final attention weight is obtained through softmax:

[0020] α=softmax(a)∈R h×w ;

[0021] Among them, a={a ij},i=1,…,h,j=1,…,w;

[0022] The visual features and text features are weighted using bilinear attention weights to obtain the multimodal fusion feature f m ∈R o , which is calculated as:

[0023]

[0024] Among them, o is the dimension of multimodal fusion features, U'∈R n×d , V'∈R m×d , P∈R d×o is a learnable parameter, d is a hyperparameter, d≤min(n,m).

[0025] Preferably, the anomaly detection result of step S4 is to input the multimodal fusion features obtained in step S3 into a classifier composed of multiple layers of fully connected layers. The probability of getting an abnormality p θ , M is the number of categories, and M = 2, and its calculation method is:

[0026] p θ =c(f m );

[0027] Among them, θ is the parameter of the classifier;

[0028] The cross entropy loss function is used to train the multimodal fusion module and classifier, and its calculation formula is:

[0029]

[0030] Where N is the number of samples, y i is the true label, which is 1 if the input component is abnormal, otherwise it is 0.

[0031] Preferably, the visual branch in step S5 is a visual branch independent of the multimodal fusion module of the visual-text pre-trained model machine, and the visual branch includes an image encoder composed of a convolutional neural network and a classifier. The image of the part to be detected is input into the visual branch, and the multimodal fusion feature information is distilled to the visual branch as the classification result of the multimodal fusion model. Specifically, the Kullback-Leibler divergence of the output result of the multimodal fusion module and the output result of the visual branch is used as the distillation loss function of the visual branch, and the calculation method is:

[0032]

[0033] Among them, θ s are the parameters of the visual branch, is the probability of component abnormality.

[0034] Preferably, the abnormality detection result of step S6 is that the image of the component to be detected is input into the visual branch, and the classification result is obtained by a classifier composed of multiple layers of fully connected layers. Therefore, the loss function of the visual branch is the Kullback-Leibler divergence plus the cross entropy loss function, which is calculated as follows:

[0035]

[0036] Among them, L CE is the cross entropy loss, λ is a custom hyperparameter, λ=0.5.

[0037] The visual branch only requires images as input, which allows the model to be extended to scenarios where text information is missing. Distilling the multimodal fusion feature information to the visual branch enables the visual branch to inherit the performance of the multimodal fusion module, thereby improving the accuracy of component anomaly detection.

[0038] Therefore, the present invention adopts the above-mentioned component anomaly detection method based on visual and text prompts, which has the following beneficial effects:

[0039] (1) By leveraging the robustness and generalization of the vision-language pre-trained model, good anomaly detection results can be achieved when limited component image samples and text descriptions are used as training data;

[0040] (2) Make full use of the additional information provided by the text to improve the accuracy of component anomaly detection.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a specific flow chart of a method for detecting component anomalies based on visual and text prompts according to the present invention. DETAILED DESCRIPTION

[0043] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 , a method for detecting component anomalies based on vision and textual cues, comprising the following steps:

[0045] S1. Obtain visual and text prompt data of parts; wherein the visual data is a color image of the parts to be inspected; and the text prompt data is a normal or abnormal description corresponding to the color image of the parts to be inspected.

[0046] S2, using the visual and textual prompt data in step S1, and using the visual-textual pre-trained model to calculate the visual features and textual features of the parts; the visual-textual pre-trained model is a model pre-trained on a large number of image-text pairs, the input of the model is the visual and textual prompt data, and the output is the visual features and textual features, specifically:

[0047] Input the color image and text prompt data of the parts to be detected into the visual-text pre-training model to obtain the visual feature H∈R n×c and text features S∈R m×e , where c is the number of visual feature channels, m is the sequence length of text features, and e is the number of text feature channels. The two-dimensional image features are expanded into one dimension to obtain the visual feature H∈R n×c , where n=wh is the size of the visual feature.

[0048] S3, using a multimodal fusion module to fuse the visual features and text features of the parts to obtain a multimodal fusion feature; the multimodal fusion module fuses the visual features and text features obtained in step S2 through an attention mechanism to obtain a multimodal fusion feature, specifically, the visual features and text features are fused through a bilinear attention mechanism, and the attention weight is calculated as follows:

[0049]

[0050] Among them, U∈R n×h ,V∈R m×h is the weight matrix, b is the bias, 1∈R h is a vector of all ones, h is a hyperparameter, and h≤(min(n,m)), H i ∈R c is the i-th visual feature vector, S j ∈R e is the jth text feature vector, T represents transposition;

[0051] The final attention weight is obtained through softmax:

[0052] α=softmax(a)∈R h×w ;

[0053] Among them, a={a ij},i=1,…,h,j=1,…,w;

[0054] The visual features and text features are weighted using bilinear attention weights to obtain the multimodal fusion feature f m ∈R o , which is calculated as:

[0055]

[0056] Among them, o is the dimension of multimodal fusion features, U'∈R n×d , V'∈R m×d , P∈R d×o is a learnable parameter, d is a hyperparameter, d≤min(n,m).

[0057] S4, calculating the anomaly detection result based on the multimodal fusion features; the anomaly detection result is obtained by inputting the multimodal fusion features obtained in step S3 into a classifier composed of multiple layers of fully connected layers. The probability of getting an abnormality p θ , M is the number of categories, and M = 2, and its calculation method is:

[0058] p θ =c(f m );

[0059] Among them, θ is the parameter of the classifier;

[0060] The cross entropy loss function is used to train the multimodal fusion module and classifier, and its calculation formula is:

[0061]

[0062] Where N is the number of samples, y i is the true label, which is 1 if the input component is abnormal, otherwise it is 0.

[0063] S5, distilling the multimodal fusion feature information to the visual branch; the visual branch is a visual branch independent of the multimodal fusion module of the visual-text pre-trained model machine, the visual branch includes an image encoder composed of a convolutional neural network and a classifier, the image of the part to be detected is input into the visual branch, and the multimodal fusion feature information is distilled to the visual branch as the classification result of the multimodal fusion model, specifically, the Kullback-Leibler divergence of the output result of the multimodal fusion module and the output result of the visual branch is used as the distillation loss function of the visual branch, and the calculation method is:

[0064]

[0065] Among them, θ s are the parameters of the visual branch, is the probability of component abnormality.

[0066] S6. Input the image of the component into the visual branch to calculate the anomaly detection result; the anomaly detection result is obtained by inputting the image of the component to be detected into the visual branch and passing through a classifier composed of multiple layers of fully connected layers to obtain the classification result. Therefore, the loss function of the visual branch is the Kullback-Leibler divergence plus the cross entropy loss function, which is calculated as follows:

[0067]

[0068] Among them, L CE is the cross entropy loss, λ is a custom hyperparameter, λ=0.5.

[0069] The visual branch only requires images as input, which allows the model to be extended to scenarios where text information is missing. Distilling the multimodal fusion feature information to the visual branch enables the visual branch to inherit the performance of the multimodal fusion module, thereby improving the accuracy of component anomaly detection.

[0070] Therefore, the present invention adopts the above-mentioned component anomaly detection method based on visual and textual cues, and utilizes the robustness and generalization of the visual-language pre-training model to achieve good anomaly detection effects when limited component image samples and textual descriptions are used as training data, and fully utilizes the additional information provided by the text to improve the accuracy of component anomaly detection.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A component anomaly detection method based on visual and textual prompts, characterized in that: The following steps are involved: S1. Obtain visual and textual prompt data of parts; S2, using the visual and textual prompt data in step S1 to calculate the visual features and textual features of the parts using the visual-textual pre-training model; S3, using a multimodal fusion module to fuse the visual features and text features of the parts to obtain multimodal fusion features; S4, calculating anomaly detection results based on multimodal fusion features; S5, distill the multimodal fusion feature information to the visual branch; S6. Input the image of the component into the visual branch to calculate the anomaly detection result.

2. The method for detecting component anomalies based on visual and textual prompts according to claim 1, characterized in that: In step S1, the visual data of the component is to collect the color image of the component to be detected; the text prompt data is to collect the normal or abnormal description corresponding to the color image of the component to be detected.

3. The method for detecting component anomalies based on vision and text prompts according to claim 1, characterized in that: The visual-text pre-trained model of step S2 is a model pre-trained on a large number of image-text pairs, specifically: Input the color image and text prompt data of the parts to be detected into the visual-text pre-training model to obtain the visual feature H∈R n×c and text features S∈R m×e , where c is the number of visual feature channels, m is the sequence length of text features, e is the number of channels of text features, and n=wh is the size of visual features.

4. The method for detecting component anomalies based on visual and textual prompts according to claim 3, characterized in that: Step S3 specifically integrates visual features and text features through a bilinear attention mechanism, and the attention weight is calculated as follows: Among them, U∈R n×h ,V∈R m×h is the weight matrix, b is the bias, 1∈R h is a vector of all ones, h is a hyperparameter, and h≤(min(n,m)), H i ∈R c is the i-th visual feature vector, S j ∈R e is the jth text feature vector, T represents transposition; The final attention weight is obtained through softmax: α=softmax(a)∈R h×w ; Wherein, a = {a ij }, i = 1, ..., h, j = 1, ..., w; The visual features and text features are weighted using bilinear attention weights to obtain the multimodal fusion feature f m ∈R o , which is calculated as: Among them, o is the dimension of multimodal fusion features, U'∈R n×d , V'∈R m×d , P∈R d×o is a learnable parameter, d is a hyperparameter, d≤min(n,m).

5. The method for detecting component anomalies based on visual and textual prompts according to claim 4, characterized in that: The anomaly detection result of step S4 is obtained by inputting the multimodal fusion features obtained in step S3 into a classifier composed of multiple layers of fully connected layers. The probability of getting an abnormality p θ , M is the number of categories, and M = 2, and its calculation method is: p θ =c(f m ); Among them, θ is the parameter of the classifier; The cross entropy loss function is used to train the multimodal fusion module and classifier, and its calculation formula is: Where N is the number of samples, y i is the true label, which is 1 if the input component is abnormal, otherwise it is 0.

6. A method for detecting component anomalies based on visual and textual prompts according to claim 5, characterized in that: The visual branch in step S5 is a visual branch independent of the multimodal fusion module of the visual-text pre-trained model machine. The visual branch includes an image encoder composed of a convolutional neural network and a classifier. The image of the part to be detected is input into the visual branch, and the multimodal fusion feature information is distilled to the visual branch as the classification result of the multimodal fusion model. Specifically, the Kullback-Leibler divergence between the output result of the multimodal fusion module and the output result of the visual branch is used as the distillation loss function of the visual branch, which is calculated as follows: Among them, θ s are the parameters of the visual branch, is the probability of component abnormality.

7. The method for detecting abnormal parts based on vision and text prompts according to claim 6, characterized in that: The abnormality detection result of step S6 is that the image of the component to be detected is input into the visual branch, and the classification result is obtained through a classifier composed of multiple layers of fully connected layers. Therefore, the loss function of the visual branch is the Kullback-Leibler divergence plus the cross entropy loss function, which is calculated as follows: Among them, L CE is the cross entropy loss, λ is a custom hyperparameter, λ=0.5.

Images