Apparent defect detection method and system based on multi-modal large model

Abstract

The invention discloses an apparent defect detection method and system based on a multi-modal large model. The method comprises the following steps: firstly, designing cue words, and constructing a defect image-mask-text description data set by utilizing GPT-4V to automatically label defect images; secondly, utilizing the data set to finely adjust a pre-trained big language backbone network, so that the big language backbone network can better understand a defect detection task; secondly, designing a multi-vision scale token group mechanism according to the visual features of each layer of the encoder, and outputting a defect detection result by using a lightweight mask decoder; finally, in combination with character description output of the large language backbone network, a defect detection algorithm combining the defect detection mask and defect character description is designed, and the defect detection accuracy in the industrial production environment is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and system for detecting surface defects based on a multimodal large model. Background Art

[0002] In an industrial production environment, surface defect detection is particularly important to ensure product quality and yield. In actual application scenarios, it is far from enough to detect defects. Factory managers hope to monitor the production of products in real time based on the results of defect detection in order to correct errors and adjust production methods in a timely manner. However, most of the defect detection results are now evaluated by experienced engineers. This manual evaluation method is not only time-consuming and labor-intensive, but also affects accuracy and credibility due to the engineers' changing energy and work status, which seriously affects the factory's production efficiency. Therefore, it is extremely important to explore a defect detection and automatic evaluation mechanism based on artificial intelligence technology.

[0003] In recent years, with the proposal and development of large language backbone networks, algorithms based on multimodal large models have been widely used in image segmentation, image classification, target detection, visual question answering and other fields, and have been proven by many research works to have excellent application effects. With the powerful reasoning ability and zero-sample ability of the large language backbone network, the multimodal large model can not only understand the input of multiple modalities, such as text modality, image modality, mask modality, etc., but also output the detection and evaluation results of multiple modalities that users want through reasoning. The existing deep learning-related cutting-edge defect detection technologies can be roughly divided into two methods: 1) feature embedding-based methods; 2) image reconstruction-based methods. The former relies on a feature extractor pre-trained with a large amount of data to extract image features, and then uses a custom standard and nearest neighbor strategy to measure the distance between normal data and abnormal data to obtain the detection result. The latter relies on the generative model to learn the data distribution of normal images, reconstruct the input abnormal image into a normal image, and then compare the difference between the two to obtain the detection result.

[0004] Both detection methods have disadvantages that cannot be ignored: 1) The method based on feature embedding is subject to the quality of feature extraction. If the feature extractor is not properly trained or not adapted to the domain, it will seriously affect the effect of defect detection. In addition, in high-dimensional space, sample sparsity will cause the distance metric to fail, affecting the identification of outliers. It is worth noting that in actual industrial production environments, the defect area of ​​products is often very small, which means that in high-dimensional space, the method based on feature embedding may not be able to accurately identify outliers and cause misjudgment, thereby allowing defective products to leave the factory and enter the market. The economic and reputational losses caused by this are difficult to measure; 2) The method based on image reconstruction is subject to the quality of the selected generative model. If the generative model cannot effectively capture the distribution of normal data, the reconstruction error may be inaccurate, resulting in poor defect detection results. In addition, since the data distribution of defective images cannot be observed during training, the reconstruction quality of the defective area is relatively large during reasoning, which affects the defect detection quality. What is more noteworthy is that the current defect detection field focuses more on identifying and outputting visual inspection results, and lacks textual description and detailed evaluation of defective parts. The description and detailed evaluation of defective parts still require experienced engineers. The lack of textual description and detailed evaluation affects the decision-making efficiency and adjustment speed of production work. Summary of the invention

[0005] In order to overcome the above-mentioned shortcomings, the present invention provides a method and system for surface defect detection based on a multimodal large model, which greatly empowers the defect detection link in the industrial production environment and improves production efficiency by utilizing a defect detection method that combines defect detection mask and defect text description.

[0006] The surface defect detection method based on the multimodal large model designed by the present invention comprises the following steps: Annotating a training data set, wherein the training data set includes defect images and corresponding masks and text descriptions; Encode the defect image and assign L×N tokens to the encoder, where L represents the number of visual scales and N represents the number of tokens assigned to each visual scale; Align the encoded visual features to the language feature space; Using a large language backbone network, the text description of the defect image, the multi-scale token group and the visual features aligned to the language feature space are input, and the language type token and the visual type token are obtained after processing. The language type token is decoded into the text description and evaluation of the defect image; Align visual type tokens to visual feature space; Decode the encoded visual features and aligned visual type tokens to obtain the final detection mask image; The multimodal large model consisting of the encoder, language feature alignment with visual features, large language backbone network, visual type token alignment with language features and decoder is trained, and defect detection is performed using the trained multimodal large model.

[0007] Furthermore, the text description in the training data set includes the type, shape, size of the defect, the cause of the defect and the repair suggestion for the defect.

[0008] Furthermore, GPT-4V is used to generate labeled training datasets.

[0009] Furthermore, the alignment of the visual features to the language features and the alignment of the language features to the visual features all adopt the same MLP structure, which specifically includes a linear layer, a ReLU layer, a linear layer and a Dropout layer.

[0010] Furthermore, the encoder adopts a pre-trained CLIP-ViT encoder; the large language backbone network adopts a pre-trained Vicuna-v1.5-13B, and the encoder and the large language backbone network are in a frozen state during the training of the multimodal large model, and the parameters are not updated.

[0011] Furthermore, the decoder has a total of L layers, each of which adopts a self-attention layer structure of a multi-dimensional convolutional head transposed attention and a gated convolutional feedforward network.

[0012] Furthermore, when training a large multimodal model, the following loss is used: = +

[0013] in, Represents autoregressive cross entropy loss to ensure the quality of text generation; represents the pixel-by-pixel binary cross entropy loss to ensure the quality of mask generation; and is a hyperparameter.

[0014] Based on the same inventive concept, the present invention also discloses a system for an apparent defect detection method based on a multi-modal large model, comprising: Vision Encoder : Encode the input defect image and extract visual features. The encoder is assigned L×N tokens, where L represents the number of visual scales and N represents the number of tokens assigned to each visual scale; Vision-Language Mapper : Align the encoded visual features to the language feature space; Large language backbone network F: receives the text description of the defect image, the multi-scale token group and the visual features aligned to the language feature space as input, obtains the language type token and the visual type token after processing, and directly decodes the language type token into the text description and evaluation of the defect image; Language-Visual Mapper : Align visual type tokens to visual feature space; Visual Decoder : Decode the encoded visual features and the aligned visual type tokens, and output the final detection mask.

[0015] Based on the same inventive concept, the method also discloses an electronic device, including: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement a surface defect detection method based on a multi-modal large model.

[0016] Based on the same inventive concept, the method also discloses a computer-readable medium on which a computer program is stored. When the program is executed by a processor, a surface defect detection method based on a multi-modal large model is implemented.

[0017] The advantages of the present invention are: 1. This patent innovatively introduces an automatic defect assessment mechanism in the defect detection task by utilizing a multimodal large model, which greatly optimizes the adjustment efficiency of the production method, reduces management decision-making costs, and improves production efficiency.

[0018] 2. This patent innovatively designs a multi-scale token group mechanism to decouple the visual features output by the encoder, optimize the quality of mask generation, and improve the accuracy of defect detection.

[0019] 3. This patent innovatively designs a lightweight visual decoder, which reduces the time complexity of the mask generation process from O( ) is reduced to O( ), which greatly accelerates the training and reasoning process of the entire model and improves the efficiency of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall architecture diagram of the present invention.

[0021] Figure 2 It is a detailed structural diagram of the visual decoder part of the present invention. DETAILED DESCRIPTION

[0022] In order to facilitate ordinary technicians in this field to understand and implement the patent of the present invention, the patent of the present invention is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the patent of the present invention, and are not used to limit the patent of the present invention.

[0023] Terminology explanation: GPT-4V: A large multimodal model for visual tasks developed by openAI based on the GPT-4 model.

[0024] CLIP-ViT encoder: It consists of CLIP and VIT (Vision Transformer). CLIP (Contrastive Language–Image Pre-training) is a model proposed by OpenAI, which includes an image encoder, a text encoder, and a contrastive training method. ViT is a transformer architecture suitable for computer vision tasks. ViT is similar to the Transformer that processes text. It uses a self-attention mechanism to capture local and global features of the image. CLIP-ViT can understand and process images and text at the same time, and is suitable for tasks such as image retrieval and text description generation. Zero-shot learning: CLIP-ViT can be directly used for a variety of vision and language tasks without retraining. This makes it more versatile when facing new tasks.

[0025] Vicuna-v1.5-13B: Vicuna is based on the transformer architecture and contains multiple layers of self-attention mechanisms, which can understand the context of the input text. This design makes it perform well in dialogue generation and natural language processing tasks. Vicuna can generate dialogue texts with natural and fluent responses, which is suitable for chatbots, customer service systems, etc. Text understanding and generation: It can complete a series of natural language processing tasks such as text summarization, question and answer, and text generation. Control and customization: It supports adjusting the response style according to user needs, such as requiring a more formal or more relaxed language style.

[0026] Token: refers to the small units into which text is decomposed during text processing. These units can be words, subwords, characters, or other text elements. In natural language processing, models usually cannot process raw text directly, so they must first be decomposed into a form suitable for model operations. These decomposed units are tokens.

[0027] The time complexity is : The time complexity of an algorithm in the computer field is square. : stands for "asymptotic upper bound", which is used to describe the performance of an algorithm, such as the growth rate of time or space complexity. It gives an upper limit on the running time or space requirements of the algorithm when the input size increases infinitely. n: stands for the size or size of the input data. It can be any measure. Usually in algorithm analysis, n usually represents the number of elements of the data to be processed. For example, in an array or list, n can represent the length of the array.

[0028] Gated-Dconv Feed-Forward Network (GDFN): It consists of multiple convolutional layers and is used to extract global features and perform denoising. It integrates multiple layers of features to enhance detail recovery capabilities.

[0029] Multi-Dconv Head Transposed Attention (MDTA): It consists of multiple convolutional layers and a self-attention calculation mechanism. It calculates the cross-covariance across channels to generate an attention map that implicitly encodes the global context. It functionally captures long-distance dependencies, thereby improving the feature expression ability and information transfer effect in the image generation process.

[0030] Embodiment 1 This embodiment provides a method for detecting surface defects based on a multi-modal large model. Figure 1 As shown, the specific implementation method includes the following steps: Step S1, labeling a training data set to train the surface defect detection model established in the method.

[0031] S1-1, design a prompt word to prompt GPT-4V how to generate data. In this embodiment, the prompt word is: "Suppose you are now an experienced defect detection engineer. I will give you a picture. Please carefully examine this picture and answer the following three questions according to the requirements I give: 1. Describe the defect type, defect shape, and defect size in this picture. 2. Analyze the cause of this defect. 3. Give your suggestions for repairing this defect. Requirements: Each paragraph of your answer should correspond to the corresponding question, and your answer should be as detailed as possible." This prompt word can not only allow GPT-4V to draw an analysis and description of the defect through reasoning, but also provide reference suggestions for repairing this defect; S1-2, call GPT-4V API to implement batch annotation of data. The basic data set contains a large number of defect images and their masks. All defect images in the basic data set and the prompt words in S1-1 are input into GPT-4V, and GPT-4V will output a text description and evaluation of this defect image. Examples of text description and evaluation are shown below: 1. Defect type, defect shape, defect size This image shows an asphalt road surface. Some fine cracks and uneven texture areas can be observed, which may be minor crack defects. These cracks are not very obvious, are irregularly distributed, and are narrow in width. Roughly estimated, these cracks are about 1-2 mm wide and several centimeters long.

[0032] 2. Analysis of the causes of defects This fine crack defect may be caused by the following reasons: Natural aging: As time goes by, pavement materials are affected by natural factors such as weathering, UV radiation, and temperature changes, and gradually become brittle and prone to cracks.

[0033] Temperature changes: Alternating between hot and cold will cause asphalt to expand and contract, resulting in thermal expansion and contraction, which will cause cracks.

[0034] Load effect: Under the action of long-term vehicle rolling, the road surface material will be fatigued and damaged, especially when heavy vehicles pass frequently, it is more likely to form cracks.

[0035] 3. Repair suggestions To repair this type of hairline crack, the following measures are recommended: Surface sealing: A layer of asphalt sealant can be applied to the pavement to fill cracks and prevent moisture from seeping in, thereby slowing the expansion of cracks.

[0036] Preventive maintenance: It is recommended to check the pavement condition regularly and repair early cracks in time to prevent further expansion of cracks.

[0037] Material improvement: If cracks occur frequently on this road section, you can consider using more crack-resistant asphalt materials or adding anti-cracking agents when repaving to improve the durability of the material.

[0038] After labeling each defect image, a data set containing three types of data, defect image, text description, and mask, can be obtained; Step S2, designing a mapper.

[0039] Designing a visual-linguistic mapper and language-visual mappers , both mappers use the same MLP structure, which consists of: linear layer → ReLU → linear layer → Dropout. For the vision-language mapper In general, the main function of the first layer of linear transformation is to linearly map the input visual features, adjust the dimension and distribution of the features, and make them suitable for subsequent processing. This process can be understood as feature selection and preliminary transformation of visual features in order to align them with the text space. The introduction of the ReLU activation function is mainly to introduce nonlinear factors so that the model can capture complex feature relationships. It truncates the input and only retains the positive part, thereby improving the expressiveness of the model so that the visual features can be better nonlinearly aligned with the features of the text space. The second linear layer further linearly maps the features processed by the activation function to map the features to the dimensions of the target text space. Through this layer, the model can transform the nonlinearly transformed visual features into representations that are more consistent with the text features, so that the visual information can be effectively converted into a form that can be understood by the text. Dropout randomly discards the output of a part of the neurons during training to prevent the model from overfitting. Although it does not directly participate in the process of feature alignment, through this regularization method, Dropout can improve the generalization ability of the model, making the generated alignment features more robust on different samples; for language-visual mappers The functions of each layer are similar to those mentioned above, so I will not go into details here. The role of the two mappers in the entire model is formally described in mathematical language as follows: = , represents the input image All visual features output after being encoded by the visual encoder E, and They represent the visual features output by each layer of E. In particular, It is the feature output by the last layer of E. Used to align visual features into language feature space: = ( )(1) in is the feature after alignment to the language feature space. The language-visual mapper The role of is to transform the The embedding form of is aligned to the visual feature space that the decoder D can understand: = ( )(2) in, for The features after alignment to the visual feature space. ={ ,...,}, For the visual type token output by the large language backbone network, l The layer uses the tokens fused by the linear layer.

[0040] Step S3, design a multi-scale token group mechanism to enhance defect detection results.

[0041] In the process of visual encoder E extracting image features layer by layer, existing research has confirmed that the features extracted by the shallower layers of E contain more information on texture, geometry, edges, and corners, and the deeper layers contain more high-dimensional semantic information. In the field of defect detection, to accurately detect the location and shape of defects, it is necessary to comprehensively consider texture geometry information and high-dimensional semantic information. The prior art only allocates one token to the last output layer of E, so that information of different visual scales is tightly coupled in one token, and geometric information and high-dimensional semantic information are mixed up. The present invention proposes a multi-scale token group mechanism, which not only allocates one token to the last layer output of E, but also allocates N tokens to each layer of E. There are L layers in E, and L×N tokens are allocated to represent visual features. The L×N tokens are divided into L groups with each N token as a group, and each group has different visual feature granularity. At the beginning of training, L×N tokens are randomly initialized to represent the visual features of a defect image at different scales. During the process of calculating the loss and back propagating, the model will learn the token features that carry visual information and add them to the large language backbone network vocabulary (LLM vocabulary); S3-2, use mathematical language to formally explain the multi-scale token group, as follows: like E( ), = With L layers, for the Visual characteristics of layers , all tokens assigned to it are ={ }. The layer has N tokens in total, and N tokens are regarded as a token group, denoted as . L token groups are composed of ={ }. Finally, , and input text prompt Input into the large language backbone network: =F( , , )(3) Output of the large language backbone network In the above example, the language type token is directly decoded into the final defect text description and evaluation, and the remaining visual type token is recorded as = , the definitions of N and L remain unchanged, s has L groups, each with N tokens. After alignment, it is input into the decoder D.

[0042] Step S4, design visual decoder

[0043] As attached Figure 2 As shown, the decoder D is used to output the final defect detection mask. It consists of L attention blocks, each of which processes The features output by each layer and the token group corresponding to that layer. Each attention block will output an attention score map, and the attention score map is weighted and summed to obtain the final defect detection mask. The internal structure of each attention block is the same, using the structure of multi-dimensional convolution head transposed attention MDTA + gated convolution feedforward network GDFN, referred to as MDTA + GDFN.

[0044] Since the processed images come from the actual production environment, the image clarity may be high, and the system has high requirements for processing speed and efficiency, so a lightweight decoder with fast inference rate is needed. Traditional methods mostly use SAM (Segment Anything Model) decoders. This invention adopts a Transformer-like structure decoder architecture. When the traditional Transformer architecture processes images, the computational overhead mainly comes from the self-attention layer, and the time complexity of processing images is Therefore, the traditional self-attention layer is inefficient in processing high-definition images, which will cause great time consumption during training and reasoning. The present invention uses the structure of the self-attention layer of MDTA+GDFN, and the time complexity of calculating the self-attention score is . It greatly reduces the computational consumption and improves the efficiency of defect detection. Specifically, the decoder D has a total of L layers, each of which is composed of MDTA+GDFN, where MDTA calculates the self-attention score with linear complexity, and GDFN ensures that useful information can be further transmitted and can eliminate useless information.

[0045] Describe the decoder formally in mathematical language The specific functions are as follows: Output visual tokens on a large language backbone network = Afterwards, for the All outputs of the group { }, use a linear layer Linear to merge all tokens into one token: = Linear( )(4) The fused token is used to generate the language-visual encoder. Aligning to the visual feature space gives ={ }, after which it will be passed into the decoder D together with the visual features encoded by the visual encoder to obtain the mask mask: mask =D( , )(5) Specifically, the operations performed inside D are: = (6) = (7) in is an element-by-element multiplication operation, is the sigmoid operation, The decoder An attention block consisting of MDTA and GDFN. The attention values ​​calculated for each attention block are used to calculate the final detection mask through weighted sum operation: mask = (8) in It is learned as a weight factor during training.

[0046] Step S5, training a multimodal large model consisting of an encoder, language feature alignment with visual features, a large language backbone network, visual feature alignment with language features, and a decoder.

[0047] In order to make the generated data conform to the format of visual question answering and fine-tune the instructions. The dialogue format during training is fixed as: "User: <image> Please detect and analyze the image I gave you, and give the defect detection mask and corresponding detailed evaluation of this image. Multimodal large model: {text} <mask>". After fine-tuning the instructions using data in this format, the large language backbone network can show strong instruction-following ability and solve a variety of downstream tasks through zero-shot learning. The loss of {text} will be calculated together with the text description in the dataset, denoted as The loss is calculated by comparing the mask with the ground truth mask in the dataset in step S1, denoted as , using these two loss functions to optimize the parameters to achieve the purpose of training the model. The entire combined network model is trained in an end-to-end manner. During the back propagation of the gradient, the trainable part of the model is gradually optimized. In this embodiment, LoRA technology is used to fine-tune the pre-trained large language backbone network. The trainable parts during training are: vision-language mapper , LoRA parameters in the large language backbone network F, language-vision mapper , visual decoder Parameters of the attention block, weight factors .

[0048] The definition of the loss function is formally explained in mathematical language as follows: To ensure the quality of text generation, autoregressive cross entropy loss is used as : = - (9) Where Y={ } is the real text label sequence in the dataset, It is the conditional probability given by the large language backbone network at time step t, that is, the probability of predicting the label at the current moment based on the known true text label sequence Y. The predicted probability of each time step is evaluated and accumulated by the corresponding probability of the true label. This method enables the model to learn how to generate new outputs based on previous output information during training. In order to ensure the quality of mask generation, is defined as follows: = (10) in represents the pixel-by-pixel binary cross entropy loss, H and W represent the height and width of the mask respectively. represents the binary value of the i-th pixel in the predicted mask image, represents the ground truth mask in the dataset. The total loss is defined as: = + (11) in and is a hyperparameter.

[0049] The input data of the multimodal large model is the defect image to be detected and text prompts . After conversion to embedding, it is directly input into the large language backbone network. The defect image After passing through the encoder E( ) Extract features to get visual features = Then, the visual features The last layer of features Then pass it to the vision-language mapper In, through ( ) After aligning the features of the visual feature space to the text feature space, we get , input into the large language backbone network. At the same time, the defect image After tokenization, the token representation that best describes the visual features of each scale is taken out from the vocabulary of the big language backbone network model for each layer of features, and then the L×N tokens are converted into embeddings and input into the big language backbone network.

[0050] After inference, the large language backbone network will output a sequence of tokens { }. A subsequence at a certain position { } is a token related to visual features, that is, = , which is used to generate visual features in the subsequent steps. For this part of tokens, each token group is fused through a linear layer to obtain { }back,{ }Through the language-visual mapper Convert to the feature space that the decoder D can understand, that is, = ( ). For text-related tokens, { } is decoded into a text description. The decoder D receives the aligned features The output of the last layer of encoder E As input, a defect detection mask is output.

[0051] Embodiment 2 Based on the same inventive concept, the present invention also discloses a system for an apparent defect detection method based on a multi-modal large model, comprising: Vision Encoder : Encode the input defect image, extract visual features, and the encoder assigns L×N tokens, where L represents the number of visual scales and N represents the number of tokens assigned to each visual scale; Vision-Language Mapper : Align the encoded visual features to the language feature space; Large language backbone network F: receives the defect image text description, multi-scale token group and visual features aligned to the language feature space as input, obtains language type tokens and visual type tokens after processing, and decodes the language type tokens into text description and evaluation of the defect image; Language-Visual Mapper : Align visual type tokens to visual feature space; Visual Decoder : Decode the encoded visual features and the aligned visual type tokens, and output the final detection mask.

[0052] Since the system introduced in the second embodiment of the present invention is a system adopted for implementing the surface defect detection method based on the multimodal large model in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the technical personnel in this field can understand the specific structure and deformation of the electronic device, so it will not be repeated here.

[0053] Embodiment 3 Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; when one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example 1.

[0054] Since the device introduced in the third embodiment of the present invention is an electronic device used to implement the surface defect detection method based on the multimodal large model in the first embodiment of the present invention, the technical personnel in the field can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All electronic devices used in a method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0055] Embodiment 4 Based on the same inventive concept, the present invention further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first embodiment is implemented.

[0056] Since the device introduced in the fourth embodiment of the present invention is a computer-readable medium used to implement the surface defect detection method based on the multimodal large model in the first embodiment of the present invention, the technical personnel in the field can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All electronic devices used in a method in the embodiment of the present invention belong to the scope of protection of the present invention.

[0057] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. The surface defect detection method based on multi-modal large model is characterized by: The following steps are involved: Annotating a training data set, wherein the training data set includes a defect image, a mask corresponding to the defect image, and a text description; Encode the defect image, extract visual features, and assign L×N tokens to the encoder, where L represents the number of visual scales and N represents the number of tokens assigned to each visual scale; Align the encoded visual features to the language feature space; Using a large language backbone network, during training, the text description of the defect image, the multi-scale token group, and the visual features aligned to the language feature space are input. After processing, the language type token and the visual type token are obtained, and the language type token is decoded into the text description and evaluation of the defect image; Align visual type tokens to visual feature space; Decode the encoded visual features and aligned visual type tokens to obtain the final detection mask image; The multimodal large model consisting of the encoder, language feature alignment with visual features, large language backbone network, visual feature alignment with language features, and decoder is trained, and defect detection is performed using the trained multimodal large model.

2. The surface defect detection method based on multimodal large model according to claim 1 is characterized in that: The text description in the training data set includes the type, shape, size of the defect, the cause of the defect and the repair suggestion for the defect.

3. The surface defect detection method based on multimodal large model according to claim 1 is characterized in that: GPT-4V is used to generate annotated training datasets.

4. The surface defect detection method based on multimodal large model according to claim 1 is characterized in that: The alignment of the visual features to the language features and the alignment of the visual type token to the visual features all adopt the same MLP structure, specifically including a linear layer, a ReLU layer, a linear layer and a Dropout layer.

5. The surface defect detection method based on multimodal large model according to claim 1 is characterized in that: The encoder uses a pre-trained CLIP-ViT encoder; The large language backbone network adopts the pre-trained Vicuna-v1.5-13B. The encoder and the large language backbone network are in a frozen state during the multimodal large model training, and the parameters are not updated.

6. The surface defect detection method based on multimodal large model according to claim 1 is characterized in that: The decoder has a total of L layers, each of which adopts a self-attention layer structure of a multi-dimensional convolutional head transposed attention and a gated convolutional feedforward network.

7. The surface defect detection method based on multimodal large model according to claim 1, characterized in that: The following losses are used when training the multimodal large model: = + in, Represents autoregressive cross entropy loss to ensure the quality of text generation; represents the pixel-by-pixel binary cross entropy loss to ensure the quality of mask generation; and is a hyperparameter.

8. A system for implementing the surface defect detection method based on a multimodal large model as described in any one of claims 1 to 7, characterized in that: include: Vision Encoder : Encode the input defect image and extract visual features. The encoder is assigned L×N tokens, where L represents the number of visual scales and N represents the number of tokens assigned to each visual scale; Vision-Language Mapper : Align the encoded visual features to the language feature space; Large language backbone network F: receives the defect image text description, multi-scale token group and visual features aligned to the language feature space as input, obtains language type tokens and visual type tokens after processing, and decodes the language type tokens into text description and evaluation of the defect image; Language-Visual Mapper : Align visual type tokens to visual feature space; Visual Decoder : Decode the encoded visual features and the aligned visual type tokens, and output the final detection mask.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the surface defect detection method as described in any one of claims 1-7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the surface defect detection method as described in any one of claims 1 to 7 is implemented.

Images