Steel plate surface defect detection method and system based on multi-modal large model

Abstract

The invention relates to the technical field of defect detection, in particular to a steel plate surface defect detection method and system based on a multi-modal large model. Comprising the following steps: acquiring steel plate surface image data including a multi-scene standard image, a defect image, text description and a pixel-level position label; constructing a multi-modal defect positioning analysis model; extracting image feature vectors of the standard image and the defect image, generating defect position codes according to the image feature vectors, and performing feature fusion and up-sampling on the defect position codes through an image decoder to obtain pixel-level defect positions; extracting a defect graph feature vector and a range prompt word vector, performing vector splicing with the text description, inputting into the large language model, and outputting a defect analysis result; and optimizing the multi-modal defect positioning analysis model through the joint loss function. The problems that in the aspect of steel plate surface defect detection, defects can be accurately recognized, defect position information can be output, and a large amount of labeling box data with position information is not depended on are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and particularly to a method and system for detecting steel plate surface defects based on a multimodal large model. Background Art

[0002] In the field of steel plate production, due to the complexity of the processing technology and the diversity of the on-site engineering survey environment, defects such as scratches, stains, and damages are likely to appear on the steel plate surface. It is necessary to achieve accurate detection of steel plate surface defects. Currently, deep learning algorithms based on machine vision are commonly used for steel plate surface defect detection. For example, Chinese Patent CN119130912A proposes a method for detecting strip steel surface defects based on depth maps. This method obtains the point cloud data on the steel plate surface and generates a point cloud data set, and determines the defect category based on the depth information of the point cloud; Chinese Patent CN119313653A proposes a method for detecting inclusion defects on the surface of drawn steel plates based on machine vision. It first obtains a multi-light source image sequence on the steel plate surface, then decomposes the brightness-normalized image sequence through a Laplacian of Gaussian pyramid, performs Laplacian pyramid fusion and image reconstruction, then analyzes the directionality of the drawn texture of the reconstructed image, and finally classifies the image features with the help of ResNet18. Defect recognition algorithms based on traditional images and deep learning are limited by the feature extraction ability of the algorithm model and have insufficient scene generalization ability, and are prone to false alarms or missed detections.

[0003] With the development of artificial intelligence technology, relying on the powerful analysis ability of large models, defect detection is expected to achieve more accurate and intelligent recognition. However, current visual multimodal large models mainly focus on the question-and-answer analysis function, that is, they can only provide text description answers based on the input images and text descriptions, and cannot provide the position information of defects. Taking the detection of steel plate defects as an example, although multimodal large models such as DeepseekJanus and MiniCPM-V can identify defects in images, they cannot give the accurate positions of the defects, which makes it difficult to accurately locate and repair defects in actual production. The multimodal large model Qwen2.5-VL launched by Alibaba supports the output of target box position information on the basis of providing text description answers, but it requires a large amount of labeled data for training. In the scenario of steel plate defect detection, it is difficult to obtain a large amount of labeled box data with position information, which not only increases the cost of data collection and annotation, but also limits the practical application of such models in steel plate defect detection.

[0004] Therefore, the existing methods for detecting steel plate surface defects have obvious deficiencies in accurately positioning the defect positions and data dependence. It is necessary to propose a method that can not only accurately identify defects and output defect position information, but also does not rely on a large amount of labeled box data with position information to provide a solution for steel plate surface defect detection. Summary of the Invention

[0005] To solve the problem of accurately identifying defects and outputting defect location information without relying on a large amount of labeled box data with location information, the present invention provides a steel plate surface defect detection method based on a multimodal large model.

[0006] In a first aspect, a steel plate surface defect detection method based on a multimodal large model provided by the present invention adopts the following technical solutions: A steel plate surface defect detection method based on a multimodal large model includes: Obtain steel plate surface image data, including standard images, defect images, text descriptions, and pixel-level location labels in multiple scenarios; Construct a multimodal defect location analysis model; Extract the graph feature vectors of the standard image and the defect image, generate defect location codes according to the graph feature vectors, and perform feature fusion and upsampling on the defect location codes through an image decoder to obtain pixel-level defect locations; Extract defect graph feature vectors and range prompt word vectors, splice them with the text description and input them into a large language model to output defect analysis results; Optimize the multimodal defect location analysis model through a joint loss function.

[0007] Further, the obtaining of the pixel-level defect location includes: Select the graph feature vectors of the standard image and the defect image in different scale regions for feature extraction; Perform image encoding on the graph feature vectors using an image encoder; Calculate the Euclidean distance of the graph feature vectors. When the Euclidean distance is greater than a set threshold, the defect location code is set to 1, otherwise the defect location code is set to 0; Decode the defect location code back to the original image size through an image decoder to generate a binary map of the pixel-level defect location.

[0008] Further, the image encoder uses the image encoder in the multimodal defect location analysis model , and the selection of the graph feature vectors of the standard image and the defect image in different scale regions for feature extraction includes selecting four different scale regions of 1, 1 / 2, 1 / 4, and 1 / 8 of the standard image and the defect image, and the feature vectors include A total of 4 scales.

[0009] Further, the calculation formula of the Euclidean distance is: , where represents the Euclidean distance between the defect feature and the standard feature of the th dimension of the feature vector of the n-scale image with a scale of 1, represents the d-dimensional defect feature indicating the d-dimensional standard feature at scale 1 of picture n

[0010] Further, the decoding of the defect location encoding back to the original picture size by the image decoder includes: mapping the Euclidean distance between the standard picture and the defect picture to an activation value; weighting the activation value using the defect location encoding; performing a convolution operation on the weighted activation value to fuse features and adjust the dimension; enlarging the size of the processed low-scale feature map to the adjacent high scale through an upsampling layer; starting from the smallest scale, repeating the convolution, upsampling, and splicing operations layer by layer to generate a pixel-level defect location encoding of the original picture size.

[0011] Further, the mapping of the Euclidean distance between the standard picture and the defect picture to an activation value is expressed by the formula: , where is the Euclidean distance, is the activation value, representing the mapping relationship of the Euclidean distance between the standard picture and the defect picture in the image decoder.

[0012] Further, the weighting of the activation value using the defect location encoding is expressed by the formula: , where is the defect location encoding, where n represents the nth picture, is the weighted activation value.

[0013] Further, the convolution operation on the weighted activation value to fuse features and adjust the dimension is expressed by the formula: , where is the image decoder, composed of layer, layer, and layer. The layer performs feature fusion and dimension adjustment on , and then upsamples the feature map at scale 1 / 2 to scale 1 through the layer. Then, it is spliced with the weighted activation value at scale 1 through the layer, and finally, feature fusion is performed through the layer.

[0014] Furthermore, starting from the smallest scale, repeating the convolution, upsampling, and splicing operations layer by layer is expressed by the formula: , where is the final pixel-level defect location, which means decoding the features at the 1 / 8 scale to the 1 / 4 scale, then decoding the 1 / 4 scale features to the 1 / 2 scale, and finally decoding the 1 / 2 scale to the 1 scale to predict the final pixel-level defect location.

[0015] Furthermore, extracting the defect map feature vector and the range prompt word vector, concatenating them with the text description and inputting them into the large language model, and outputting the defect analysis result, including: extracting the defect map feature vector using the image encoder of the multi-modal defect localization analysis model; generating the range prompt word vector using the text encoder of the multi-modal defect localization analysis model; performing feature fusion through matrix multiplication and convolution operations to obtain the cross-modal fusion feature; after concatenating the cross-modal fusion feature with the defect map feature vector, through convolutional feature fusion, mapping it to a feature with the same dimension as the text Token sequence through a linear layer.

[0016] Furthermore, obtaining the cross-modal fusion feature is expressed by the formula: , where is the cross-modal fusion feature, is the defect map feature vector, is the range prompt word vector.

[0017] Furthermore, after concatenating the cross-modal fusion feature with the defect map feature vector, through convolutional feature fusion, mapping it to a feature with the same dimension as the text Token sequence through a linear layer is expressed by the formula: , is the feature with the same dimension as the text Token sequence.

[0018] Furthermore, optimizing the multi-modal defect localization analysis model through the joint loss function includes constructing the loss function, and the loss function includes the defect localization loss and the defect analysis loss, which is expressed by the formula: , where is the defect localization loss, is the defect analysis loss; , Among them, is the true label of the th Token, is the probability prediction of the th Token; , Among them, is to predict the pixel-level defect location of the nth image, is the corresponding pixel-level location label of the nth image.

[0019] In a second aspect, a steel plate surface defect detection system based on a multimodal large model includes: A data acquisition module, configured to acquire steel plate surface image data, including standard images, defect images, text descriptions, and pixel-level location labels of multiple scenarios; A model construction module, configured to construct a multimodal defect location analysis model; A defect location module, configured to extract the graph feature vectors of the standard image and the defect image, generate defect location codes according to the graph feature vectors, and perform feature fusion and upsampling on the defect location codes through an image decoder to obtain pixel-level defect locations; A defect analysis module, configured to extract defect image feature vectors and range prompt word vectors, splice them with the text description and then input them into a large language model to output defect analysis results; A model optimization module, configured to optimize the multimodal defect location analysis model through a joint loss function.

[0020] In a third aspect, the present invention provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are adapted to be loaded and executed by a processor of a terminal device for the described method for detecting steel plate surface defects based on a multimodal large model.

[0021] In a fourth aspect, the present invention provides a terminal device, including a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor for the described method for detecting steel plate surface defects based on a multimodal large model.

[0022] In summary, the present invention has the following beneficial technical effects: 1. A steel plate surface defect detection method and system based on a multimodal large model proposed by the present invention enables precise pixel-level defect localization. Without complex defect fitting, by introducing a multi-scale feature comparison mechanism between a standard image and a defect image, it only needs to focus on the feature differences between the defect area and the standard image to generate pixel-level defect position encoding. Compared with the traditional algorithm that relies on a large number of defect sample fittings, this method significantly reduces the dependence on complex defect morphology annotation data. At the same time, through multi-scale feature fusion and the background suppression mechanism of the decoder, the background area is suppressed by weighted activation values, improving the pixel-level accuracy of defect position prediction and solving the technical problem that existing models cannot provide precise position information.

[0023] 2. The present invention improves the accuracy of defect analysis through cross-modal semantic guidance. The multimodal defect analysis module uses the cross-modal alignment ability of the CLIP model to perform matrix multiplication and convolution fusion on the defect image features and the range prompt word vectors to generate cross-modal features containing semantic associations. After splicing this feature with the original image features and inputting them into the LLM, combined with text descriptions, it guides the model to focus on the steel plate defect scenario, effectively avoiding misjudgments caused by semantic deviations in traditional multimodal models.

[0024] 3. The present invention uses an efficient training mechanism with low data dependence to generate position encoding through the difference analysis between the standard image and the defect image. Only a small number of standard image and defect image pairs in multiple scenarios are required to achieve effective training. Compared with models such as Alibaba Qwen2.5-VL that strongly rely on large-scale annotated data, this method significantly reduces the data collection and annotation costs and is more suitable for the actual needs of difficult defect sample acquisition in industrial scenarios.

[0025] 4. The present invention synchronously optimizes the defect position localization loss and the LLM analysis loss through a joint loss function. By designing range prompts, the correlation between the LLM input and the real defect image is increased, thereby improving the accuracy of LLM defect analysis. Ensure that the model synergistically improves at the pixel-level localization and semantic analysis levels. Combining multi-scale feature encoding and cross-modal fusion technologies effectively enhances the generalization ability of the model for complex industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram of a steel plate surface defect detection method based on a multimodal large model according to Embodiment 1 of the present invention; Figure 2 is a schematic diagram of four different scale regions of a selected standard image and a defect image according to Embodiment 1 of the present invention; Figure 3 is a schematic diagram of the method according to Embodiment 1 of the present invention; Figure 4 is a flowchart of Embodiment 1 of the present invention; Figure 5 is a block diagram of Embodiment 2 of the present invention. Specific Embodiments

[0027] The present invention will be further described in detail below with reference to the accompanying drawings.

[0028] Example 1 Referring to Figure 1 and Figure 3 , a steel plate surface defect detection method based on a multi-modal large model in this embodiment includes: S1. Obtain steel plate surface image data, including standard images, defect images, text descriptions, and pixel-level position labels in multiple scenarios; by establishing a steel plate surface defect image text database, the database is divided into multiple scenarios, and multiple images are obtained for each scenario, including a standard image without defects and multiple defect images with defects, as well as the corresponding text data and pixel-level position labels for each defect image.

[0029] Collect steel plate surface images through an industrial line array camera or a 3D vision sensor, covering multiple process scenarios such as cold rolling, hot rolling, and plating. For a single scenario, no less than 200 groups of images (including 100 standard images and 100 defect images) are collected. The text description adopts a structured format, including defect types (such as "scratch", "pit", "crack", "scale", "hole"), position direction (with the upper left corner of the image as the origin, and the rectangular area is represented by pixel coordinates (x1, y1, x2, y2)), and severity (such as "for a scratch, length L = 15.2 mm, width W = 0.3 mm"). The pixel-level position labels are manually marked through the LabelMe tool to generate a JSON file containing the defect contour, which is converted into a single-channel binary mask (Mask) through OpenCV. The pixel value of the defect area is 255, and the background is 0.

[0030] S2. Construct a multi-modal defect location analysis model; the multi-modal defect location analysis model includes a defect location part and a multi-modal defect analysis part. The model is improved based on the CLIP-ViT-B / 16 architecture. The input size of the image encoder is adjusted to 384×384 pixels, which contains 12 layers of Transformer encoders, each layer has 12 attention heads, the hidden layer dimension is 768, and LayerNorm normalization is adopted. The text encoder adds a domain prompt word embedding layer for "steel plate defects". During pre-training, prompt words such as "cold-rolled plate scratches" and "hot-rolled scale" are aligned with the corresponding image features through contrastive loss (temperature parameter τ = 0.07). The decoder uses the U-Net++ structure, which includes 4 layers of downsampling paths (each layer has 2×3×3 convolution + ReLU + max pooling, stride 2) and 4 layers of upsampling paths (each layer has transposed convolution + skip connection). The output layer uses the Sigmoid activation function to generate a 0-1 probability map. The large language model (LLM) selects the fine-tuned version of LLaMA-7B-IndustrialDefect. The pre-training data contains 100,000 steel plate defect report texts, the vocabulary is extended to 50,000 tokens, and new professional vocabulary such as "rolling direction" and "grain size" is added.

[0031] S3. Extract the graph feature vectors of the standard graph and the defect graph, generate the defect location encoding according to the graph feature vectors, and perform feature fusion and upsampling on the defect location encoding through the image decoder to obtain the pixel-level defect location. It includes: S31. Select the graph feature vectors of the standard graph and the defect graph in different scale regions for feature extraction; the image encoder uses the image encoder in the multi-modal defect location analysis model , and the selection of the graph feature vectors of the standard graph and the defect graph in different scale regions for feature extraction refers to Figure 2 , including selecting four different scale regions of 1, 1 / 2, 1 / 4, and 1 / 8 of the standard graph and the defect graph. The feature vectors include a total of 4 scales. The Gaussian pyramid is used to perform 4-level downsampling on the image to generate four scale images of 1 (384×384), 1 / 2 (192×192), 1 / 4 (96×96), and 1 / 8 (48×48). For each scale image, a 1024-dimensional feature vector is extracted through the image encoder, denoted as . The feature vector normalization formula is: , where and are the mean and standard deviation of the standard graph features at this scale.

[0032] S32. Use the image encoder to perform image encoding on the graph feature vectors.

[0033] S33. Calculate the Euclidean distance of the graph eigenvector. When the Euclidean distance is greater than the set threshold, the defect position code is set to 1; otherwise, the defect position code is set to 0. The calculation formula of the Euclidean distance is: , where represents the Euclidean distance between the defect feature and the standard feature of the -th dimension of the eigenvector with scale 1 of picture n, represents the defect feature of the -th dimension of picture n with scale 1, represents the standard feature of the -th dimension of picture n with scale 1.

[0034] This embodiment gives a method for setting the threshold. The standard graph and the defect graph are segmented at multiple scales by using the sliding window technique. Scale 1 is the original graph (such as 2048×1024 pixels), scale 1 / 2 is 1024×512 pixels, and so on to scale 1 / 8 (256×128 pixels). Each scale generates a 1024-dimensional eigenvector. The cross-scale feature consistency is calculated by cosine similarity, and the Euclidean distance threshold is dynamically adjusted. It is expressed by the formula: Euclidean distance threshold = mean value of the current scale feature + λ×standard deviation, where λ∈[1.5,2.0] is an empirical coefficient. If the cross-scale feature consistency is relatively high, the Euclidean distance threshold can be appropriately increased to filter noise. If the cross-scale feature consistency is poor, it means that the feature fluctuation is large, and the Euclidean distance threshold can be appropriately reduced to avoid missing detections.

[0035] S34. Decode the defect position code back to the original graph size through an image decoder to generate a binary graph of the pixel-level defect position.

[0036] Decoding the defect position code back to the original graph size includes: S341. Map the Euclidean distance between the standard graph and the defect graph to an activation value. The mapping of the Euclidean distance between the standard graph and the defect graph to an activation value is expressed by the formula: , where is the Euclidean distance, is the activation value, representing the mapping relationship of the Euclidean distance between the standard graph and the defect graph in the image decoder.

[0037] S342. Weight the activation value by using the defect position code. The weighting of the activation value by using the defect position code is expressed by the formula: , where is the defect position code, where n represents the n-th picture, is the weighted activation value.

[0038] When the position encoding is 0, it represents the distance less than the threshold. Since most regions in the actual image are background regions without defects, the feature difference between the standard image and the defect image in the background region is very small, that is will be very small. Therefore, most of the regions with position encoding 0 are background regions unrelated to defects. When 0 is multiplied by the corresponding position activation value, it will reduce the contribution of the pixel at that position to the loss function. Therefore it will suppress the pixels in the background region unrelated to defects.

[0039] S343. Perform a convolution operation on the weighted activation value to fuse features and adjust the dimensions.

[0040] The convolution operation on the weighted activation value to fuse features and adjust the dimensions is expressed by the formula: , where is the image decoder, which consists of layers, layers and layers. The layer performs feature fusion and dimension adjustment on , and then through the layer, upsamples the feature map at the 1 / 2 scale to the 1 scale, and then concatenates it with the weighted activation value at the 1 scale through the layer. Finally, it passes through the layer for feature fusion.

[0041] S344. Enlarge the size of the processed low-scale feature map to the adjacent high scale through the upsampling layer.

[0042] S345. Starting from the smallest scale, repeat the convolution, upsampling, and concatenation operations layer by layer to generate the pixel-level defect position encoding of the original image size.

[0043] The repetition of the convolution, upsampling, and concatenation operations layer by layer starting from the smallest scale is expressed by the formula: , where is the final pixel-level defect position, which means decoding the feature at the 1 / 8 scale to the 1 / 4 scale, then decoding the 1 / 4 scale feature to the 1 / 2 scale, and finally decoding the 1 / 2 scale to the 1 scale to predict the final pixel-level defect position.

[0044] Specific operation implementation example. For a 1-scale image (384×384), it is divided into 12×12 windows of 32×32 pixels, and each window generates a 1024-dimensional feature vector through the ViT-B / 16 encoder . Calculate the mean and standard deviation of the standard image , which are used for normalizing the defect map features.

[0045] Dynamic threshold calculation example scenario: For a defect map of a hot-rolled steel plate (scale 1 / 8, 48×48 pixels), a suspected scale defect is detected. Step 1: Calculate the Euclidean distance of all windows at this scale to obtain a distance matrix (single window), with a value of . Step 2: Statistically analyze the distance distribution of the standard map at this scale to obtain . Step 3: Calculate the cross-scale consistency (cosine similarity with the distance matrix of the 1 / 4 scale ): (Note: sim = 0.75 belongs to medium consistency, and λ takes the default value of 1.8). Step 4: Dynamic threshold calculation: , Step 5: Threshold judgment: , , and mark this window as the defect area.

[0046] Example of layer-by-layer operation of the image decoder. Input: Defect position encoding at the 1 / 8 scale (48×48 pixels, with a value of 1), corresponding to the weighted activation value .

[0047] Decoding process: 1 / 8 scale → 1 / 4 scale (48×48 → 96×96): Perform a 3×3 convolution (64 channels, padding = 1) on , and output . Upsample bilinearly to 96×96 pixels to obtain . Concatenate it with the weighted activation value (already normalized) at the 1 / 4 scale in the channel dimension to obtain . After a 3×3 convolution (128 channels), output the fused feature . 1 / 4 scale → 1 / 2 scale (96×96 → 192×192): Perform a 3×3 convolution (64 channels) on , and output . Upsample to 192×192 pixels, concatenate it with , and then perform convolution to output . 1 / 2 scale → 1 scale (192×192 → 384×384): Perform a convolution on Perform a 3×3 convolution (32 channels), upsample to 384×384 pixels, and concatenate with After concatenation, perform convolution to output the final mask . After activation by Sigmoid and binarization with a threshold of 0.5, a binary image with the defective area being white (pixel value 255) is generated, clearly marking the position of the scale.

[0048] Refer to Figure 4 , S4. Extract the defect map feature vector and the range prompt word vector, concatenate them with the text description, and then input them into the large language model to output the defect analysis result. The large language model used in this embodiment is LLaMA3 (an open-source large language model released by Meta).

[0049] Including: S41. Use the image encoder of the multi-modal defect localization analysis model to extract the defect map feature vector.

[0050] In an actual industrial production environment, the images of the steel plate surface may be disturbed by various factors such as lighting, stains, and shooting angles. The pre-trained CLIP model can effectively resist the influence of these interference factors. It constructs a rich and comprehensive image semantic space through learning 400 million image-text pairs. When extracting the defect map feature vector, global average pooling (GAP) integrates the features of each patch in the ViT encoder, removes redundant information in the features, and at the same time retains the overall semantic information of the image. After normalization by the LayerNorm layer, the distribution of the feature vector becomes more stable, which is beneficial to subsequent calculations and processing.

[0051] S42. Use the text encoder of the multi-modal defect localization analysis model to generate the range prompt word vector.

[0052] The image encoder uses the pre-trained CLIP model, whose weights are learned through training with 400 million image-text pairs, and can encode the image semantics into a vector representation in a high-dimensional feature space. When extracting the defect map feature vector, global average pooling (GAP) aggregates the features of all patches in the ViT encoder, and then normalizes through the LayerNorm layer. Finally, a 768-dimensional vector is output through the CLS token , and this vector has strong semantic representation ability for visual features such as texture anomalies and geometric defects on the steel plate surface.

[0053] The range prompt word generation module has an industrial semantic dictionary built-in, which contains more than 200 terms related to steel plate defects (such as "rolling line", "intergranular corrosion", "galvanized layer thickness"). Through domain adaptation fine-tuning, the text encoder can generate semantic vectors aligned with visual features. For example, when the input prompt word is "Steel plate surface defect detection: type = pit, location = middle", the text encoder will preferentially activate semantic dimensions such as "depth" and "irregular edge", forming cross-modal associations with local depression features in the image.

[0054] The defect map feature vector is extracted through the last layer CLS token of the image encoder (768 dimensions). The range prompt word uses the template "Steel plate surface defect detection: type={type}, location={location}", where {type} is the preset defect category and {location} are regional keywords such as "upper left" and "middle", and a 768-dimensional vector is generated through the text encoder.

[0055] S43. Feature fusion is performed through matrix multiplication and convolution operations to obtain cross-modal fusion features.

[0056] It is expressed by the formula: , where is the cross-modal fusion feature, is the defect map feature vector, is the range prompt word vector.

[0057] Cosine similarity is used for matrix multiplication: , generating a 2D similarity matrix (H×W), which is compressed into a 768-dimensional cross-modal feature through a 3×3 convolution (32 channels, stride = 1) 。

[0058] The range prompt word is fixed as "Steel plate surface defect detection: {defect type}", and a 768-dimensional feature vector is generated through the text encoder. Cosine similarity is used for matrix multiplication , highlighting the feature response of the semantic matching region, and then a cross-modal fusion feature is generated through a 3×3 convolution.

[0059] S44. After concatenating the cross-modal fusion features and the defect map feature vectors, through convolutional feature fusion, they are mapped to features with the same dimension as the text Token sequence through a linear layer. The text Tokens adopt Byte-Pair Encoding (BPE), and the vocabulary size is expanded to 50,000, including professional terms such as "cold-rolled sheet" and "scale". After the fusion features and the image features are concatenated, they are normalized by the LayerNorm layer, and then mapped to the LLM input dimension (2048 dimensions) through a linear layer to ensure consistency with the embedding space of the text Tokens.

[0060] It is expressed by the formula: , are features with the same dimension as the text Token sequence.

[0061] In this embodiment, a range hint design is designed. In addition to the text description corresponding to the defect image, a short range hint is provided, corresponding to Figure 3 "Steel plate defect image" in the lower left corner. After the range hint words and the defect map are multimodally aligned and fused, they are concatenated with the text description corresponding to the image ("This is an image of a steel plate. Find the scratches in the picture") and then input into the LLM (Large Language Model). In this way, the correlation between the LLM input and the real defect image is increased, thereby improving the accuracy of LLM defect analysis.

[0062] The specific process is as Figure 3 shown in the lower part. In the first step, the image encoder of the CLIP model is used to extract the defect image feature vector , and the Transformer text encoder of the CLIP model is used to generate the range hint word vector . The range hint words mainly include two parts: the item type and the item status (normal or defective). In the second step, through matrix multiplication and operations, feature fusion is performed to obtain . The CLIP model can learn the matching relationship between text-image pairs, that is, the defect map feature vector is the feature vector with the highest matching degree with the range hint word vector . Therefore, the obtained after feature fusion contains the corresponding relationship between the defect image and the range hint words (steel plate, defect). In the third step, and the defect image feature are concatenated through , and after passing through feature fusion, they are mapped into features with the same dimension as the text Tokens through the Linear linear layer , After being concatenated with text tokens and input into the LLM, the large model analyzes and outputs the text description result "There is a black scratch in the middle of the image extending from the upper left to the lower right". Compared with ordinary multimodal models, the multimodal large model design proposed in this embodiment, in addition to the image feature vector and the prompt word vector also additionally introduces a vector that provides range information, increasing the correlation between the LLM input and the real defect image, thereby improving the accuracy of LLM defect analysis.

[0063] The cross-modal fusion features and the defect map feature vectors are concatenated to integrate the original feature information of the image and the feature information after cross-modal fusion. This can make full use of the global features of the image and the features after association with the text, improving the description ability of steel plate defects. After convolutional feature fusion, the potential information in the concatenated features can be further mined, enhancing the expression ability of the features. In the process of mapping the fusion features to the LLM input dimension (2048 dimensions), the linear layer plays a key role. It learns a linear transformation matrix to map the fusion features from the original dimension to the same dimension as the text token sequence. The purpose of this is to ensure that the feature vectors are consistent with the embedding space of the text tokens, enabling the LLM to better process and understand this feature information. The text tokens adopt byte pair encoding (BPE), and the vocabulary size is expanded to 50,000, including professional terms such as "cold rolled plate" and "scale". This encoding method can effectively process various complex text information, and the expanded vocabulary can cover more industrial field professional terms, improving the accuracy and professionalism of the description of steel plate defects.

[0064] In actual industrial application scenarios, relying solely on the text description corresponding to the defect image may not accurately convey the defect information in the image. The range hint design proposed in this embodiment provides more prior information for the LLM by providing a short range hint. After the range hint is multimodally aligned and fused with the defect map and then concatenated with the text description corresponding to the image and input into the LLM, the LLM can more accurately understand the defect situation in the image. For example, when there are multiple similar defects in the image, the range hint can help the LLM determine the specific defect location and type to be analyzed. Compared with ordinary multimodal models, the additional range information vector introduced in the multimodal large model design of this embodiment is like providing an accurate navigation for the LLM. It can guide the LLM to focus more on the key regions and features in the image, thereby increasing the correlation between the LLM input and the real defect image and significantly improving the accuracy of the LLM defect analysis. In actual tests, after adopting the design of this embodiment, the accuracy rate of the LLM for steel plate defect analysis has increased by 15% - 20%, greatly improving the quality inspection efficiency in industrial production.

[0065] S5. Optimize the multimodal defect localization and analysis model through the joint loss function.

[0066] The optimization of the multimodal defect localization and analysis model through the joint loss function includes constructing a loss function, and the loss function includes a defect localization loss and a defect analysis loss, which are expressed by the formula: , where, is the defect localization loss, is the defect analysis loss; , where, is the true label of the th Token, is the probability prediction of the th Token; , where, is the pixel-level defect location predicted for the nth image, is the corresponding pixel-level location label of the nth image.

[0067] The training adopts an end-to-end manner, and the optimizer is AdamW. (β1 = 0.9, β2 = 0.999, weight decay = 0.01. β1 is the exponential decay rate of the first - moment estimate (default 0.9) for calculating the average of historical gradients (bias correction), β2 is the exponential decay rate of the second - moment estimate (default 0.999) for calculating the average of the squares of historical gradients. Weight decay is the weight decay coefficient (0.01 here), which is equivalent to L2 regularization and prevents overfitting by penalizing large weight parameters), learning rate 1e - 4, batch size 16, training for 50 epochs. Defect localization loss Adopt a mixed loss of Dice loss and binary cross - entropy (BCE): Defect analysis loss Adopt cross - entropy to calculate the per - word loss for the Token sequence output by the LLM. Total loss weight = 1:0.8, dynamically adjusted by the F1 score of the validation set.

[0068] Example 2 Refer to Figure 5 , this example provides a steel plate surface defect detection system based on a multimodal large model, including: A data acquisition module, configured to acquire steel plate surface image data, including standard images, defect images, text descriptions, and pixel - level position labels in multiple scenarios; A model construction module, configured to construct a multimodal defect localization and analysis model; A defect position localization module, configured to extract the graph feature vectors of the standard image and the defect image, generate defect position encodings according to the graph feature vectors, and perform feature fusion and upsampling on the defect position encodings through an image decoder to obtain pixel - level defect positions; A defect analysis module, configured to extract defect graph feature vectors and range prompt word vectors, splice them with the text description and then input them into a large language model to output defect analysis results; A model optimization module, configured to optimize the multimodal defect localization and analysis model through a joint loss function.

[0069] A computer - readable storage medium, in which multiple instructions are stored, and the instructions are adapted to be loaded and executed by a processor of a terminal device for the described steel plate surface defect detection method based on a multimodal large model.

[0070] A terminal device, including a processor and a computer - readable storage medium, the processor is used to implement each instruction; the computer - readable storage medium is used to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor for the described steel plate surface defect detection method based on a multimodal large model.

[0071] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A method for detecting surface defects of steel plates based on a multimodal large model, characterized in that, Comprising: Obtain the image data of the steel plate surface, including standard images, defect images, text descriptions, and pixel-level position labels in multiple scenarios; Construct a multi-modal defect location analysis model; Extract the graph feature vectors of the standard image and the defect image, generate defect location codes according to the graph feature vectors, and perform feature fusion and upsampling on the defect location codes through an image decoder to obtain pixel-level defect locations; Extract the defect image feature vectors and range prompt word vectors, splice them with the text description and input them into the large language model to output the defect analysis results; Optimize the multi-modal defect location analysis model through a joint loss function.

2. A method for detecting surface defects of steel plates based on a multimodal large model according to claim 1, characterized in that The obtaining of the pixel-level defect location includes: Select the graph feature vectors of the standard image and the defect image in different scale regions for feature extraction; Perform image encoding on the graph feature vectors using an image encoder; Calculate the Euclidean distance of the graph feature vectors. When the Euclidean distance is greater than the set threshold, the defect location code is set to 1, otherwise the defect location code is set to 0; Decode the defect location code back to the original image size through an image decoder to generate a binary map of the pixel-level defect location.

3. A method for detecting steel plate surface defects based on a multimodal large model according to claim 2, characterized in that, The decoding of the defect location code back to the original image size includes: Map the Euclidean distance between the standard image and the defect image to an activation value; Weight the activation value using the defect location code; Perform a convolution operation on the weighted activation value to fuse features and adjust the dimensions; Enlarge the size of the processed low-scale feature map to the adjacent high scale through an upsampling layer; Starting from the smallest scale, repeat the convolution, upsampling, and splicing operations layer by layer to generate a pixel-level defect location code of the original image size.

4. A method for detecting steel plate surface defects based on a multimodal large model according to claim 3, characterized in that, The mapping of the Euclidean distance between the standard image and the defect image to an activation value is expressed by the formula: ， Among them, is the Euclidean distance, is the activation value, representing the Euclidean distance between the standard image and the defective image and the mapping relationship in the image decoder; The weighting of the activation value using the defect location code is expressed by the formula: ， Among them, is the defect location code, where n represents the nth picture, is the weighted activation value.

5. A method for detecting surface defects of steel plates based on a multimodal large model according to claim 1, characterized in that, The extraction of the defect image feature vectors and range prompt word vectors, splicing with the text description and inputting into the large language model to output the defect analysis results includes: Extract the defect image feature vectors using the image encoder of the multi-modal defect location analysis model; Generate range prompt word vectors using the text encoder of the multi-modal defect location analysis model; Perform feature fusion through matrix multiplication and convolution operations to obtain cross-modal fusion features; After splicing the cross-modal fusion features with the defect image feature vectors, perform convolution feature fusion and map them to features with the same dimension as the text Token sequence through a linear layer.

6. The method for detecting steel plate surface defects based on a multi-modal large model according to claim 5, characterized in that, The obtaining of the cross-modal fusion features is expressed by the formula: ， Among them, is the cross-modal fusion feature, is the defect map feature vector, is the range prompt word vector; After splicing the cross-modal fusion features with the defect image feature vectors, performing convolution feature fusion and mapping them to features with the same dimension as the text Token sequence is expressed by the formula: ， Features with the same dimension as the text Token sequence.

7. A method for detecting steel plate surface defects based on a multimodal large model according to claim 1, characterized in that, The optimization of the multi-modal defect location analysis model through the joint loss function includes constructing a loss function, and the loss function includes a defect location loss and a defect analysis loss, which is expressed by the formula: ， Among them, is the defect location loss, is the defect analysis loss; ， Among them, is the true label of the th Token, is the probability prediction of the th Token; ， Among them, to predict the pixel-level defect location of the nth image, is the corresponding pixel-level location label of the nth image.

8. A steel plate surface defect detection system based on a multimodal large model, characterized in that, Including: A data acquisition module configured to obtain the image data of the steel plate surface, including standard images, defect images, text descriptions, and pixel-level position labels in multiple scenarios; A model construction module configured to construct a multi-modal defect location analysis model; The defect location positioning module is configured to extract the graph feature vectors of the standard graph and the defect graph, generate defect location codes according to the graph feature vectors, and perform feature fusion and upsampling on the defect location codes through an image decoder to obtain the pixel-level defect locations; The defect analysis module is configured to extract the defect graph feature vectors and the range prompt word vectors, perform vector splicing with the text description and then input them into the large language model, and output the defect analysis results; The model optimization module is configured to optimize the multi-modal defect location analysis model through the joint loss function.

9. A computer-readable storage medium storing multiple instructions, characterized in that, The instructions are suitable for being loaded and executed by the processor of the terminal device for a steel plate surface defect detection method based on a multi-modal large model as claimed in claim 1.

10. A terminal device, comprising a processor and a computer-readable storage medium, the processor being configured to implement each instruction; the computer-readable storage medium being configured to store a plurality of instructions, characterized in that, The instructions are suitable for being loaded and executed by the processor for a steel plate surface defect detection method based on a multi-modal large model as claimed in claim 1.

Images