Industrial image quality inspection method integrating open set detection and segmentation

Abstract

The invention relates to an industrial image quality inspection method integrating open set detection and segmentation. Comprising the steps of image and text data input and coding, spatial position and multi-scale texture fusion coding of image features, cross-modal fusion of image-text features, target prediction decoding of fusion features, screening and prompt generation of high-confidence targets and mask prediction decoding based on sparse prompt. Through integrated model design, the problems of low reasoning efficiency, poor segmentation consistency, high resource consumption, lack of module collaborative optimization, poor generalization ability and the like in the prior art are solved, the precision, efficiency and generalization ability of industrial defect detection and segmentation are improved, and the industrial application prospect is good.

Description

Technical Field

[0001] This invention relates to the field of visual imaging inspection technology, and in particular to an industrial image quality inspection method that integrates open set detection and segmentation. Background Technology

[0002] With the rapid development of deep learning in computer vision, industrial vision quality inspection systems are gradually shifting from traditional manual rule design to end-to-end automated detection and segmentation based on deep models. Deep models offer higher recognition accuracy and stronger adaptability when faced with large-scale image data and diverse defect morphologies. However, industrial defects are typically characterized by sparse distribution, complex shapes, and insufficient prior information, leading to generalization difficulties and efficiency bottlenecks for the models.

[0003] To address the above issues, the technical approaches commonly adopted by academia and industry are as follows:

[0004] 1. Open set target detection models (such as Grounding DINO): This is an open set target detection framework that integrates visual and linguistic information. It can achieve open localization of target regions through natural language or category names. It has the ability to identify unseen class targets and is suitable for detection tasks with low sample sizes or no prior defects in industrial quality inspection.

[0005] 2. Cue-Driven Segmentation Models (e.g., Segment Anything Model, SAM): SAM is a general segmentation model that supports various forms of cue input, such as points, boxes, and text. It achieves image segmentation of arbitrary regions through an adaptive mask generation mechanism. SAM has strong transferability across multiple domains and is widely used in general segmentation tasks.

[0006] 3. Serial Segmentation Process of Grounding DINO and SAM: In current mainstream solutions, Grounding DINO and SAM models are often combined in a serial stream to achieve a joint task of open set object detection and cue segmentation. This process generally includes the following steps: First, Grounding DINO detects potential defects in the image and outputs several bounding boxes; then, each bounding box is used as a cue input to the SAM model to generate a corresponding segmentation mask; multiple targets will trigger multiple SAM inferences, and finally the segmentation results are merged.

[0007] Although the above methods have a certain degree of generalization ability and task universality, they also expose a series of key problems:

[0008] 1. Low inference efficiency: Each target box needs to be independently input into SAM for inference once. The calculation process is redundant, and the overall inference time increases linearly with the number of targets, which cannot meet the requirements of industrial quality inspection for real-time detection.

[0009] 2. Poor segmentation consistency: Each segmentation task is independent of the others and lacks unified semantic context information integration, which may lead to problems such as blurred boundaries and misclassification of targets.

[0010] 3. High resource consumption: Grounding DINO and SAM are inferenced as two independent models, each requiring a complete feature encoding of the image, resulting in a large number of repetitive computations. Especially in industrial scenarios with large image sizes and a large number of defects, this repetitive feature extraction significantly increases the system's GPU / CPU load, substantially increasing the consumption of computing resources and deployment costs.

[0011] 4. Lack of module collaborative optimization mechanism: In the current serial structure, there is a lack of joint training and feature sharing mechanism between the two models. The overall system performance is constrained by the bottleneck model, limiting the space for further performance improvement.

[0012] Therefore, how to break through the computational bottleneck of the existing serial mode and build an integrated detection and segmentation model with open set perception capability, high segmentation accuracy, excellent inference efficiency and low computational cost is one of the key technical challenges that urgently need to be solved in the current industrial quality inspection field. Summary of the Invention

[0013] The technical problem to be solved by this invention is to provide an industrial image quality inspection method that integrates open set detection and segmentation, aiming to solve the problems of low inference efficiency, poor segmentation consistency, high resource consumption, lack of module collaborative optimization mechanism, and poor generalization ability of traditional segmentation models in existing industrial defect detection and segmentation technologies.

[0014] The technical solution adopted by this invention to solve its technical problem is: an industrial image quality inspection method integrating open set detection and segmentation, comprising the following steps:

[0015] S1. Image and text data input and encoding: Input a three-channel industrial image I and a defect descriptive language prompt T. The image encoding module extracts features from I to obtain the image feature embedding FEI. The text encoding module performs semantic embedding on T to obtain the semantic feature embedding FET.

[0016] S2. Spatial location and multi-scale texture fusion encoding of image features: Input FEI into the multi-scale adaptive feature weighting fusion module to generate a fused unified scale feature map FEI_f, and then send FEI_f into the random location encoding module for spatial information enhancement to obtain the image feature embedding FEI_total after multi-scale and location fusion encoding.

[0017] S3. Cross-modal fusion of image and text features: Deep modal interaction is performed between FEI and the FET input image and text cross-attention fusion module to obtain multimodal fusion feature embedding FE_MM;

[0018] S4. Target prediction decoding with fused features: Combine FE_MM with FEI and FET and input them into the prediction decoding module to predict the candidate detection box Boxes_pred;

[0019] S5. Filtering and generating high-confidence targets: Filter the top-k high-confidence non-background target boxes from Boxes_pred, extract their corresponding FE_MM and spatial location features, and generate sparse hint embedding P_sparse.

[0020] S6. Mask prediction and decoding based on sparse cueing: FEI_total and P_sparse are fed into the mask prediction module based on the modified SAM structure to generate a semantic segmentation mask that matches the detection target.

[0021] Furthermore, in step S1 of the present invention, the image encoding module uses a Swing Transformer or an Efficient-Swin Transformer visual backbone network; the text encoding module uses a BERT or CLIP pre-trained language model.

[0022] Furthermore, in step S2 of the present invention, the processing procedure of the multi-scale adaptive feature weighted fusion module includes: setting a learnable weight vector consistent with the number of feature maps, and weighting the feature maps at each scale according to their importance; using convolution operations to unify the feature maps at each scale to the maximum number of channels at the largest scale; using bilinear interpolation to align the feature map sizes; and summing the weighted data to generate FEI_f.

[0023] Furthermore, in step S2 of the present invention, the processing procedure of the random position encoding module includes: initializing a learnable transformation matrix of Gaussian distribution, projecting the normalized two-dimensional coordinates onto the high-frequency space; calculating the sine and cosine functions of the projected coordinates respectively, and concatenating them as the position encoding FEI_pos; adding FEI_pos to FEI_f to obtain FEI_total.

[0024] Furthermore, in step S3 of the present invention, the image-text cross-attention fusion module includes two directions: image-to-text attention and text-to-image attention. It retains the original image and text features and uses them together with the cross-attention result as an intermediate fusion representation to obtain FE_MM.

[0025] Furthermore, in step S4 of the present invention, the prediction decoding module uses a position decoding strategy to predict the target box coordinates and corresponding confidence scores to obtain Boxes_pred.

[0026] Furthermore, in step S5 of the present invention, the calculation process of the spatial location features includes: shaping the positions of the top-k candidate boxes according to the upper left and lower right coordinates to obtain feature vectors, and encoding them through MLP; extracting the geometric attributes of the candidate boxes to form vectors and encoding them; concatenating the two sets of encoded features with FE_MM, and obtaining P_sparse by single-layer MLP dimensionality reduction.

[0027] Furthermore, in step S6 of the present invention, the mask prediction module uses P_sparse as a prompt and FEI_total to provide texture and spatial support to generate a pixel-level segmentation mask.

[0028] The beneficial effect of this invention is that it solves the defects existing in the prior art.

[0029] 1. Addressing the issue of low inference efficiency: This invention no longer performs segmentation inference for each target box separately. Instead, it uses a many-to-one prompting mechanism to centrally process multiple detection targets in the segmentation module, significantly reducing repetitive calculation steps. This prevents the overall inference time from increasing linearly with the number of targets, thereby improving inference efficiency and meeting the real-time requirements of industrial scenarios.

[0030] 2. To address the issue of poor segmentation consistency: By introducing a unified image coding and image-text multimodal prompting mechanism, the segmentation module's ability to understand the overall semantic information of the image is enhanced, enabling semantic information sharing among different targets and effectively improving boundary clarity and target recognition consistency.

[0031] 3. Addressing the issue of high resource consumption: This invention integrates the detection and segmentation processes into the same feature extraction process, avoiding the problem of repeated image feature encoding in existing concatenated schemes of Grounding DINO and SAM, significantly reducing GPU / CPU load and computing resource consumption, and improving deployment efficiency and cost-effectiveness.

[0032] 4. To address the lack of a module collaborative optimization mechanism: This invention designs a multi-level feature self-learning weighted fusion structure, which supports deep information interaction and adaptive feature guidance between detection and segmentation modules, enabling the overall model to be jointly optimized end-to-end, breaking through the performance bottleneck of the serial structure, and achieving better overall detection and segmentation performance.

[0033] 5. To address the poor generalization ability of traditional segmentation models: By fusing sparse cueing and open set detection results to guide the model to focus on potential defect regions, and combining image-text multimodal contextual understanding capabilities, the model's generalization ability on novel or low-frequency defect targets is effectively improved.

[0034] In summary, this invention systematically optimizes the traditional serial scheme in terms of model structure and information flow mechanism, which not only improves the segmentation accuracy and generalization ability, but also significantly improves the operating efficiency and system deployment flexibility, and has good prospects for industrial application and promotion value. Attached Figure Description

[0035] Figure 1 This is an overall framework diagram of the method of the present invention;

[0036] Figure 2 This is a flowchart of the overall method of the present invention;

[0037] Figure 3 This is a flowchart of the multimodal information sparse prompt information encoding process of the present invention;

[0038] Figure 4 This is a flowchart of the multi-scale adaptive feature fusion and position enhancement coding process of the present invention;

[0039] Figure 5 This is a segmentation result image of the cross-domain generalization capability of the present invention; the first row is the training data, the second row is the test data, the data has been scaled and edge-filled, the scale has been reduced, and the background and defect type morphology are different from the training data, and the third row is a magnified local area of ​​the second row. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0041] like Figures 1-5This paper presents an industrial image quality inspection method integrating open set detection and segmentation. Based on a multimodal open set target detection and semantic segmentation fusion system for industrial defect detection, this system fully integrates image information and language prompts, significantly improving the detection and segmentation capabilities for complex and unknown types of defects. The system inherits and extends the functionality and structure of existing open set target detection models (such as Grounding DINO and YOLO-world). The specific implementation steps are as follows:

[0042] 1. Image and text data input and encoding

[0043] The input consists of a three-channel industrial image I, such as an image of the surface of a component collected from an industrial production line, and a descriptive language prompt T for defects, such as "white spots" or "black scratches." The image encoding module uses a Swing Transformer as the visual backbone network to extract features from image I, resulting in an image feature embedding (FEI). The text encoding module uses a BERT pre-trained language model to perform semantic embedding on the text prompt T, resulting in a semantic feature embedding (FET). FEI and FET are used for subsequent cross-modal fusion and localization prediction, respectively.

[0044] 2. Spatial location of image features and multi-scale texture fusion coding

[0045] FEI is a list of multi-scale feature maps containing image information at different semantic levels and resolutions. The FEI is input into the multi-scale adaptive feature weighting fusion module: a learnable weight vector is set with the same number of feature maps as the number of feature maps. Assuming there are three scales of feature maps, the weight vector is [w1, w2, w3], where w1, w2, and w3 are learnable parameters used to weight the importance of each scale of feature map; convolution operations are used to unify the number of channels of each scale feature map to the number of channels of the largest scale feature map. For example, if the largest scale feature map has 256 channels, then the other scale feature maps are adjusted to 256 channels through convolution; bilinear interpolation is used to align the dimensions (width and height) of each feature map to the size of the largest scale feature map; the weighted feature maps of each scale are then summed to generate the fused unified scale feature map FEI_f.

[0046] The FEI_f is fed into the random location coding module: a Gaussian-distributed learnable transformation matrix W is initialized with a dimension of 2×d, where d is the dimension of the location coding; the normalized two-dimensional coordinates (x,y) of the image pixels are projected onto the high-frequency space to obtain (x',y')=(x,y)×W; the sine and cosine functions of x' and y' are calculated respectively, namely sin(x'), cos(x'), sin(y'), cos(y'), and concatenated as the location code FEI_pos; FEI_pos is directly added to FEI_f to obtain the image feature embedding FEI_total after multi-scale and location fusion coding.

[0047] 3. Cross-modal fusion of image and text features

[0048] FEI and FET are input into the image-to-text cross-attention fusion module, which contains two sub-modules: image-to-text attention and text-to-image attention. In the image-to-text attention sub-module, FEI is used as the query and FET as the key and value for attention calculation; in the text-to-image attention sub-module, FET is used as the query and FEI as the key and value for attention calculation. The original features of FEI and FET are preserved and concatenated with the attention calculation results of the two sub-modules to obtain the multimodal fusion feature embedding FE_MM. This fusion feature can better achieve the alignment of visual and linguistic semantics.

[0049] 4. Target prediction decoding based on fused features

[0050] The FE_MM is concatenated with FEI and FET and input into the prediction decoding module. The prediction decoding module employs a positional decoding strategy, using a multilayer perceptron (MLP) to predict the coordinates (x1, y1, x2, y2) of the target box, where (x1, y1) are the coordinates of the top-left corner of the target box and (x2, y2) are the coordinates of the bottom-right corner. The confidence score of the corresponding target box is then calculated using a softmax function. Finally, a set of candidate detection boxes, Boxes_pred, is obtained, each containing its spatial location and confidence score information.

[0051] 5. Filtering and prompting high-confidence targets

[0052] Candidate boxes in `Boxes_pred` are sorted from highest to lowest confidence, and background predictions are excluded, selecting the top-k high-confidence target boxes (e.g., k=10). For these selected target boxes, their spatial features are calculated: the position (k×4) of each target box is shaped according to the coordinates of the top-left (x1, y1) and bottom-right (x2, y2) to obtain a 2k×2 feature vector; this feature vector is then encoded into a 2k×192 feature vector using an MLP (Multi-Level Processing), with an input dimension of 2 and an output dimension of 192. Simultaneously, the geometric attributes of each target box are extracted, including width (w=x2-x1), height (h=y2-y1), area (s=w×h), and aspect ratio (r=w / h), forming a 4-dimensional vector. This 4-dimensional vector is then encoded into a k×64 feature vector using another MLP (with an input dimension of 4 and an output dimension of 64). The two encoded features are concatenated to obtain a 2k×192+k×64 feature, which is then concatenated with the corresponding image-text fusion feature FE_MM. A single-layer MLP is used to reduce the dimensionality to 2k×256, resulting in the sparse cue embedding P_sparse.

[0053] 6. Mask prediction decoding based on sparse cues

[0054] The FEI_total and P_sparse are input into a mask prediction module based on a modified SAM structure. This module uses P_sparse as a cue to guide the model to focus on the region specified by the target bounding box. Simultaneously, it utilizes multi-scale image features provided by FEI_total to model texture and spatial information, generating pixel-level segmentation masks through multi-layer Transformer and convolution operations. The final output is a semantic segmentation mask that matches the detected target, completing the detection and segmentation of defects in industrial images.

[0055] Compared with traditional industrial defect detection and segmentation methods, the results of the comparison between the present invention and traditional methods are shown in the table below:

[0056] plan Processing image resolution Processing time Model concatenation (goudingdino + MobileSAM) 6192*14220 5420ms Invention Solution 6192*14220 702ms

[0057] Therefore, the present invention has the following significant advantages:

[0058] 1. Support for open category defect recognition: By integrating images and natural language prompts, it can recognize undefined defect categories, improving adaptability in open scenarios.

[0059] 2. Multi-scale fusion enhances texture representation: By using feature weighting and position encoding modules, the ability to model fine-grained texture and structural information is improved, thereby enhancing the detection and localization accuracy of minute defects.

[0060] 3. Cross-modal information deep alignment: Employs a cross-attention mechanism between images and text to achieve deep integration between visual and linguistic modalities, thereby enhancing multimodal understanding and localization capabilities.

[0061] 4. Prompt-driven mask prediction mechanism: Combined with the improved segmentation mechanism of the SAM architecture, it can realize accurate mask generation based on the detection target. At the same time, compared with the direct concatenation of models, the efficiency is significantly improved, meeting the requirements of high precision and high efficiency in industrial quality inspection.

[0062] The above description is only a specific embodiment of the present invention. Various examples and illustrations do not constitute a limitation on the substantive content of the present invention. Those skilled in the art can make modifications or variations to the above-described specific embodiments after reading the specification without departing from the substance and scope of the invention.

Claims

1. An industrial image quality inspection method integrating open set detection and segmentation, characterized in that: Includes the following steps, S1. Image and text data input and encoding: Input a three-channel industrial image I and a defect descriptive language prompt T. The image encoding module extracts features from I to obtain the image feature embedding FEI. The text encoding module performs semantic embedding on T to obtain the semantic feature embedding FET. S2. Spatial location and multi-scale texture fusion encoding of image features: Input FEI into the multi-scale adaptive feature weighting fusion module to generate a fused unified scale feature map FEI_f, and then send FEI_f into the random location encoding module for spatial information enhancement to obtain the image feature embedding FEI_total after multi-scale and location fusion encoding. S3. Cross-modal fusion of image and text features: Deep modal interaction is performed between FEI and the FET input image and text cross-attention fusion module to obtain multimodal fusion feature embedding FE_MM; S4. Target prediction decoding with fused features: Combine FE_MM with FEI and FET and input them into the prediction decoding module to predict the candidate detection box Boxes_pred; S5. Filtering and generating high-confidence targets: Filter the top-k high-confidence non-background target boxes from Boxes_pred, extract their corresponding FE_MM and spatial location features, and generate sparse hint embedding P_sparse. S6. Mask prediction and decoding based on sparse cueing: FEI_total and P_sparse are fed into the mask prediction module based on the modified SAM structure to generate a semantic segmentation mask that matches the detection target.

2. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S1, the image encoding module uses a Swing Transformer or an Efficient-Swin Transformer visual backbone network; the text encoding module uses a BERT or CLIP pre-trained language model.

3. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S2, the processing of the multi-scale adaptive feature weighted fusion module includes: setting a learnable weight vector consistent with the number of feature maps, and weighting the feature maps at each scale according to their importance; using convolution operations to unify the feature maps at each scale to the maximum number of channels; using bilinear interpolation to align the feature map sizes; and summing the weighted data to generate FEI_f.

4. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S2, the processing procedure of the random position encoding module includes: initializing a learnable transformation matrix of Gaussian distribution, projecting the normalized two-dimensional coordinates onto the high-frequency space; calculating the sine and cosine functions of the projected coordinates respectively, and concatenating them as the position encoding FEI_pos; adding FEI_pos to FEI_f to obtain FEI_total.

5. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S3, the image-text cross-attention fusion module includes two directions: image-to-text attention and text-to-image attention. It retains the original image and text features and uses them together with the cross-attention result as an intermediate fusion representation to obtain FE_MM.

6. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S4, the prediction decoding module uses a location decoding strategy to predict the target box coordinates and corresponding confidence scores to obtain Boxes_pred.

7. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S5, the calculation process of the spatial location features includes: shaping the positions of the top-k candidate boxes according to the upper left and lower right coordinates to obtain feature vectors, and encoding them through MLP; extracting the geometric attributes of the candidate boxes to form vectors and encoding them; concatenating the two sets of encoded features with FE_MM, and obtaining P_sparse by single-layer MLP dimensionality reduction.

8. The industrial image quality inspection method integrating open set detection and segmentation as described in claim 1, characterized in that: In step S6, the mask prediction module uses P_sparse as a prompt and FEI_total to provide texture and spatial support to generate a pixel-level segmentation mask.

Images