Box supervision cover abnormal segmentation method based on CLIP semantic guidance

Abstract

The invention discloses a CLIP semantic guidance-based box supervision cover anomaly segmentation system and method, belongs to the technical field of industrial automation and quality control, and solves the problems that pixel-level mask labeling cost is high and a box supervision method lacks semantic understanding. Extracting multi-scale visual features of the input image; s2, after mapping the multi-scale visual features through a linear projection layer, inputting the multi-scale visual features into a semantic anomaly flow, and performing semantic similarity calculation through embedding with normal and abnormal texts to generate a semantic anomaly thermodynamic diagram; s3, inputting the multi-scale visual features into a weak supervision segmentation stream, and generating a high-resolution prediction segmentation mask; s4, constructing a mixed anomaly guide loss function which fuses geometric constraint loss based on a bounding box and semantic guide loss based on the semantic anomaly thermodynamic diagram, and training the weak supervision segmentation flow end to end by using the loss function; and S5, discarding the semantic anomaly flow in a reasoning stage to realize efficient detection. According to the method, training is carried out only by using bounding box labeling, so that the data labeling cost is greatly reduced, and meanwhile, the segmentation accuracy is remarkably improved through semantic guidance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of industrial automation and quality control, and specifically relates to an automatic surface defect detection method based on computer vision. More specifically, the present application proposes a pixel-level anomaly segmentation method for manufacturing class covers (such as automobile body panels, consumer electronic device housings, injection molded parts, etc.), aiming to achieve high-precision defect region mask segmentation only by using low-cost bounding box labeling. BACKGROUND

[0002] In modern manufacturing, computer vision-based automated defect detection is a key technology to ensure product quality. However, the existing technology faces the "cost-precision paradox of data labeling": on the one hand, industrial applications urgently need high-precision pixel-level segmentation masks to accurately locate irregular defects; on the other hand, the process of obtaining such mask labels is extremely time-consuming and costly, which is difficult to achieve economically and timely.

[0003] To solve this problem, existing Box-Supervised Instance Segmentation (BSIS) methods (such as BoxInst) can be trained only by bounding boxes, but they lack advanced semantic understanding ability and are prone to misjudging normal visual features (such as logos or textures) within the bounding box as anomalies. At the same time, methods based on visual language models (VLM) have semantic understanding ability, but the positioning maps generated by them are usually rough and cannot meet the pixel-level precision requirements. Therefore, there is an urgent need in the field for a unified framework that combines the low cost of box supervision with the semantic understanding ability of VLM. SUMMARY

[0004] To solve the above-mentioned shortcomings in the prior art, i.e., the "cost-precision paradox of data labeling", and the pain points of existing box supervision methods (such as BoxInst) lacking semantic understanding and being prone to misjudging normal features as anomalies, the present application proposes a CLIP semantic guided box supervision cover anomaly segmentation method.

[0005] The present application proposes a neural network model that can be trained end-to-end, and its core goal is to achieve high-precision, pixel-level segmentation of industrial cover surface anomalies only by using low-cost bounding box labeling as a supervision signal.

[0006] To achieve the above-mentioned purposes, the present application adopts the following technical solutions: A CLIP semantic guided box supervision cover anomaly segmentation method, comprising the following steps: S1, multi-scale feature extraction: a shared pre-trained CLIP visual backbone network is used to extract multi-scale visual features of the input image; S2, semantic anomaly heat map generation: input the multi-scale visual features extracted in S1 into a semantic anomaly stream; the semantic anomaly stream generates a semantic anomaly heat map by performing semantic similarity calculation on the features and preset "normal" and "abnormal" text embeddings ; S3, predicted segmentation mask generation: input the multi-scale visual features extracted in S1 into a weakly supervised segmentation stream; the weakly supervised segmentation stream generates a high-resolution predicted segmentation mask through a lightweight segmentation head ; S4, mixed loss function construction and training: construct a mixed anomaly-guided loss function , which includes a geometric constraint loss based on bounding box labeling , and a semantic-guided loss based on the semantic anomaly heat map generated in S2 ; the weakly supervised segmentation stream is trained end-to-end by minimizing the mixed loss function; S5, inference: in the inference stage, discard the semantic anomaly stream and loss calculation module, and only keep the shared backbone network and the trained weakly supervised segmentation stream to generate the final abnormal segmentation mask.

[0007] Preferably, the specific process of step S1 is: S11, load a pre-trained CLIP Vision Transformer (ViT) model as a shared feature extractor, and set its parameters to be frozen; S12, extract multi-scale feature maps from multiple intermediateTransformer blocks of the ViT backbone network , where represents different stages.

[0008] Preferably, the specific process of step S2 is: S21, map the visual features of each level extracted in S1 to a joint embedding space aligned with CLIP text features through respective independent linear projection layers (defined by trainable weights and biases ) to achieve cross-modal semantic alignment, obtaining mapped features , which is mathematically expressed as: (1) S22, prepare text prompts for "normal" and "abnormal" states, and encode them into text embedding vectors and using a CLIP text encoder, and combine them into text features ; S23, calculating the similarity between the mapped image features and the text features , and fusing the multi-stage similarity maps with upsampling, thereby generating the semantic anomaly heat map The generation process can be summarized as follows: (2) wherein, denotes the fusion (e.g. weighted sum) of the feature maps of all stages, is an upsampling operation for restoring the similarity maps of different stages to a unified resolution, is a normalization function, is the mapped image feature in S21, is the transposed text feature in S22.

[0009] Preferably, the specific process of step S3 is as follows: S31, inputting the multi-scale feature maps extracted in S1 into a lightweight segmentation head; S32, the segmentation head adopts a feature pyramid network (FPN) structure for feature fusion to fuse high-level semantic information and low-level spatial details, followed by convolution layers and upsampling layers, and finally outputs the high-resolution predicted segmentation mask ; S33, the function of which can be summarized as follows: (3) wherein, is the multi-scale visual feature, is the feature pyramid network fusion function, is the segmentation head function.

[0010] Preferably, the mixed loss function of step S4 is constructed and trained, specifically including: S41, the mixed anomaly-guided loss function is defined as the weighted sum of three components: (4) wherein, is a hyperparameter for balancing the loss.

[0011] S42, the projection loss is a bounding box-based geometric constraint loss, which can be mathematically expressed as: (5) ​wherein and represent the projection operation along x-axis and y-axis respectively, is a distance metric function, which loss ensures that the geometric projection of the predicted mask is consistent with the projection of the ground truth bounding box .

[0012] S43, the pair-wise affinity loss is a low-level vision constraint loss based on image color prior, which is used to ensure that the pixels which are spatially adjacent and color or feature similar have the same segmentation label, and its mathematical expression is: (6) wherein, is the total number of edges in the set , represents an edge connecting two adjacent pixels, is the set of edges with at least one pixel inside the ground truth bounding box, is an indicator function, is the color similarity between the two pixels connected by the edge , is a pre-set color similarity threshold, is the probability that the model predicts the same label for the two pixels, and its calculation method is: (7) wherein and are the probability values of the predicted mask at the two end pixels and of the edge .

[0013] S44, the semantic guidance loss is a high-level semantic constraint loss based on semantic flow, which is used to minimize the consistency or distribution difference between the predicted segmentation mask and the semantic anomaly heat map . This loss term forces the output of the weakly supervised segmentation flow to align with the judgment of the semantic anomaly flow in terms of semantics, which is preferably implemented as a Dice loss, and its mathematical expression is: (8) or its implementation can also be a weighted binary cross-entropy loss.

[0014] S45, the training process is: In each training iteration, forward propagation is performed, and the ​​and ; according to the real bounding box and the generated calculate the total loss ; perform back propagation, only update the parameters of the linear projection layer of the semantic anomaly stream and all layers of the weakly supervised segmentation stream, and the shared backbone network parameters remain frozen.

[0015] Preferably, the inference process of step S5 specifically comprises: S51, in the inference deployment stage, completely discard the semantic anomaly stream defined in step S2, and the mixed anomaly guidance loss function defined in step S4 and all related calculation modules thereof; S52, only keep the shared backbone network defined in step S1 and the weakly supervised segmentation stream defined in step S3 and trained in step S4; S53, sequentially input the input image into the shared backbone network and the weakly supervised segmentation stream for single forward propagation, so as to generate the final anomaly segmentation mask, and realize efficient and lightweight inference deployment.

[0016] The present application has the following advantages: while ensuring high-quality output, the cost of data labeling is greatly reduced, which is the most core advantage of the present application. Through the innovative training paradigm, the present application realizes that only using economic and efficient bounding box labeling can achieve the segmentation accuracy that traditionally requires expensive pixel-level mask labeling. This directly solves the long-standing "data labeling paradox" in industrial applications, and clears the key cost barrier for large-scale deployment of high-precision visual detection technology. Achieving excellent segmentation accuracy through semantic guidance: the standard box supervision method is often confused by complex but normal surface patterns (such as logos, assembly holes, and specific textures) due to the lack of semantic understanding, resulting in false segmentation. The present application introduces CLIP to understand the semantics of "abnormalities" as a guidance signal (this signal comes from the semantic anomaly stream of the present application), which can effectively distinguish between real defects and normal visual features, thereby significantly reducing such false positive errors and improving the accuracy and reliability of segmentation. Enhancing the robustness and generalization ability of the model: since the system is built on a powerful pre-trained visual language model (CLIP), it naturally inherits the model's strong zero-shot generalization ability. This means that the present system can not only detect defect types that have been seen in training data, but also effectively identify and segment new types of abnormalities that have never appeared in the training phase. End-to-end architecture simplicity and inference efficiency: the present application is a unified, end-to-end trained single network model. Compared to a complex, multi-stage cascade process composed of multiple independent models like ClipSAM, the architecture of the present application is more concise. During inference, since only the lightweight segmentation stream is activated (the semantic anomaly stream is discarded), the computational overhead is small, enabling faster detection speed and better meeting the stringent real-time requirements of industrial production lines. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The system architecture diagram of the present application shows the parallel structure of the shared backbone network, the semantic anomaly stream, and the weakly supervised segmentation stream.

[0018] Figure 2 The training phase flowchart of the present application shows the calculation process of the hybrid loss function.

[0019] Figure 3 The inference phase flowchart of the present application shows the discarded components and the retained lightweight detection stream. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the following embodiments are used to further explain the present application. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0021] REFERENCE Figures 1 to 3A CLIP semantic-guided box supervision overlay abnormal segmentation system and method. Embodiments of the present invention include the following steps: S1, referring to Figure 1 The system of the present invention employs a shared pre-trained CLIP Vision Transformer (ViT) model (e.g., ViT-L / 14@336px) as the backbone network.

[0022] The specific process of step S1 is as follows: S11, backbone network loading and freezing: load a pre-trained CLIP ViT model and set all its parameters to be non-trainable (frozen). This is done to preserve the powerful, general-purpose visual representation capabilities and zero-shot generalization capabilities learned by the CLIP model from large-scale image-text pair data.

[0023] S12, multi-scale feature extraction: input the preprocessed image (e.g., scaled to `518 × 518` pixels) into the ViT backbone network. Extract feature maps from multiple intermediate layers of ViT (e.g., after the 6th, 12th, 18th, and 24th Transformer blocks) (where represents the stage index). These multi-scale feature maps contain rich spatial details (from shallow layers) and high-level semantic information (from deep layers), providing input for subsequent parallel processing streams.

[0024] S2, semantic anomaly heat map generation, referring to Figure 1 The upper half branch of the semantic anomaly stream ( Figure 1 ) is used to generate semantic guidance signals. The core of this step is to convert the intermediate visual features in the ViT backbone network that are not directly aligned with the text into a semantically interpretable anomaly heat map.

[0025] The specific process of S2 is as follows: S21, feature mapping and semantic alignment: since the dimensions of are usually different and not aligned in the joint space, the present invention introduces an independent, trainable linear projection layer (defined by weights and bias ) for each stage . This layer maps the visual features to the joint embedding space: (1) where is the mapped feature.

[0026] S22, Text Embedding Preparation: Define text prompts describing normal and abnormal states (e.g., prompts = ["a photo of a flawless covering part", "a photo of a damaged covering part"]). Encode these prompts using CLIP's text encoder to obtain text embedding vectors and and combine them into text features where is the joint embedding space dimension.

[0027] S23, Heatmap Generation: Generate anomaly maps for each stage by computing the similarity (e.g., matrix multiplication followed by Softmax) between the mapped image features and the text features . Finally, fuse (e.g., weighted sum) all stage anomaly maps and upsample to the input image resolution to obtain the final semantic anomaly heatmaps .

[0028] (2)

[0029] where, denotes the fusion (e.g., weighted sum) of feature maps for all stages , is an upsampling operation to restore the similarity maps of different stages to a unified resolution, is a normalization function, is the mapped image features in S21, is the transposed text features in S22.

[0030] S3: Predicted Segmentation Mask Generation, refer to Figure 1 , the lower half branch in the weakly supervised segmentation flow ( Figure 1 ) is used to generate the final pixel-level mask.

[0031] S31, Feature Fusion: The same multi-scale feature maps extracted in S12 are input into a lightweight Feature Pyramid Network (FPN) structure. FPN effectively fuses high-level semantic information and low-level spatial details to generate an information-rich high-resolution feature map.

[0032] S32, Mask Prediction: The fused feature map from FPN is input into a segmentation head (usually composed of several 3x3 convolution layers and upsampling layers). The final output of the segmentation head is a high-resolution, single-channel predicted segmentation mask ), whose pixel value represents the probability that the pixel belongs to the abnormal region. The training process of this data stream does not use any pixel-level mask annotation at all. The function of this step can be summarized by the following formula: (3) wherein is the multi-scale visual feature, is the feature pyramid network fusion function, is the segmentation head function.

[0033] S4, mixed loss function construction and training, refer to Figure 2 The technical core of the present application lies in the innovative mixed abnormality guiding loss function which organically combines two different types of supervision signals (geometric constraints from the bounding box and semantic constraints from S2 of the present application).

[0034] The specific process of step S4 is as follows: S41, the total loss function applied to the “predicted segmentation mask” generated in S3 is the weighted sum of the following three components: (4) wherein, is a hyperparameter used to balance the importance of each loss (for example, ).

[0035] S42, projection loss (This loss term is a geometric constraint, aiming to minimize the difference between the one-dimensional projection of the predicted mask on the horizontal and vertical axes and the projection of the real bounding box . This constraint ensures that the overall size, shape and position of the predicted mask strictly comply with the given bounding box annotation. Its mathematical expression can be formalized as: (5) wherein, is the predicted mask, is the real bounding box, and represent the projection operation on the x-axis and y-axis respectively, is a distance metric function (such as L1 loss).

[0036] S43, pairwise affinity loss This loss term is a low-level visual constraint that leverages the local consistency prior of the image, meaning that spatially adjacent pixels with similar colors (or depth features) are highly likely to belong to the same category (both foreground or both background). This loss encourages the model to generate masks with clear boundaries and smooth interiors.

[0037] (6)

[0038] in, For the set The total number of sides in the middle. This represents an edge connecting two adjacent pixels. It is the set of edges that have at least one pixel inside the actual bounding box. It is an indicator function. It is the edge Color similarity between two connected pixels. This is a preset color similarity threshold. This is the probability that the model predicts these two pixels have the same label, and it is calculated as follows: (7) Here, and It is the prediction mask On the side The two endpoint pixels and The probability value at that point, i.e. ; S44, Semantic Guidance Loss This loss term is a key innovation of this invention; it is a high-level semantic constraint. It directly measures the predicted segmentation mask generated in S3. Compared with the semantic anomaly heatmap generated in S2 The consistency between them. This loss term penalizes inconsistencies between the mask generated by the segmentation stream and the semantically anomalous regions identified by the semantic stream, thereby forcing the segmentation stream to learn "what is a real anomaly". Wherein: (8) S45. Training process, refer to... Figure 2 In each training iteration, the model performs parallel computation. (via S2) and (via S3). Then, based on the true bounding box... and generated Calculate total loss The gradient is propagated back to update only the parameters of the linear projection layer of the semantic stream (S22) and all layers of the segmentation stream (S3). The CLIP backbone network parameters in S1 are always kept frozen.

[0039] S5, reasoning, referring to Figure 3 The inference stage is optimized for industrial real-time detection. The specific process of step S5 is: S51, inference flow, as shown in Figure 3 When deploying the application, the semantic anomaly stream and all loss calculation parts (step S4) described in step S2 are discarded. The input image only needs to pass through the frozen CLIP backbone network (S1) and the trained weakly supervised segmentation stream (S3) to directly and efficiently generate the final high-quality anomaly segmentation mask.

[0040] S52, deployment optimization, the trained model (only the backbone network and the segmentation stream) can be optimized using tools such as NVIDIA TensorRT to speed up the inference process.

[0041] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A box-supervised overlay anomaly segmentation method based on CLIP semantic guidance, characterized in that, Includes the following steps: S1. Multi-scale feature extraction: A shared pre-trained CLIP visual backbone network is used to extract multi-scale visual features from the input image. S2. Semantic Anomaly Heatmap Generation: The multi-scale visual features extracted in step S1 are input into the semantic anomaly stream; the semantic anomaly stream generates a semantic anomaly heatmap by calculating the semantic similarity between the features and preset "normal" and "abnormal" text embeddings. ; S3. Predictive Segmentation Mask Generation: The multi-scale visual features extracted in S1 are input into the weakly supervised segmentation stream; the weakly supervised segmentation stream generates a high-resolution predictive segmentation mask through a lightweight segmentation head. ; S4. Construction and Training of Hybrid Loss Function: Constructing a hybrid anomaly-guided loss function. The loss function includes bounding box annotations. The geometric constraint loss, and the semantic anomaly heatmap generated in S2. The semantic guidance loss; the weakly supervised segmentation stream is trained end-to-end by minimizing the hybrid loss function; S5. Inference: During the inference phase, the semantic anomaly stream and loss calculation module are discarded, and only the shared backbone network and the trained weakly supervised segmentation stream are retained to generate the final anomaly segmentation mask.

2. The method as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11. Load a pre-trained CLIP Vision Transformer (ViT) model as a shared feature extractor and set its parameters to frozen. S12. Extract multi-scale feature maps from multiple intermediate Transformer blocks of the ViT backbone network. ,in Indicates different stages.

3. The method as described in claim 1, characterized in that, The specific process of step S2 is as follows: S21. Extract the visual features at each level from step S1. Through their respective independent linear projection layers (composed of trainable weights) and bias (Definition), mapped to a joint embedding space aligned with CLIP text features to achieve cross-modal semantic alignment, resulting in mapped features. Its mathematical expression is: （1） S22. Prepare text prompts for "normal" and "abnormal" states, and encode them into text embedding vectors using the CLIP text encoder. and and combine them into text features ; S23. Calculate the features of the mapped image. With the text features The similarity between them is calculated, and the multi-stage similarity maps are fused and upsampled to generate the semantic anomaly heatmap. Its generation process can be summarized by the following formula: （2） in, Indicates all stages The feature maps are fused (e.g., by weighted summation). It is an upsampling operation used to restore similarity maps from different stages to a uniform resolution. It is a normalization function. These are the mapped image features in S21. It is the text feature after transposition in S22.

4. The method as described in claim 1, characterized in that, The specific process of step S3 is as follows: S31, Extract the multi-scale feature map from S1 Input into a lightweight segmentation header; S32. The segmentation head uses a Feature Pyramid Network (FPN) structure for feature fusion to integrate high-level semantic information with low-level spatial details. This is followed by convolutional layers and upsampling layers, ultimately outputting the high-resolution predicted segmentation mask. ; S33, its function can be summarized by the following formula: （3） in For the aforementioned multi-scale visual features, For the feature pyramid network fusion function, This is the split header function.

5. The method as described in claim 1, characterized in that, Step S4, the construction and training of the hybrid loss function, specifically includes: S41, the hybrid anomaly-guided loss function Defined as a weighted sum of three components: （4） in, It is a hyperparameter used to balance various losses; S42, the projection loss The geometric constraint loss based on the bounding box is mathematically expressed as follows: （5） in and These represent the projection operations along the x-axis and y-axis, respectively. It is a distance metric function, and this loss ensures that the predicted mask... Geometric projection and true bounding box The projections are consistent; S43, the aforementioned pairwise affinity loss This is a low-level visual constraint loss based on image color prior, used to ensure that spatially adjacent pixels with similar colors or features have the same segmentation label. Its mathematical expression is: （6） in, For the set The total number of sides in the middle. This represents an edge connecting two adjacent pixels. It is the set of edges that have at least one pixel inside the true bounding box. It is an indicator function. It is the edge Color similarity between two connected pixels. The preset color similarity threshold, This is the probability that the model predicts these two pixels have the same label, and it is calculated as follows: （7） in and It is the prediction mask On the side The two endpoint pixels and The probability value at that location, i.e. ; S44, the semantic guidance loss A high-level semantic constraint loss based on semantic flow is used to minimize the predicted segmentation mask. With the aforementioned semantic anomaly heatmap The consistency or distributional differences between them are realized as Dice loss or weighted binary cross-entropy loss; S45. The training process is as follows: In each training iteration, forward propagation is performed, and computation is carried out in parallel. and ; Based on the actual bounding box and generated Calculate total loss ; Perform backpropagation, updating only the parameters of the linear projection layer of the semantically anomalous flow and all layers of the weakly supervised segmentation flow, while keeping the shared backbone network parameters frozen.

6. The method as described in claim 1, characterized in that, The reasoning in step S5 specifically includes: S51. Discard the semantic anomaly flow defined in step S2, the hybrid anomaly guidance loss function defined in step S4, and all related calculations. S52. Only retain the shared backbone network defined in step S1 and the weakly supervised segmentation stream defined in step S3 and trained in step S4; S53. The input image is sequentially fed into the shared backbone network and the weakly supervised segmentation stream for a single forward propagation, thereby generating the final anomaly segmentation mask and achieving efficient and lightweight inference deployment.

Images