A CLIP-based method for Zero-Shot automatic seat belt segmentation

Abstract

This invention discloses a method for automatic zero-shot segmentation of seat belts based on CLIP, comprising: preprocessing the work image; extracting features of each image block and text prompt word using CLIP's image and text encoders respectively; calculating the similarity between each text feature vector and the feature vectors of each image block to obtain n similarity matrices corresponding to n text prompt words; selecting image blocks as positive and negative sample blocks according to the similarity matrices and mapping them to the original image to obtain positive and negative sample pixel sets; clustering the positive sample pixel sets corresponding to each text prompt word to separate multiple different instances; segmenting the instances obtained from all text words using a SAM model, and the positive sample pixel set included in the instances of seat belt type is the seat belt obtained from image segmentation. This invention achieves accurate seat belt segmentation without requiring labeled data to train the model.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a method for automatic zero-shot segmentation of seat belts based on CLIP. Background Technology

[0002] During power construction, workers are required to perform work at heights on towers. To ensure their safety, safety belts must be worn. Therefore, safety belt testing is proposed to improve operational safety.

[0003] Currently, most seatbelt detection methods (such as R-CNN and YOLO models) rely on manually labeled data for supervised training, thus consuming significant manpower and time for image acquisition and annotation. When detecting seatbelts using object detection techniques, the background within the detection bounding box is often part of the human body, making it difficult to determine if the model has learned seatbelt features, leading to many false positives. Therefore, accurate seatbelt segmentation is crucial for determining whether the seatbelt is being worn correctly. Furthermore, similarity maps obtained directly from CLIP prompts often exhibit contrasting visualization areas (i.e., focusing more on the background and ignoring the foreground), and a single prompt may contain multiple objects. This results in poor performance when CLIP is directly combined with SAM for segmentation based on region points. Summary of the Invention

[0004] This invention provides a method for achieving zero-shot automatic segmentation of seat belts based on CLIP, which achieves accurate segmentation of seat belts without the need for labeled data to train the model.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] A method for implementing Zero-Shot automatic seatbelt splitting based on CLIP includes:

[0007] Step 1: Preprocess the work image. Use the image encoder and text encoder in the CLIP model to extract the features of the preprocessed image blocks and text prompts, respectively, to obtain the image feature vector of each image block and the text feature vector corresponding to each of the n text prompts; where the n text prompts correspond to n instance types, and the n instance types include at least the seat belt.

[0008] Step 2: Calculate the similarity between each text feature vector and all image block feature vectors to obtain n similarity matrices corresponding to the n text prompt words;

[0009] Step 3: Select image blocks as positive and negative sample blocks based on the similarity matrix, and map them to the preprocessed image to obtain a set of positive and negative sample pixels;

[0010] Step 4: Cluster the positive sample pixel set corresponding to each text prompt word to separate the multiple different instances corresponding to each text prompt word;

[0011] Step 5: Use the SAM model to segment the image region where each instance of each text word is located, and obtain the region including the seat belt segmentation region.

[0012] Furthermore, the attention module of the image encoder adopts a vv self-attention module.

[0013] Furthermore, step 1 also includes denoising the extracted text feature vectors and using the denoised text feature vectors for subsequent processing; wherein, the denoising process is as follows: using a text encoder to extract the features of empty text prompt words, which are denoised as redundant text features; then, the extracted text feature vectors corresponding to each text prompt word are subtracted from the redundant text features to obtain the denoised text feature vectors of each text prompt word.

[0014] Furthermore, cosine similarity is used to calculate similarity in step 2.

[0015] Furthermore, in step 3, the image blocks are selected as positive and negative sample blocks based on the similarity matrix. Specifically, the image blocks corresponding to the similarity elements in the similarity matrix that are greater than a set threshold are all taken as positive sample blocks, and the number of positive sample blocks is denoted as m. At the same time, the image blocks corresponding to the m smallest similarity elements in the similarity matrix are all taken as negative sample blocks.

[0016] Further, step 3 maps the positive and negative sample blocks to the preprocessed image to obtain a set of positive and negative sample pixels, specifically as follows:

[0017] First, calculate the two-dimensional scaling factor from the preprocessed image to the similarity matrix:

[0018] scale_h=float(h) / sm.shape[0], scale_w=float(w) / sm.shape[1]

[0019] Wherein, float() means converting an integer to a floating-point number, h and w represent the height and width of the preprocessed image, sm represents the similarity matrix, sm.shape[0] and sm.shape[1] represent the size of the 0th and 1st dimensions of the similarity matrix sm, and the 0th and 1st dimensions correspond to the height and width directions, respectively; scale_h and scale_w represent the scaling factors in the width and height directions, respectively;

[0020] Then, the center pixel (x, y) of each image block is found based on the two-dimensional scaling factor:

[0021] x=min((idx%sm.shape[1]+0.5)*scale_w,w-1)

[0022] y=min((idx / / sm.shape[1]+0.5)*scale_h,h-1)

[0023] Where x and y are the coordinates of the center pixel in the width and height directions in the preprocessed image, respectively, idx represents the sequence number of the image block, % represents the modulo operation, and / / represents the integer division operation;

[0024] All pixels in the image block corresponding to each center pixel in the preprocessed image are assigned positive and negative sample values ​​according to the positive and negative sample label values ​​of the image block.

[0025] Furthermore, in step 4, the density-based spatial clustering method DBSCAN is used for clustering, and the mean of the contour coefficients of all positive sample pixels is used to determine whether the current clustering has converged; the formula for calculating the contour coefficient of each positive sample pixel is:

[0026]

[0027] Where s(i) is the contour coefficient of positive sample pixel i, a(i) is the average distance of positive sample pixel i to other sample pixels in its cluster, and b(i) is the minimum average distance of positive sample pixel i to positive sample pixels in other clusters.

[0028] Further, step 5 uses the SAM model to segment the instances corresponding to each text word. Specifically, for each cluster obtained from the clustering in step 4, its minimum bounding rectangle is calculated. Then, the negative sample pixels within the minimum bounding rectangle are found. The prediction function SamPredictor.predict in the SAM model is then used to segment the image within the minimum bounding rectangle based on the positive and negative sample pixels within the minimum bounding rectangle.

[0029] Furthermore, the n instance types in step 1 also include human bodies; after obtaining the regions of human body type and seat belt type through segmentation in step 5, the regions of human body type and seat belt type are merged, and then a morphological closing operation is performed on the merged region. Then, the closing operation result is ANDed with the segmentation result obtained in step 5 to obtain a refined segmentation result.

[0030] Beneficial effects

[0031] This invention employs a zero-shot training method to directly segment the target image into seatbelts. It innovatively applies a large text-image model to the seatbelt segmentation problem, solving the issue of traditional detection and segmentation requiring extensive manual annotation, while also saving model training resources. Specifically, this is reflected in:

[0032] The acquired images are directly processed by CLIP's image encoder and text encoder to extract features from the images and text respectively. The relevant point locations are found through the text-image feature relationship. Finally, the relevant points and the original image information are combined and input into the SAM model to directly obtain the segmented region. The whole process does not require labeled data or model training, saving manpower and computing power.

[0033] The seatbelt segmentation method used in this invention is more accurate than the detection boxes predicted by object detection methods, down to the pixel level. Due to the shape and occlusion issues of seatbelts, existing recognition methods can only rely on object detection; therefore, this invention provides significantly more accurate seatbelt segmentation results. Attached Figure Description

[0034] Figure 1 This is a general framework diagram of the segmented seat belt according to an embodiment of this application;

[0035] Figure 2 This is a schematic diagram illustrating the principle of feature similarity matching in the embodiments of this application.

[0036] Figure 3 This is a similarity matching heatmap of an embodiment of the present invention applied to real high-altitude operation images;

[0037] Figure 4 The safety belt area is shown in the embodiment of the present invention applied to real high-altitude operation images.

[0038] Figure 5 This is a segmentation result diagram of an embodiment of the present invention applied to real high-altitude operation images. Detailed Implementation

[0039] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0040] This embodiment provides a method for automatic zero-shot segmentation of seat belts based on CLIP, achieving accurate seat belt segmentation without requiring labeled data to train the model. (See reference...) Figure 1 , Figure 2 As shown, it includes the following steps:

[0041] Step 1: Preprocess the work image. Use the image encoder and text encoder in the CLIP model to extract the features of the preprocessed image blocks and text prompts, respectively, to obtain the image feature vector of each image block and the text feature vector corresponding to each of the n text prompts; where the n text prompts correspond to n instance types, and in this embodiment, the n instance types include seat belt, human body, head, hand, and foot.

[0042] The images captured by the drone have a large field of view and high resolution, and can be processed in segments, such as dividing them into four regions: upper left, upper right, lower left, and lower right, and adjusting the size of each region to 512*512; the RGB3 channels are standardized channel by channel.

[0043] Load the CLIP model Vit-B / 16, which belongs to the ViT (VisionTransformer) model family. Adjust the image attention module in the CLIP model: change the original qk self-attention module to a vv self-attention module, which can effectively solve the problem that the attention region extracted by CLIP is biased towards areas outside the target (such as the seat belt). That is, attn qk =softmax(Q·K) T ·scale)·V is adjusted to: Attn vv = softmax(V·V) T ·scale)·V.

[0044] The image feature vector (I_f) is extracted using the image encoder in the CLIP model, and the text feature vector (T_f) corresponding to the text prompt (e.g., seat belt) is extracted using the text encoder in the CLIP model. The image feature vector (I_f) and the text feature vector (T_f) are then standardized to obtain the corresponding I_e and T_e.

[0045] Since the features extracted by CLIP from empty text prompts are similar to the noise features extracted from other texts, the features extracted from empty string text are used as redundant features T_r to subtract from the aforementioned standardized text feature vectors, thereby completing the denoising process of the text feature vectors.

[0046] Step 2: Calculate the similarity between each text feature vector and all image block feature vectors to obtain n similarity matrices corresponding to the n text prompt words.

[0047] In this embodiment, the 512*512 region image is divided into blocks, each block being 16*16 in size, resulting in a total of 32*32 image blocks.

[0048] For each text prompt word, the cosine similarity between its text feature vector and the feature vector of each image block is calculated, resulting in n similarity matrices corresponding to the n text prompt words, and the size of each similarity matrix is ​​32*32.

[0049] Step 3: Select image blocks as positive and negative sample blocks based on the similarity matrix, and map them to the preprocessed image to obtain a set of positive and negative sample pixels.

[0050] The image block filtering process is as follows: Since a higher similarity indicates a better match between the image block and the text prompt, the image blocks corresponding to similarity elements in the similarity matrix that are greater than a set threshold (e.g., the threshold t is 0.8) are all taken as positive sample blocks, and the number of positive sample blocks is denoted as m. At the same time, the image blocks corresponding to the m smallest similarity elements in the similarity matrix are all taken as negative sample blocks.

[0051] The specific steps for obtaining the set of positive and negative sample pixels through mapping are as follows:

[0052] First, calculate the two-dimensional scaling factor from the preprocessed image to the similarity matrix:

[0053] scale_h=float(h) / sm.shape[0], scale_w=float(w) / sm.shape[1]

[0054] What does float() represent? h and w represent the height and width of the preprocessed image, respectively. sm represents the similarity matrix. sm.shape[0] and sm.shape[1] represent the size of the 0th and 1st dimensions of the similarity matrix sm, respectively. The 0th and 1st dimensions correspond to the height and width directions, respectively. scale_h and scale_w represent the scaling factors in the width and height directions, respectively.

[0055] Then, the center pixel (x, y) of each image block is found based on the two-dimensional scaling factor:

[0056] x=min((idx%sm.shape[1]+0.5)*scale_w,w-1)

[0057] y=min((idx / / sm.shape[1]+0.5)*scale_h,h-1)

[0058] Where x and y are the coordinates of the center pixel in the width and height directions in the preprocessed image, respectively, idx represents the sequence number of the image block, % represents the modulo operation, and / / represents the integer division operation;

[0059] All pixels in the image block corresponding to each center pixel in the preprocessed image are assigned positive and negative sample values ​​according to the positive and negative sample label values ​​of the image block.

[0060] Step 4: Cluster the positive sample pixel set corresponding to each text prompt word to separate the multiple different instances corresponding to each text prompt word.

[0061] In this embodiment, the density-based spatial clustering DBSCAN method is used for clustering, and the mean of the contour coefficients of all positive sample pixels is used to determine whether the current clustering has converged. If it has converged, the clustering ends.

[0062] The contour coefficient of each positive sample pixel is calculated using the following formula:

[0063]

[0064] Where s(i) is the contour coefficient of positive sample pixel i, a(i) is the average distance of positive sample pixel i to other sample pixels in its cluster, and b(i) is the minimum average distance of positive sample pixel i to positive sample pixels in other clusters.

[0065] Since n text prompts correspond to n instance types, we can obtain multiple instances for each of the n instance types. For example, if we cluster the human body type and get 3 clusters, it means there are 3 human bodies in the image.

[0066] Step 5: Use the SAM model to segment the image region where each instance of each text word is located, and obtain the region including the seat belt region.

[0067] For each cluster obtained from the clustering in step 4, calculate its minimum bounding rectangle, then find the negative sample pixels within the minimum bounding rectangle, and then use the prediction function SamPredictor.predict in the SAM model to perform region segmentation of the image within the minimum bounding rectangle based on the positive and negative sample pixels within the minimum bounding rectangle.

[0068] Since the positive and negative sample pixels are obtained by first downsampling the original image to extract features, and then upsampling and mapping them back to the original image, some features will have positional offset errors, resulting in some interference in SAM segmentation. Therefore, in a better embodiment, after segmenting into various types of regions in step 5, the human body type and seat belt type regions are merged. Then, a morphological closing operation is performed on the merged region, and the closing operation result is ANDed with the segmentation result obtained in step 5 to remove disconnected regions (mainly interference caused by reflections from nearby towers, etc.), thus obtaining a refined segmentation result.

[0069] refer to Figure 3 This is a similarity matching heatmap of an embodiment of the method of the present invention applied to real high-altitude operation images. Positive and negative sample blocks of the region image are mapped onto the original image through bilinear interpolation.

[0070] refer to Figure 4 The image shows the feature points in the lower right corner where workers are wearing safety belts after instance clustering. It can be seen that most feature points are accurate, with a few showing deviations due to linear interpolation during the mapping of the feature map to the original image.

[0071] refer to Figure 5 This is a segmentation result image of an embodiment of the method of the present invention applied to real high-altitude operations. Red dots represent the head, legs, and feet areas, while blue dots represent the safety belt areas.

[0072] The results show that although the method of this invention does not perform model training on seat belts, it can accurately identify the segmentation region of seat belts, providing basic support for subsequent practical applications such as identifying whether seat belts are worn correctly.

[0073] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A method for implementing Zero-Shot automatic seatbelt splitting based on CLIP, characterized in that, include: Step 1: Preprocess the work image. Use the image encoder and text encoder in the CLIP model to extract the features of the preprocessed image blocks and text prompts, respectively, to obtain the image feature vector of each image block and the text feature vector corresponding to each of the n text prompts; where the n text prompts correspond to n instance types, and the n instance types include at least the seat belt. Step 2: Calculate the similarity between each text feature vector and all image block feature vectors to obtain n similarity matrices corresponding to the n text prompt words; Step 3: Select image blocks as positive and negative sample blocks based on the similarity matrix, and map them to the preprocessed image to obtain a set of positive and negative sample pixels; Step 4: Cluster the positive sample pixel set corresponding to each text prompt word to separate the multiple different instances corresponding to each text prompt word; Step 5: Use the SAM model to segment the image region where each instance of each text word is located, and obtain the region including the seat belt region.

2. The method according to claim 1, characterized in that, The attention module of the image encoder adopts the v-vself-attention module.

3. The method according to claim 1, characterized in that, Step 1 also includes denoising the extracted text feature vectors and using the denoised text feature vectors for subsequent processing. The denoising process is as follows: the text encoder is used to extract the features of the empty text prompt words, which are denoised as redundant text features. Then, the extracted text feature vectors corresponding to each text prompt word are subtracted from the redundant text features to obtain the denoised text feature vectors of each text prompt word.

4. The method according to claim 1, characterized in that, In step 2, cosine similarity is used to calculate the similarity.

5. The method according to claim 1, wherein In step 3, image blocks are selected as positive and negative sample blocks based on the similarity matrix. Specifically, the image blocks corresponding to similarity elements in the similarity matrix that are greater than a set threshold are all taken as positive sample blocks, and the number of positive sample blocks is denoted as m. At the same time, the image blocks corresponding to the m smallest similarity elements in the similarity matrix are all taken as negative sample blocks.

6. The method according to claim 1, characterized in that, Step 3 maps the positive and negative sample blocks to the preprocessed image to obtain a set of positive and negative sample pixels, specifically: First, calculate the two-dimensional scaling factor from the preprocessed image to the similarity matrix: scale_h=float(h) / sm.shape[0], scale_w=float(w) / sm.shape[1] Wherein, float() means converting an integer to a floating-point number, h and w represent the height and width of the preprocessed image, sm represents the similarity matrix, sm.shape[0] and sm.shape[1] represent the size of the 0th and 1st dimensions of the similarity matrix sm, and the 0th and 1st dimensions correspond to the height and width directions, respectively; scale_h and scale_w represent the scaling factors in the width and height directions, respectively; Then, the center pixel (x, y) of each image block is found based on the two-dimensional scaling factor: x=min((idx%sm.shape[1]+0.5)*scale_w,w-1) y=min((idx / / sm.shape[1]+0.5)*scale_h,h-1) Where x and y are the coordinates of the center pixel in the width and height directions in the preprocessed image, respectively, idx represents the sequence number of the image block, % represents the modulo operation, and / / represents the integer division operation; All pixels in the image block corresponding to each center pixel in the preprocessed image are assigned positive and negative sample values ​​according to the positive and negative sample label values ​​of the image block.

7. The method according to claim 1, characterized in that, Step 4 employs the density-based spatial clustering method DBSCAN for clustering, and determines whether the clustering has converged based on the mean of the contour coefficients of all positive sample pixels; the formula for calculating the contour coefficient of each positive sample pixel is as follows: Where s(i) is the contour coefficient of positive sample pixel i, a(i) is the average distance of positive sample pixel i to other sample pixels in its cluster, and b(i) is the minimum average distance of positive sample pixel i to positive sample pixels in other clusters.

8. The method according to claim 1, characterized in that Step 5 uses the SAM model to segment the instances corresponding to each text word. Specifically, for each cluster obtained from the clustering in Step 4, its minimum bounding rectangle is calculated. Then, the negative sample pixels within the minimum bounding rectangle are found. The prediction function SamPredictor.predict in the SAM model is then used to segment the image within the minimum bounding rectangle based on the positive and negative sample pixels within the minimum bounding rectangle.

9. The method according to claim 1, characterized in that, The n instance types in step 1 also include human body; after obtaining the regions of human body type and seat belt type through segmentation in step 5, the regions of human body type and seat belt type are merged, and then a morphological closing operation is performed on the merged region. Then, the closing operation result is ANDed with the segmentation result obtained in step 5 to obtain the refined segmentation result.

Images