A method for annotating semantic segmentation datasets based on public large models

Through the joint graphic learning method of Swin Transformer and BERT, combined with the SAM model, the problem of traditional semantic segmentation methods depend on a large number of labeled data is solved, and efficient and accurate semantic segmentation data set annotation is achieved, improving the labeling efficiency and accuracy.

CN118570480BActive Publication Date: 2024-11-08JIANGSU ZHENGFANG TRANSPORTATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411054108.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Traditional computer vision methods rely on a large amount of labeled data in semantic segmentation tasks, resulting in high labeling costs, time-consuming and difficult to achieve both efficiency and accuracy, especially when processing complex image data.

Method used

Using the joint graphic learning method based on Swin Transformer and BERT, the image feature extraction, text feature extraction, feature fusion and multi-convolution layer annotation prediction are used to realize efficient semantic segmentation data set annotation, using the loss function L (cross entropy) Loss and Dice Loss) optimize prediction results.

Benefits of technology

Reliance on large amounts of labeled data is reduced, labeling efficiency and accuracy is improved, and more accurate and comprehensive labeling results are obtained with less data through text prompts, improving the quality and usability of image data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570480B_ABST
    Figure CN118570480B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of computer vision technology, and in particular to a method for annotating a semantic segmentation data set based on a public large model. By using Swin Transformer and BERT as a backbone network, the features of images and texts are effectively extracted, and the annotation prediction and image segmentation technologies are combined to improve the efficiency and accuracy of annotation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for annotating a semantic segmentation data set based on a public large model. Background Art

[0002] In the field of computer vision, technologies such as image recognition, image segmentation, and target detection have been widely used in many industries such as medicine, security, and autonomous driving, greatly promoting the improvement of social productivity. However, behind these technologies, there is a large amount of high-quality annotated data to support them.

[0003] Although traditional computer vision methods can achieve good performance, they often rely on large-scale labeled data sets for training. This not only increases the cost of data collection and processing, but the labeling work itself is also time-consuming and labor-intensive. Especially when processing complex image data, it is often difficult to achieve both accuracy and efficiency in labeling.

[0004] As an important task in the field of computer vision, semantic segmentation requires accurate classification of each pixel in an image. Although traditional semantic segmentation methods have achieved certain results, they still face many difficulties in practical applications due to their strong dependence on labeled data. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention provides a method for annotating semantic segmentation datasets based on a public large model, which is a method for annotating semantic segmentation datasets with text prompts based on joint learning of images and texts and efficient instance segmentation. The method uses the backbone network Swin Transformer for images and the backbone network BERT for texts to extract features, and implements the annotation of datasets through a series of calculation steps.

[0006] In order to achieve the purpose of the present invention, the technical solution adopted is: a method for annotating a semantic segmentation dataset based on a public large model, comprising the following steps:

[0007] S1. Image feature extraction: Use Swin Transformer as the backbone network of the image to extract features from the input image and obtain image features F;

[0008] S2, text feature extraction: add text prompts corresponding to the input image, use BERT as the backbone network of the text, extract features from the text prompts, and obtain text features T;

[0009] S3, feature fusion: fuse the image feature F and the text feature T to obtain the fused feature G;

[0010] S4. Based on the fusion feature G, multiple convolutional layers are used to perform annotation prediction to obtain the prediction result Y. The coordinates in the prediction result Y form a bounding box.

[0011] S5, passing the coordinates in the prediction result Y into the SAM model, and then obtaining the mask of the predicted image to obtain the final image segmentation coordinates;

[0012] S6, drawing the final image segmentation coordinates and bounding box on the original image and saving them to the output directory;

[0013] S7. Save the mask data and bounding box information of the predicted image as an image and a labeled file in JSON format.

[0014] As an optimization scheme of the present invention, the difference between the prediction result and the true annotation is measured by the loss function L;

[0015] L = L CE + λL Dice

[0016]

[0017]

[0018] Where: L CE is the cross entropy loss, L Dice is the Dice loss, λ is the weight for balancing the cross entropy loss and the Dice loss, N is the number of categories, and Y i is the i-th pixel of the prediction result, is the i-th pixel of the real annotation, Y true is the real segmentation map.

[0019] As an optimization solution of the present invention, in step S4, the model of the prediction result Y is:

[0020] Y = W × G + b

[0021] Where: W is the weight matrix of the convolutional layer, b is the bias term; the prediction result Y includes the category of the target and the coordinates of the prediction result Y.

[0022] As an optimization scheme of the present invention, the model parameters are adjusted by gradient descent to minimize the loss function L. In each iteration, the gradient of the loss function L with respect to the model parameters is calculated, and then the model parameters are updated to reduce the loss. The back propagation algorithm is executed to update W and b of the model according to the gradient.

[0023] As an optimization solution of the present invention, in step S1, it specifically includes:

[0024] S1-1, initialize the Swin Transformer model and set the parameters of the Swin Transformer model, including the number of layers, the number of heads and the resolution of the model;

[0025] S1-2, preprocessing the input image I, including scaling and normalization to meet the input requirements of Swin Transformer;

[0026] S1-3. Use Swin Transformer to extract features from the input image I. Swin Transformer gradually extracts multi-scale features of the image through self-attention mechanism and hierarchical representation. For each layer, the calculation formula can be expressed as:

[0027] F i = Swin(F i-1 )

[0028] Among them, F i is the feature representation of the i-th layer, F i-1 It is the feature representation of the previous layer;

[0029] S1-4, Multi-scale feature fusion: After extracting multi-scale feature representations, features at different layers can be combined through feature fusion technology to obtain richer feature representations;

[0030] S1-5. After multi-scale feature fusion, the final image feature representation F is obtained.

[0031] As an optimization solution of the present invention, in step S1-4, it specifically includes:

[0032] S1-4-1. Build a feature pyramid, where each level represents a feature map of different resolutions. The low level provides fine detail information, and the high level provides abstract global information.

[0033] S1-4-2, increase the resolution of the high-level feature map through upsampling operations to match the size of the low-level feature map. The upsampling operations include bilinear interpolation and transposed convolution;

[0034] S1-4-3, directly concatenate the upsampled high-level features with the low-level features;

[0035] S1-4-4, weighted sum of features at different levels through learnable weights;

[0036] S1-4-5, use a specific network structure to fuse features at different levels to obtain multi-scale features;

[0037] S1-4-6. After fusing the multi-scale features, it is usually necessary to further extract features through some refinement operations including additional convolutional layers, normalization layers or activation functions;

[0038] S1-4-7, multi-scale predictions are integrated by voting or averaging to obtain the final prediction result;

[0039] S1-4-8, introduce cross-scale connection, a bidirectional information flow mechanism, so that high-level features can influence low-level decisions, and low-level detailed information can also be fed back to high-level, thereby enhancing high-level feature representation;

[0040] S1-4-9. By introducing the attention mechanism, the most important parts of features of different scales are dynamically selected for fusion.

[0041] As an optimization solution of the present invention, in step S2, it specifically includes:

[0042] S2-1, preprocessing text: preprocess the input text prompt P, including word segmentation, adding special tags, generating corresponding input ID and attention mask;

[0043] S2-2. Load the pre-trained BERT model: Load the pre-trained BERT model, which can capture the complex patterns of language;

[0044] S2-3, Feature extraction: Input the preprocessed text into the BERT model, which processes the text layer by layer and generates an embedding representation for each word. These embedding representations can capture the contextual information of the word.

[0045] S2-4. Extract global feature representation from the text through pooling operation; obtain the global feature representation of the text to obtain text feature T.

[0046] As an optimization scheme of the present invention, the pooling operation is one of labeling pooling, average pooling, maximum pooling and attention pooling;

[0047] Tag pool: The embedding vector corresponding to the CLS tag is usually used as the representation of the entire sentence. The CLS tag is usually located at the beginning of the sequence, and its embedding vector is optimized during the pre-training process to represent the entire input sequence;

[0048] Average pooling: average the embedding vectors of all words to obtain a global text feature representation;

[0049] Max Pooling: Select max pooling to highlight the most important features in the text;

[0050] Attention Pooling: Use the attention mechanism to weight the embeddings of different words to generate a global feature representation.

[0051] As an optimization solution of the present invention, in step S3, the image feature F and the text feature T are concatenated along the feature dimension, G = Concatenate(F, T).

[0052] As an optimization solution of the present invention, in step S3, the interaction relationship between features is captured by element-by-element multiplication, weighted sum or using a network layer, G = Interaction(F, T).

[0053] The present invention has positive effects: 1) The present invention uses text prompts to annotate semantic segmentation datasets, which reduces the reliance on a large amount of annotated data, greatly reduces the burden of data annotation, and achieves a dual improvement in annotation efficiency and accuracy. By introducing text prompts as auxiliary information, the present invention can obtain more accurate and comprehensive annotation results with less annotated data, thereby improving the quality and availability of image data;

[0054] 2) The present invention uses Swin Transformer and BERT as the backbone network. Through their organic combination, the present invention can capture key information in images and texts more accurately and efficiently, further improving the performance and stability of annotation, and expanding the application boundaries of deep learning in the field of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0056] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The implementation effect of the method is described by a specific operation process. This implementation case is implemented based on the technical solution of the present invention, but the protection scope of the present invention is not limited to the following embodiments.

[0058] like Figure 1 As shown, the present invention discloses a method for annotating a semantic segmentation dataset based on a public large model, comprising the following steps:

[0059] S1. Image feature extraction: Use Swin Transformer as the backbone network of the image to extract features of the input image and obtain image features F.

[0060] Swin Transformer is a visual model based on the Transformer architecture. It enhances the representation ability of the model by introducing hierarchical representation and shift mechanism. The design of Swin Transformer takes into account the local and global features of the image, enabling it to process image data more efficiently. The following are the steps for feature extraction using Swin Transformer as the image backbone network:

[0061] S1-1, initialize the Swin Transformer model and set the parameters of the Swin Transformer model, including the number of layers, the number of heads, and the resolution of the model;

[0062] S1-2, preprocessing the input image I, including scaling and normalization to meet the input requirements of Swin Transformer;

[0063] S1-3. Use Swin Transformer to extract features from the input image I. Swin Transformer gradually extracts multi-scale features of the image through self-attention mechanism and hierarchical representation. For each layer, the calculation formula can be expressed as:

[0064] F i = Swin(F i-1 )

[0065] Among them, F i is the feature representation of the i-th layer, F i-1 It is the feature representation of the previous layer;

[0066] S1-4, Multi-scale feature fusion: After extracting multi-scale feature representations, features at different layers can be combined through feature fusion technology to obtain richer feature representations;

[0067] In step S1-4, it specifically includes:

[0068] S1-4-1. Build a feature pyramid, where each level represents a feature map of different resolutions. The low level provides fine detail information, and the high level provides abstract global information.

[0069] S1-4-2, increase the resolution of the high-level feature map through upsampling operations to match the size of the low-level feature map. The upsampling operations include bilinear interpolation and transposed convolution;

[0070] S1-4-3, directly concatenate the upsampled high-level features with the low-level features;

[0071] S1-4-4, weighted sum of features at different levels through learnable weights;

[0072] S1-4-5. Use specific network structures (such as feature pyramid network FPN, U-Net or multi-scale attention module) to fuse features at different levels to obtain multi-scale features. These network structures effectively combine high-level semantic information and low-level detail information through upsampling, skip connections and adaptive weight allocation, thereby obtaining feature representations containing rich multi-scale information;

[0073] S1-4-6. After fusing multi-scale features, it is usually necessary to further extract features (useful information) through some refinement operations including additional convolutional layers, normalization layers or activation functions.

[0074] S1-4-7, Multi-scale predictions are fused by voting or averaging to obtain the final prediction result; predictions are generated at different scales. These predictions can be fused to improve performance.

[0075] S1-4-8, introduce cross-scale connection, a bidirectional information flow mechanism, so that high-level features can influence low-level decisions, and low-level detailed information can also be fed back to high-level, thereby enhancing high-level feature representation;

[0076] S1-4-9. By introducing the attention mechanism, the most important parts of features of different scales are dynamically selected for fusion.

[0077] The purpose of multi-scale feature fusion is to make full use of the features extracted by the network at different levels. In this way, both the local details and the overall structure of the image can be taken into account at the same time, thereby obtaining a more comprehensive feature representation, so as to improve the model's ability to understand the image content.

[0078] S1-5. After multi-scale feature fusion, the final image feature representation F is obtained.

[0079] S2. Text feature extraction: Add text prompts corresponding to the input image, use BERT as the backbone network of the text, extract features from the text prompts, and obtain text features T.

[0080] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained deep bidirectional representation model based on Transformer. It learns the deep semantic information of language by pre-training on a large amount of text data. The bidirectional characteristics of the BERT model enable it to consider both the left and right sides of the context at the same time, thereby generating richer word embedding representations.

[0081] The following are the steps to extract features from the input text prompt using BERT as the text backbone network:

[0082] S2-1. Preprocessing text: Preprocess the input text prompt P, including tokenization, adding special tags (such as [CLS] and [SEP] tags), and generating the corresponding input ID and attention mask.

[0083] S2-2. Load the pre-trained BERT model: Load the pre-trained BERT model, which can capture the complex patterns of language and has been trained on a large-scale corpus.

[0084] S2-3, Feature extraction: Input the preprocessed text into the BERT model, which processes the text layer by layer and generates an embedding representation for each word. These embedding representations can capture the contextual information of the word.

[0085] S2-4. Extract global feature representation from the text through pooling operation; obtain the global feature representation of the text to obtain text feature T.

[0086] The pooling operation is one of label pooling, average pooling, maximum pooling and attention pooling. The pooling operation is used to achieve a balance between effect and computational performance.

[0087] Token pooling [CLS]: The embedding vector corresponding to the CLS token is usually used as the representation of the entire sentence. The CLS token is usually located at the beginning of the sequence, and its embedding vector is optimized during the pre-training process to represent the entire input sequence;

[0088] Average pooling: average the embedding vectors of all words to obtain a global text feature representation;

[0089] Max Pooling: Select max pooling to highlight the most important features in the text;

[0090] Attention Pooling: Use the attention mechanism to weight the embeddings of different words to generate a global feature representation.

[0091] Get the global feature representation T of the text. This representation can be used for various downstream tasks such as text classification, question answering, named entity recognition, etc.

[0092] S3, feature fusion: fuse the image feature F and the text feature T to obtain the fused feature G; the fusion is performed in the following two ways:

[0093] 1) Concatenate the image feature F and the text feature T along the feature dimension. This method is simple and intuitive, and can directly merge the information of the two modalities together.

[0094] G = Concatenate(F, T).

[0095] 2) Capture the interaction between features through element-wise multiplication, weighted sum, or using network layers (such as convolutional layers, attention mechanisms, etc.).

[0096] G = Interaction(F, T)

[0097] S4. Based on the fusion feature G, multiple convolutional layers are used to perform annotation prediction to obtain the prediction result Y. The coordinates in the prediction result Y form a bounding box.

[0098] The model for predicting the outcome Y is:

[0099] Y = W × G + b

[0100] Where: W is the weight matrix of the convolutional layer, b is the bias term; the prediction result Y includes the category of the target and the coordinates of the prediction result Y. The prediction result Y is the annotation of the image. The output format is: target category, x1, y1, x2, y2.

[0101] S5. Pass the coordinates in the prediction result Y into the SAM model, and then obtain the mask of the predicted image to get the final image segmentation coordinates. Pass x1, y1, x2, and y2 into the SAM model for target segmentation. The SAM model will use its advanced image segmentation technology to refine and optimize the preliminary results to achieve accurate segmentation of each object in the image. The prediction result Y contains the category and coordinate information of the target. The coordinate information is passed to the SAM model, and the SAM model will automatically return the boundary coordinates of the object in the target box.

[0102] The difference between the predicted result and the true annotation is measured through the loss function L;

[0103] L = L CE + λL Dice

[0104]

[0105]

[0106] Where: L CE is the cross-entropy loss, L Diceis the Dice loss, λ is the weight for balancing the cross entropy loss and the Dice loss, N is the number of categories (for example, in image classification, the number of categories may be more than 1,000, corresponding to different object categories, such as dogs, cats, tables, chairs, cars, etc.), Y i is the i-th pixel of the prediction result, is the i-th pixel of the real annotation, Y true is the real segmentation map. The cross entropy loss function measures the difference between the probability distribution of the category predicted by the model and the probability distribution of the real label for each pixel. Dice loss is used to measure the similarity of two sample sets. The loss function of the present invention considers both the category distribution and the pixel-level similarity. By optimizing the pixel-level loss, the model can better learn and retain the details of the image, such as edges and textures.

[0107] The model parameters of the predicted result Y are adjusted by gradient descent to minimize the loss function L. In each iteration, the gradient of the loss function L with respect to the model parameters is calculated, and then the model parameters are updated to reduce the loss. The backpropagation algorithm is executed to update the model's W and b according to the gradient. Because there will be errors between the model's predicted results and the actual results, the model parameters are adjusted by the backpropagation algorithm to ensure that the predicted data is consistent with the actual data and improve the accuracy of the prediction.

[0108] Iterative training: Repeat the above process until the model's performance on the validation set no longer improves, or until the preset number of iterations is reached. In this way, the model can learn how to accurately predict the segmentation annotations of the image, thereby minimizing the difference between the predicted result Y and the true annotation Y true The difference between.

[0109] In addition, based on the annotation results, the annotated data can be manually checked and the erroneous data annotated by the model can be manually modified.

[0110] 1) Load and preprocess images.

[0111] 2) For a dataset containing 1,000 images, first use Swin Transformer to extract features from each image to obtain image features F.

[0112] 3) Input text prompts (enter the target to be detected, such as "Horse, louds, Grassers, Sky, Hill", etc.), and use BERT to extract features for the text prompts corresponding to each image to obtain text features T.

[0113] 4) Fuse the image feature F and the text feature T to obtain the fused feature G.

[0114] 5) Use the fused feature G to perform annotation prediction through multiple convolutional layers to obtain the prediction result Y (bounding box and target category).

[0115] 6) Pass the coordinates in the prediction result Y into the SAM model and obtain the mask of the predicted image (the boundary of the target) to obtain the final image segmentation coordinates.

[0116] 7) Draw the final image segmentation coordinates and bounding box on the original image and save to the output directory.

[0117] 8) Manually correct and fine-tune the annotation results.

[0118] 9) Save the mask data and bounding box information of the predicted image as an image and an annotation file in JSON format.

[0119] Among them: the bounding box provides the approximate location and size of the object, while the mask provides the precise shape and boundary of the object.

[0120] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for annotating a semantic segmentation dataset based on a public large model, characterized by: The steps include: S1. Image feature extraction: Use Swin Transformer as the backbone network of the image to extract features from the input image and obtain image features F; S2, text feature extraction: add text prompts corresponding to the input image, use BERT as the backbone network of the text, extract features from the text prompts, and obtain text features T; S3, feature fusion: fuse the image feature F and the text feature T to obtain the fused feature G; S4. Based on the fusion feature G, multiple convolutional layers are used to perform annotation prediction to obtain the prediction result Y. The coordinates in the prediction result Y form a bounding box. S5, passing the coordinates in the prediction result Y into the SAM model, and then obtaining the mask of the predicted image to obtain the final image segmentation coordinates; S6, drawing the final image segmentation coordinates and bounding box on the original image and saving them to the output directory; S7, saving the mask data and bounding box information of the predicted image as an image and a labeled file in JSON format; The difference between the predicted result and the true annotation is measured through the loss function L; L = L CE + λL Dice ; ; ; Where: L CE is the cross entropy loss, L Dice is the Dice loss, λ is the weight for balancing the cross entropy loss and the Dice loss, N is the number of categories, and Y i is the i-th pixel of the prediction result, is the i-th pixel of the real annotation, Y true is the true segmentation map; In step S4, the model of the prediction result Y is: Y = W × G + b; Where: W is the weight matrix of the convolutional layer, b is the bias term; the prediction result Y includes the category of the target and the coordinates of the prediction result Y; Adjust the model parameters through gradient descent to minimize the loss function L. In each iteration, calculate the gradient of the loss function L with respect to the model parameters, and then update the model parameters to reduce the loss. Perform the back propagation algorithm to update the model's W and b according to the gradient. In step S1, it specifically includes: S1-1, initialize the Swin Transformer model and set the parameters of the Swin Transformer model, including the number of layers, the number of heads and the resolution of the model; S1-2, preprocessing the input image I, including scaling and normalization to meet the input requirements of Swin Transformer; S1-3. Use Swin Transformer to extract features from the input image I. Swin Transformer gradually extracts multi-scale features of the image through self-attention mechanism and hierarchical representation. For each layer, the calculation formula is expressed as: F i = Swin(F i-1 ); Among them, F i is the feature representation of the i-th layer, F i-1 It is the feature representation of the previous layer; S1-4, multi-scale feature fusion: After extracting multi-scale feature representations, the features of different layers are combined through feature fusion technology to obtain richer feature representations; S1-5, after multi-scale feature fusion, the final image feature representation F is obtained; In step S1-4, it specifically includes: S1-4-1. Build a feature pyramid, where each level represents a feature map of different resolutions. The low level provides fine detail information, and the high level provides abstract global information. S1-4-2, increase the resolution of the high-level feature map through upsampling operations to match the size of the low-level feature map. The upsampling operations include bilinear interpolation and transposed convolution; S1-4-3, directly concatenate the upsampled high-level features with the low-level features; S1-4-4, weighted sum of features at different levels through learnable weights; S1-4-5. Use the network structure to fuse features at different levels to obtain multi-scale features; S1-4-6, after fusing the multi-scale features, further features are extracted through some refinement operations including additional convolutional layers, normalization layers or activation functions; S1-4-7, multi-scale predictions are integrated by voting or averaging to obtain the final prediction result; S1-4-8, introduce cross-scale connection, a two-way information flow mechanism, so that high-level features affect low-level decisions, and detailed information from low levels can also be fed back to high levels, thereby enhancing high-level feature representation; S1-4-9. By introducing the attention mechanism, the most important parts of features of different scales are dynamically selected for fusion.

2. According to claim 1, a method for annotating a semantic segmentation dataset based on a public large model is characterized by: In step S2, it specifically includes: S2-1, preprocessing text: preprocess the input text prompt P, including word segmentation, adding special tags, generating corresponding input ID and attention mask; S2-2. Load the pre-trained BERT model: Load the pre-trained BERT model, which can capture the complex patterns of language; S2-3, Feature extraction: Input the preprocessed text into the BERT model, which processes the text layer by layer and generates an embedding representation for each word. These embedding representations can capture the contextual information of the word. S2-4. Extract global feature representation from the text through pooling operation; obtain the global feature representation of the text to obtain text feature T.

3. The method for labeling a semantic segmentation dataset based on a public large model according to claim 2, characterized in that: The pooling operation is one of label pooling, average pooling, maximum pooling and attention pooling.

4. The method for labeling a semantic segmentation dataset based on a public large model according to claim 3, characterized in that: In step S3, the image feature F and the text feature T are concatenated along the feature dimension, G = Concatenate(F, T).

5. The method for labeling a semantic segmentation dataset based on a public large model according to claim 3, characterized in that: In step S3, the interaction between features is captured by element-wise multiplication, weighted sum, or using network layers, G = Interaction(F, T).