A text-visual reference segmentation method based on a three-stage training strategy

Abstract

The application discloses a kind of based on three-stage training strategy training text-visual reference segmentation method, the model trained by this method can strengthen the text-based reference segmentation capability of SAM model.Model adopts image text pair and pure object detection data, compared with the data specially annotated for text-visual reference segmentation task, two data types are more easily obtained, and their huge data volume and diverse sources provide more extensive text and image feature alignment field for model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, and specifically to a text-visual reference segmentation method based on a three-stage training strategy. Background Technology

[0002] In the field of computer vision, image segmentation is a core and crucial task. Its main objective is to subdivide digital images into multiple parts or regions to facilitate further image analysis and processing. Image segmentation plays a key role in various applications, such as medical imaging, robot navigation, scene understanding, and augmented reality.

[0003] Traditional image segmentation methods typically rely on explicit visual cues, such as bounding boxes or dot markers. However, this approach faces challenges when dealing with complex segmentation tasks based on text descriptions, as it requires the model to understand and parse natural language descriptions and associate them with image content.

[0004] Recently, Meta's SAM (Segment Anything Model) has shown significant potential in this field, handling various types of input cues. Its flexibility and strong visual understanding capabilities make it perform well in many situations. However, SAM has performance limitations when handling segmentation tasks that rely on text descriptions, especially when the descriptions become complex or ambiguous, at which point the model's performance degrades. Given the importance of text-guided image segmentation in various applications, this limitation becomes a key challenge. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method for training text-visual referential segmentation based on a three-stage training strategy. The model trained by this method can enhance the text-based referential segmentation capability of the SAM model.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for training text-visual referential segmentation based on a three-stage training strategy, characterized by comprising the following steps: Step 1: Input the image into the image encoder, extract the key visual features in the image, and represent these features as a series of feature vectors; Step 2: Input the text description into the text encoder to convert it into a natural language description that corresponds to the semantic space representation of the image feature vector; Step 3: Use a mask generator to fuse the key visual features extracted by the image encoder in Step 1 and the semantic space representation corresponding to the image feature vector converted by the text encoder to generate an image segmentation mask corresponding to the text description. Step four involves optimization using a three-stage training strategy, including coarse-grained text-image alignment training, fine-grained text-image alignment training, and final fine-grained mask output training. Then, the mask generation process is optimized using the dice loss function and the focus loss function.

[0007] As a further improvement of the present invention, the optimization method in step four using a three-stage strategy also includes: introducing a multi-layer adaptive training mechanism, whereby the learning rate of each layer in the text encoder is dynamically adjusted according to its performance during the training process. Specifically, if a layer shows a good performance improvement, its learning rate is reduced accordingly to avoid overfitting; conversely, if a layer performs poorly, its learning rate is increased to promote faster learning.

[0008] As a further improvement of the present invention, the image encoder in step one has a feature enhancement step, which specifically involves increasing the weight of key elements in the feature vector while reducing the weight of less important elements, so that the image encoder pays more attention to visual information that is highly relevant to the text description when processing complex images.

[0009] As a further improvement of the present invention, the mask generator in step three has an adaptive fusion step, specifically by dynamically adjusting the weight distribution of the feature vectors output by the text encoder and the image encoder when generating the mask. If the correlation between the text feature vector and the image feature vector is high, the weight of the text feature vector is increased; otherwise, the weight of the image feature vector is increased.

[0010] As a further improvement to the present invention, the optimization in step four using a three-stage training strategy is as follows: Step 41: Perform coarse-grained ITM / ITC training. Use coarse-grained text and image data to train the data and establish basic text and image alignment relationships. Step 42: Perform fine-grained alignment training. Use fine-grained image and text data to train the text encoder, further enhancing its ability to understand the relationship between images and text at a more refined level. Step 43: Perform fine-grained mask output training. Use fine-grained images and text to train the mask generator in order to optimize the mask generation process and ensure that the output mask not only accurately reflects the semantic information of the text, but also closely integrates with the image features.

[0011] The beneficial effects of this invention lie in its three-stage training strategy, which effectively combines the advantages of image-text pairs and object detection data, significantly improving the adaptability and accuracy of the text-visual reference segmentation model (OmSAM) across various application scenarios. Initially, a large-scale image-text pair dataset is used to establish an understanding of the relationship between images and basic text descriptions, crucially providing the model with basic processing capabilities for multimodal data. In the middle stage, fine-grained training using object detection data strengthens the alignment between text and images, enhancing the model's ability to recognize and process details in complex scenes. Finally, an advanced loss function optimizes the generation of the segmentation mask, resulting in high-quality segmentation results. This process not only enhances the model's semantic understanding and segmentation accuracy but also improves its versatility and adaptability across various data types. In short, OmSAM, through its innovative training method, has become a highly efficient and adaptable multimodal segmentation tool. Attached Figure Description

[0012] Figure 1 This is a diagram of the text-visual referential segmentation model architecture based on a three-stage training strategy. Figure 2 This is the activation map for the model when extracting intermediate features from the input sentence "a woman with golden hair". Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0014] Reference Figure 1 As shown in this embodiment, a method for training text-visual referential segmentation based on a three-stage training strategy is presented, aiming to achieve text-guided image segmentation according to the present invention. In this example, we will detail the architecture of the OmSAM model and its application in text-guided image segmentation tasks. The attached figure shows the overall architecture of the OmSAM model, including the image encoder, text encoder, and mask generator. The model architecture is as follows: Image Encoder: In this example, the image encoder is initialized using pre-trained weights from the SAM model, trained on a large-scale visual dataset. The primary function of the image encoder is to extract image features, including visual information such as color, texture, and shape. These image features are transformed into a series of high-dimensional feature vectors containing key visual information about the image. During training, the image encoder's weights are frozen to leverage the powerful visual recognition capabilities of the pre-trained model.

[0015] Text Encoder: In this example, the RoBERTa model is used for initialization. RoBERTa is an advanced natural language processing model that excels at capturing the deep semantics of language. The task of the text encoder is to convert the input text description into a semantic feature vector that matches the image feature vector. To enhance the interaction between text and image, cross-attention modules are introduced. These modules are initially randomly set to adaptively learn the association between text and image features during training. The role of the cross-attention modules is to incorporate visual features provided by the image encoder when the text encoder processes text information, thereby achieving more accurate text-image alignment.

[0016] The three-stage training strategy: In this example, the first stage is coarse-grained ITM / ITC training, mainly focusing on training the text encoder, while the mask generator is not involved. In this stage, the model is trained using coarse-grained image-text data to establish basic image-text alignment relationships. The focus in this stage is to enable the model to understand basic image-text relationships. By processing coarse-grained data, the model learns to match basic image features with text descriptions. The second stage is fine-grained alignment training, which further strengthens the text encoder, focusing on more refined alignment between images and text. In this stage, the model learns how to identify and understand the relationship between images and text at a more precise level. This stage increases the complexity of training, further strengthening the text encoder's ability to understand the relationship between images and text at a more granular level through fine-grained data. The third stage is fine-grained mask output training, shifting the focus to optimizing the mask generation process to ensure that the output mask not only accurately reflects the semantic information of the text but also closely integrates with image features. In the final stage, the focus shifts to how to generate accurate segmentation masks. At this point, the text encoder and mask generator work together to ensure that the output mask conforms to both the text semantics and is closely related to the image information.

[0017] Mask Generator: In this example, after all training phases are complete, the mask generator combines the outputs of the image encoder and text encoder to generate an image segmentation mask corresponding to the text description. It uses dice loss and focal loss to optimize the segmentation results, ensuring highly accurate and efficient segmentation performance. Specifically, in this example, it combines the outputs of the image encoder and text encoder to generate an accurate image segmentation mask. This example specifically uses dice loss to optimize the consistency between the mask and the true segmentation target. This example specifically uses focal loss to handle difficult-to-segment regions, maintaining high segmentation accuracy.

[0018] To further elaborate on the specific implementation methods of the present invention based on the above architecture, such as... Figure 2As shown, the following will use a specific text description example, "a woman with golden hair," as guidance information, and images taken in real-life scenarios as the processing objects to detail the application of the OmSAM model in implementing text-guided image segmentation tasks.

[0019] In the image encoder section, the input image is processed by the image encoder to extract visual features such as color, texture, and shape. For example, for an image containing a blonde woman, the "color" feature vector will highlight the hues contained in the image, while the "shape" feature will identify the outline of the person.

[0020] In the text encoder part, the RoBERTa model is used to convert the text description "a woman with golden hair" into the corresponding semantic feature vector. Since the text encoder contains cross-attention modules, these modules have learned how to combine the semantic information in the text description with the image features during training. Therefore, the resulting semantic feature vector is aligned with the image feature vector.

[0021] Because we have implemented a three-stage training strategy for the model, the model can gradually increase its complexity. First, it coarsely identifies the corresponding basic regions in the image, such as a woman's head. Then, it further refines the processing of more detailed information in the text description, such as matching "golden hair" more precisely with the corresponding hair color in the image. Finally, the model can accurately generate masks, such as precisely covering the blonde hair of a woman in the image, rather than other regions or people with other hair colors.

[0022] In the OmSAM model, through joint learning of image and text encoders and optimization of the cross-attention mechanism, the model can perform deep semantic parsing of the input natural language description and combine it with the visual features of the image. Through this comprehensive multimodal processing, when the model receives a text prompt such as "a woman with golden hair," it can activate the corresponding visual recognition module in the intermediate layer, focusing attention on the region of the blonde woman in the image. Based on this, the mask generator outputs a fine-grained image segmentation mask according to these focused features, which accurately covers the target region in the image—in this example, the blonde woman.

[0023] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for training text-visual reference segmentation based on a three-stage training strategy, characterized in that: Includes the following steps: Step 1: Input the image into the image encoder, extract the key visual features in the image, and represent these features as a series of feature vectors; Step 2: Input the text description into the text encoder to convert it into a natural language description that corresponds to the semantic space representation of the image feature vector; Step 3: Use a mask generator to fuse the key visual features extracted by the image encoder in Step 1 and the semantic space representation corresponding to the image feature vector converted by the text encoder to generate an image segmentation mask corresponding to the text description. Step four involves optimization using a three-stage training strategy, including coarse-grained text-image alignment training, fine-grained text-image alignment training, and final fine-grained mask output training. Then, a dice loss function and a focus loss function are applied to optimize the mask generation process. The specific steps of the three-stage training strategy optimization in step four are as follows: Step 41: Perform coarse-grained ITM / ITC training. Use coarse-grained text and image data to train the data and establish basic text and image alignment relationships. Step 42: Perform fine-grained alignment training. Use fine-grained image and text data to train the text encoder, further enhancing its ability to understand the relationship between images and text at a more refined level. Step 43: Perform fine-grained mask output training. Use fine-grained images and text to train the mask generator in order to optimize the mask generation process and ensure that the output mask not only accurately reflects the semantic information of the text, but also closely integrates with the image features.

2. The method for training text-visual referential segmentation based on a three-stage training strategy according to claim 1, characterized in that: The optimization method using the three-stage strategy in step four also includes: introducing a multi-layer adaptive training mechanism, whereby the learning rate of each layer in the text encoder is dynamically adjusted based on its performance during training. Specifically, if a layer shows a good performance improvement, its learning rate is reduced accordingly to avoid overfitting; conversely, if a layer performs poorly, its learning rate is increased to promote faster learning.

3. The method for training text-visual referential segmentation based on a three-stage training strategy according to claim 2, characterized in that: The image encoder in step one includes a feature enhancement step, which specifically increases the weight of key elements in the feature vector while decreasing the weight of less important elements, so that the image encoder focuses more on visual information that is highly relevant to the text description when processing complex images.

4. The method for training text-visual referential segmentation based on a three-stage training strategy according to claim 3, characterized in that: The mask generator in step three has an adaptive fusion step, which specifically involves dynamically adjusting the weight distribution of the feature vectors output by the text encoder and the image encoder when generating the mask. If the text feature vector and the image feature vector have a high correlation, the weight of the text feature vector is increased; otherwise, the weight of the image feature vector is increased.

Images