Multimodal zero-shot image segmentation model training method and device

Abstract

The present application relates to the field of computer vision processing technology, and provides a method and device for training a zero-sample image segmentation model based on multimodality. The method comprises: using an image encoder, a prompt word encoder, and a mask decoder to train a basic segmentation module to obtain a trained basic segmentation module, and using a language model, a text encoder, and a saliency detection module to train a segmentation fine-tuning module to obtain a trained segmentation fine-tuning module, splicing the trained basic segmentation module with the segmentation fine-tuning module and connecting the cross-attention fusion module and the preset decoder to obtain a zero-sample image segmentation model, and using a training set to train the zero-sample image segmentation model again to obtain a trained zero-sample image segmentation model. The embodiment of the present application solves the problem that the prior art has low accuracy in segmentation results for unseen object types and small objects.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to a multimodal zero-sample image segmentation model training method and device. Background Art

[0002] Image segmentation technology has a wide range of applications in computer vision and image processing. For example, it can be used to separate different targets or objects in an image from the background, classify different areas in an image, or be applied to behavioral analysis in surveillance systems. During the training phase of an image segmentation model, sample data representing all possible target categories is often required to accurately classify pixels in an image into different categories.

[0003] However, in practical applications, new target categories are often encountered without corresponding sample data for training. Therefore, the zero-shot segmentation algorithm is introduced to solve the zero-shot problem in segmentation tasks. The zero-shot segmentation algorithm can learn a model by using sample data of known categories and auxiliary information related to the target category (such as semantic descriptions, attribute features, etc.) to match pixels in the image with known categories and use the auxiliary information for reasoning, thereby segmenting pixels of unknown categories in the image. It is precisely because the segmentation model is not exposed to samples of relevant data during the training process that the zero-shot image segmentation model does not have sufficient understanding of the features and shapes of new objects or small objects, resulting in inaccurate output results for images containing unseen object types and small objects.

[0004] Therefore, the existing technology has the problem that the segmentation results of unseen object types and small objects are not accurate enough. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a multimodal zero-shot image segmentation model training method and device to solve the problem in the prior art that the segmentation results for unseen object types and small objects are not accurate.

[0006] In a first aspect of an embodiment of the present application, a multimodal zero-shot image segmentation model training method is provided, comprising: obtaining an image from an image training set and a prompt word from a prompt word training set, wherein the prompt word is determined based on the content of the image; inputting the image and the prompt word into a basic segmentation module, processing the image using an image encoder to obtain image coding features, and processing the prompt word using a prompt word encoder to obtain prompt word coding features; processing the image coding features and the prompt word coding features through a mask decoder to obtain a corresponding mask, processing the mask through a fully connected layer and calculating a first loss function, training the basic segmentation module based on the first loss function to obtain a trained basic segmentation module; inputting the image into a segmentation fine-tuning module, and performing segmentation fine-tuning on the image respectively. Cross-modal processing and saliency detection processing are performed to obtain text embedding features and salient feature maps, and the text embedding features and the salient feature maps are cross-attentively fused to obtain a first fused feature map; the first fused feature map is processed by a multi-layer perceptron and a fully connected layer and the second loss function is calculated, and the segmentation fine-tuning module is trained based on the second loss function to obtain a trained segmentation fine-tuning module; the trained basic segmentation module and the trained segmentation fine-tuning module are spliced, and the cross-attention fusion module and the preset decoder are connected to obtain a zero-sample image segmentation model; the image and the prompt word are input into the zero-sample image segmentation model for processing, and the third loss function is calculated, and the zero-sample image segmentation model is trained based on the third loss function to obtain a trained zero-sample image segmentation model.

[0007] According to a second aspect of an embodiment of the present application, a multimodal zero-sample image segmentation model training device is provided, comprising: an acquisition module configured to acquire images in an image training set and prompt words in a prompt word training set, wherein the prompt words are determined based on the content of the images; a first processing module configured to input the images and prompt words into a basic segmentation module, process the images using an image encoder to obtain image coding features, and process the prompt words using a prompt word encoder to obtain prompt word coding features; a first training module configured to process the image coding features and the prompt word coding features through a mask decoder to obtain corresponding masks, process the masks through a fully connected layer and calculate a first loss function, train the basic segmentation module based on the first loss function to obtain a trained basic segmentation module; and a second processing module configured to input the images into a segmentation fine-tuning module to perform image segmentation fine-tuning on the images. The first training module is configured to process the first fusion feature map through a multi-layer perceptron and a fully connected layer and calculate the second loss function, and train the segmentation fine-tuning module based on the second loss function to obtain a trained segmentation fine-tuning module; the third processing module is configured to splice the trained basic segmentation module and the trained segmentation fine-tuning module, and connect the cross-attention fusion module and the preset decoder to obtain a zero-sample image segmentation model; the third training module is configured to input the image and the prompt word into the zero-sample image segmentation model for processing, and calculate the third loss function, and train the zero-sample image segmentation model based on the third loss function to obtain a trained zero-sample image segmentation model.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0010] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0011] By constructing a basic segmentation module and a segmentation fine-tuning module, the basic segmentation module is trained using an image training set and a prompt word training set to obtain a trained basic segmentation module. The segmentation fine-tuning module is then trained using the image training set to obtain a trained segmentation fine-tuning module. The trained basic segmentation module and the trained segmentation fine-tuning module are then concatenated and connected to a cross-attention fusion module and a preset decoder to obtain a zero-shot image segmentation model. The zero-shot image segmentation model is then trained using an image training set and a prompt word training set to obtain a trained zero-shot image segmentation model. An adapter module is configured within the image encoder in the basic segmentation fine-tuning module. This adapter module is used to update the parameters of the image encoder. This adapter module is used to train the image encoder through parameter updates of the adapter module, resulting in an image encoder with enhanced semantic understanding capabilities. Multimodal comparison is achieved by training the model using images and prompt words, thereby improving model performance. This allows the trained zero-shot image segmentation model to accurately segment unseen object types and small objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a flowchart of a multimodal zero-shot image segmentation model training method provided in an embodiment of the present application;

[0014] Figure 2 This is a flowchart of another multimodal zero-shot image segmentation model training method provided in an embodiment of the present application;

[0015] Figure 3 1 is a flowchart of another multimodal zero-shot image segmentation model training method provided in an embodiment of the present application;

[0016] Figure 4 This is a flowchart of another multimodal zero-shot image segmentation model training method provided in an embodiment of the present application;

[0017] Figure 5 1 is a flowchart of another method for training a zero-shot image segmentation model based on multimodality provided in an embodiment of the present application;

[0018] Figure 6 1 is a structural diagram of a multimodal zero-shot image segmentation model training device provided in an embodiment of the present application;

[0019] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0021] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0022] In addition, it should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprises..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0023] A multimodal zero-shot image segmentation model training method and apparatus according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart of a multimodal zero-sample image segmentation model training method provided in an embodiment of the present application. Figure 1 As shown, the multimodal zero-shot image segmentation model training method includes:

[0025] S101, obtaining an image in an image training set and a prompt word in a prompt word training set, where the prompt word is determined based on the content of the image;

[0026] S102: Input the image and prompt word into the basic segmentation module, process the image using an image encoder to obtain image coding features, and process the prompt word using a prompt word encoder to obtain prompt word coding features;

[0027] S103, processing the image coding features and the prompt word coding features through a mask decoder to obtain a corresponding mask, processing the mask through a fully connected layer and calculating a first loss function, training the basic segmentation module based on the first loss function to obtain a trained basic segmentation module;

[0028] S104: Input the image into the segmentation and fine-tuning module, perform cross-modal processing and saliency detection on the image to obtain text embedding features and salient feature maps, and perform cross-attention fusion on the text embedding features and the salient feature maps to obtain a first fused feature map.

[0029] S105, processing the first fused feature map through a multi-layer perceptron and a fully connected layer and calculating a second loss function, and training the segmentation fine-tuning module based on the second loss function to obtain a trained segmentation fine-tuning module;

[0030] S106, splicing the trained basic segmentation module and the trained segmentation fine-tuning module, and connecting the cross attention fusion module and the preset decoder to obtain a zero-shot image segmentation model;

[0031] S107: Input the image and the prompt word into the zero-sample image segmentation model for processing, calculate the third loss function, and train the zero-sample image segmentation model based on the third loss function to obtain a trained zero-sample image segmentation model.

[0032] During the training process of the zero-shot image segmentation model, the basic segmentation module can be trained using the image training set and the prompt word training set, and the segmentation fine-tuning module can be trained using the image training set to obtain the trained basic segmentation module and the trained segmentation fine-tuning module. The two can be spliced ​​and connected to the cross-attention fusion module and the preset decoder to obtain a complete zero-shot image segmentation model. Then, the zero-shot image segmentation model can be trained using the image training set and the prompt word training set to obtain the trained zero-shot image segmentation model. The trained zero-shot image segmentation model can be used to perform segmentation processing on the image to be segmented, and during the processing process, it can accurately segment out unseen object types and small objects.

[0033] During the training process of the basic segmentation module, the image encoder of the pre-trained segmentation model can be directly used to deploy the adapter module. During training, all parameters of the original structure of the image encoder can be frozen, and the adapter module can be used as a module for parameter update during the training process of the image processor.

[0034] In some embodiments, an image is processed using an image encoder to obtain image coding features, including: using multiple converter modules to process the image to obtain image coding features, the converter module includes a first normalization layer, a multi-head attention layer, an adapter module, a second normalization layer and a multi-layer perceptron; wherein, the output of the multi-head attention layer is processed according to the adapter module to obtain the input of the second normalization layer, and the adapter module is configured as a parameter update module of the image encoder.

[0035] Specifically, the image encoder can be obtained by splicing multiple converter modules, and there is no specific restriction on the number of converter modules here. In the process of image processing by the image encoder, multiple first normalization layers, multi-head attention layers, adapter modules, second normalization layers and multi-layer perceptrons can be used to process the input of the image encoder in sequence to obtain image coding features. It can be understood that the converter module, such as the transformer module, can accordingly include Layer Norm, Multi-head Attention, Layer Norm, MLP modules and adapter modules. The adapter can be inserted after the Multi-head Attention, so it can be understood that the output of the multi-head attention layer is processed as the input of the adapter module to obtain the input of the second normalization layer.

[0036] It should be noted that the original structure of each converter module can include a first normalization layer, a multi-head attention layer, a second normalization layer and a multi-layer perceptron. When constructing the basic segmentation module, the embodiment of the present application can deploy an adapter module after the multi-head attention layer of the image encoder and use the adapter module as the parameter update module of the image encoder.

[0037] The structures and processing effects of the first normalization layer and the second normalization layer may be consistent.

[0038] According to the solution provided in the above embodiment, a combination of converter modules is used to construct an image encoder. By using the converter modules multiple times, the image encoder can gradually extract rich feature representations, thereby better understanding the image content. Such image coding features can be used for subsequent image segmentation tasks. During the training process of the image encoder, the configured adapter module is also used as a parameter update module, which can also enhance the expressiveness and adaptability of the image encoder, thereby improving the training effect and generalization ability of the subsequently constructed zero-sample image segmentation model to enhance the performance of the model.

[0039] In other embodiments, the output of the multi-head attention layer is processed according to the adapter module to obtain the input of the second normalization layer, including: in the adapter module, using the first linear layer, the compression linear layer, the activation layer, the recovery linear layer and the second linear layer to process the output of the multi-head attention layer in sequence to obtain the input of the second normalization layer; wherein, the output of the first linear layer after being processed by the first activation function is multiplied by the output of the recovery linear layer to obtain the input of the second linear layer, the input of the adapter module is spliced ​​with the output of the second linear layer after being processed by the global average pooling to obtain the output of the adapter module, and the output of the adapter module is used as the input of the second normalization layer.

[0040] Specifically, each adapter module may include a first linear layer, a compression linear layer, an activation layer, a restoration linear layer, and a second linear layer.

[0041] Figure 2 This is a flowchart of another multimodal zero-sample image segmentation model training method provided in the embodiment of the present application. Figure 2 Describes the processing of the adapter module.

[0042] In the adapter module, the input of the adapter module can be processed in sequence using the first linear layer, the compressed linear layer, the activation layer, the recovery linear layer and the second linear layer to obtain the input of the second normalization layer, wherein the first linear layer and the second linear layer can both use 1×1 convolution; the compressed linear layer can use 3×3 convolution to map the output of the first linear layer to a higher dimension through the compressed linear layer in order to perform more feature learning; the activation layer can use the Relu activation function to further perform activation function processing on the output of the compressed linear layer to increase the expressive power of the features; the recovery linear layer can also use 3×3 convolution to perform linear transformation on the output after activation function processing again, map it back to the original feature space, and restore its dimension to the original dimension for subsequent fusion with the backbone network. The backbone network here can be represented as the backbone network of the transformer module.

[0043] Among them, the output of the first linear layer can also be processed by a first activation function (for example, a sigmoid function) to obtain the weight of the channel dimension of the output of the first linear layer, and the weight can be multiplied by the output of the recovery linear layer as the input of the second linear layer. Furthermore, the input of the adapter module can be processed by global average pooling and spliced ​​with the output of the second linear layer to obtain the output of the adapter module. Global average pooling can help the adapter module capture global features. Splicing the result of the global average pooling processing with the output of the second linear layer can obtain a richer and more comprehensive feature representation, thereby helping the image encoder to be optimized, and then improving the expressiveness and adaptability of the basic segmentation module, thereby helping the zero-sample image segmentation model to better understand the input data, that is, the image to be segmented.

[0044] In some embodiments, a prompt word encoder is used to process the prompt word to obtain the prompt word encoding features, including: determining the type of the prompt word according to the image, the prompt word includes a coordinate point, a detection box, text and an initial mask; if the prompt word includes the coordinate point, the coordinate point is mapped to a preset dimension vector to obtain the encoding features of the coordinate point; if the prompt word includes the detection box, the upper left corner point and the lower right corner point of the detection box are respectively represented by vector features to obtain the encoding features of the detection box; if the prompt word includes the text, the text features of the text are extracted using a preset text encoder to obtain the encoding features of the text; if the prompt word includes the initial mask, the initial mask is subjected to convolution processing and channel transformation to obtain the initial mask features, the initial mask features are multiplied by the image features to obtain the encoding features of the initial mask, and the image features are obtained based on a preset image feature extraction model.

[0045] Specifically, the basic segmentation module also processes the input by the prompt word encoder, and the prompt word encoder of the pre-trained segmentation module can also be used to process the prompt word. However, there are no restrictions on the parameters and structure of the prompt word encoder.

[0046] Furthermore, the prompt word type may include coordinate points, detection boxes, text and initial masks. It should be understood that in the basic segmentation module, one or several types of prompt word types may be processed, and the input type of the prompt word can be determined according to the corresponding image content. As an example, when it is necessary to segment an object in the image to be segmented, the position coordinate point of the object can be provided as a prompt word to help the model segment it accurately. As another example, when it is necessary to segment an image based on text descriptions or keywords, text can be used as a prompt word for target segmentation. For example, given an image containing "dog", the target can be segmented by inputting the text "dog".

[0047] During the processing of the prompt word encoder, when the prompt word type includes coordinate points, the coordinate points can be mapped to a 256-dimensional vector. This vector can represent the encoding features representing the point location, and can also add two additional dimensions to represent the learnable encoding features of the point as foreground or background.

[0048] When the prompt word type includes a detection box, two vector features can be used to represent the encoding features of the upper left corner and the lower right corner respectively, and the encoding features of the corresponding detection box are obtained to represent the position information of the detection box;

[0049] When the prompt word type includes text, the text can be input into a text encoder (such as a CLIP encoder) to obtain encoding features of the text;

[0050] When the prompt word type includes an initial mask, the input image can be scaled to 1 / 4 of the original size and used as the initial mask. Then, a 2×2 convolution kernel is used with a step size of 2 for convolution operation, and the output channels are 4 and 6. Next, a 1×1 convolution kernel is used to increase the number of channels to 256 to obtain the initial mask feature. In the prompt word encoder, the initial mask feature can also be multiplied by the corresponding image feature to obtain the output of the prompt word encoder. The image feature can be obtained through the feature extraction model in the prompt word encoder, which can be an existing pre-trained image extraction model (such as the Vision Transformer, i.e., the ViT model).

[0051] It should be understood that the prompt words can be obtained based on an image training set or an image to be segmented using image processing technology, or can be obtained by manual annotation, and there is no limitation here.

[0052] According to the solution provided by the above embodiment, during the processing of the basic segmentation module, a prompt word encoder can be used to convert different types of prompt words into the same feature representation form so that they can be processed and analyzed in subsequent tasks. In this way, information such as coordinate points, detection boxes, text, and initial masks can be encoded into vector features with certain semantics and expressive capabilities, thereby better representing and understanding the input data.

[0053] In some embodiments, the mask is processed by a fully connected layer and a first loss function is calculated, and the basic segmentation module is trained based on the first loss function to obtain a trained basic segmentation module, including: processing the mask by a first function and a second function to obtain a final mask of the image; calculating the first loss function based on the final mask and the true mask, and training the basic segmentation module based on the first loss function to obtain a trained basic segmentation module.

[0054] Specifically, after the basic segmentation module processes the image and prompt word to obtain the corresponding mask, the mask can be further processed using the fully connected layer, and the final mask obtained by the fully connected layer can be used to calculate the first loss function. The first loss function can be used to reversely update the parameters of the basic segmentation module to obtain the trained basic segmentation module.

[0055] Furthermore, after processing by the image encoder and the prompt word encoder, image coding features and prompt word coding features are obtained. The image coding features and prompt word coding features are input into the mask decoder for decoding, and the corresponding mask (i.e., binary image) can be obtained. This mask can include a mask corresponding to the image coding features and a mask corresponding to the prompt word coding features. The mask is then processed by the first function and the second function in the fully connected layer to obtain a final mask. The final mask can be represented as a mask that fuses the mask corresponding to the image coding features with the mask corresponding to the prompt word coding features. In the final mask obtained, one final mask can contain multiple objects, or one final mask can contain only one object. The specific determination can be based on the image content.

[0056] It should be noted that in the fully connected layer, the received mask can be first converted into a vector representation, and then the mask can be linearly combined using the first function and the second function to obtain the final mask, and the first loss function can be calculated. As an example, the mask can be linearly combined using Focal Loss and Dice Loss, and the weight coefficients of Focal Loss and Dice Loss can be 15:1. The mean square error loss function (MSE Loss) is used to calculate the IoU score, that is, the intersection-over-union score, as the first loss function, and the MSE Loss can be used as a gradient signal for reverse propagation in each iteration to update the parameters of the basic segmentation module, so that the predicted IoU score is closer to the true IoU score, thereby optimizing the IoU prediction ability of the basic segmentation module during the training process. When calculating the IoU score, it can be based on the final mask and the true mask, where the true mask can be pre-configured manually.

[0057] According to the solution provided in the embodiment of the present application, the performance of the basic segmentation module is improved by mapping the image coding features and the prompt word coding features to the final mask and using the first loss function for training. In this way, the trained basic segmentation module can better segment the image and achieve more accurate object distinction.

[0058] Furthermore, Figure 3 This is a flowchart of another method for training a zero-sample image segmentation model based on multimodality provided in an embodiment of the present application. Figure 3 The training process of the basic segmentation module is further explained.

[0059] First, the image and the prompt word can be input into the basic segmentation model, and the image can be processed using the image encoder to obtain image coding features. The image encoder can be spliced ​​together by multiple converter modules. The adapter module can be pre-configured in each converter module, and the adapter module can be used as the parameter update module of the image encoder to train the image encoder.

[0060] The prompt word is processed using a prompt word encoder to obtain prompt word encoding features, where the prompt word type may include coordinate points, detection boxes, text, and initial masks (Marsk).

[0061] Then, the mask decoder can be used to decode the image coding features and the prompt word coding features to obtain the mask corresponding to the image coding features and the prompt word coding features. The mask is processed by the fully connected layer to obtain the segmentation result output, that is, the final mask and the first loss function. According to the first loss function, the basic segmentation module can be updated in reverse iteration to obtain the trained basic segmentation module.

[0062] In some embodiments, an image is input into a segmentation fine-tuning module, and cross-modal processing and saliency detection processing are performed on the image respectively to obtain text embedding features and saliency feature maps, and the text embedding features and the saliency feature maps are cross-attention fused to obtain a first fused feature map, including: using a language model and a text encoder to cross-modally process the image to obtain text embedding features; using a saliency detection module to extract saliency features of the image to obtain a saliency feature map; cross-attention fusion of the text embedding features and the saliency feature map to obtain a first fused feature map.

[0063] Specifically, the segmentation fine-tuning module can be trained using the image training set. During the training process of the segmentation fine-tuning module, the image can be input into the segmentation fine-tuning module, and the image can be processed through cross-modal processing and saliency detection to obtain text embedding features and salient feature maps. The text embedding features and the salient feature maps are then cross-attentively fused to obtain the first fused feature map.

[0064] Furthermore, the language model can be used to extract the attribute description of the image first, and then the attribute description can be input into the text encoder for encoding to obtain text embedding features. At the same time, the image is passed through the saliency detection module to obtain the saliency feature map of the image. Then, the text embedding feature can be used as the query (Q), and the saliency feature map as the key (K) and value (V) to perform cross-attention fusion to obtain the first fused feature map. The specific cross-attention fusion mechanism is a well-known technology in the field and will not be elaborated here.

[0065] In this way, through cross-modal processing and salient feature extraction of images, and fusing text embedding features with salient feature maps, the text information and salient feature information of the image can be integrated together for subsequent image segmentation tasks to improve the accuracy of the segmentation results and the ability of semantic understanding.

[0066] Figure 4 This is a flowchart of another method for training a zero-sample image segmentation model based on multimodality provided by the embodiment of the present application. Figure 4 The training process of the segmentation fine-tuning model is further explained.

[0067] First, the image can be input into the segmentation and fine-tuning module for cross-modal processing and saliency detection. Specifically, the image is processed using a language model (LM) and a text encoder (such as the CLIP text encoder) to obtain text embedding features. The image is then processed using the saliency detection module to obtain a saliency feature map.

[0068] Then, the text embedding features are cross-attended and fused with the saliency feature map to obtain the first fused feature map. The first fused feature map is processed by a multilayer perceptron (MLP), and the processing result is input into the fully connected layer. The sigmoid cross loss function can be used to predict whether the text description information obtained by LM matches the image, thereby reversely updating the segmentation fine-tuning module, and iterative training is performed to obtain the trained segmentation fine-tuning module.

[0069] In some embodiments, the image and the prompt word are input into the zero-sample image segmentation model for processing, and a third loss function is calculated. The zero-sample image segmentation model is trained based on the third loss function to obtain a trained zero-sample image segmentation model, including: in the zero-sample image segmentation model, the image and the prompt word are processed using the trained basic segmentation module to obtain a first output result; the image is processed using the trained segmentation fine-tuning module to obtain a second output result; the first output result and the second output result are cross-attention fused to obtain a second fused feature map; the second fused feature map is input into a preset decoder for processing to obtain a segmentation result, and a third loss function is calculated based on the segmentation result and the true mask; the zero-sample image segmentation model parameters are updated according to the third loss function to obtain a trained zero-sample image segmentation model.

[0070] Specifically, the zero-shot image segmentation model can be obtained by splicing the trained basic segmentation module and the trained segmentation fine-tuning module. Before the zero-shot image segmentation model is applied, the zero-shot image segmentation model can also be trained using the image training set and the prompt word training set.

[0071] Specifically, the images and prompt words in the training set can first be input into the basic segmentation module of the zero-shot image segmentation model. This basic segmentation module is a trained module that can generate a preliminary segmentation result, i.e., the first output, based on the image and prompt word. Simultaneously, the image can be input into the segmentation fine-tuning module of the zero-shot image segmentation model, where it is further processed to obtain a second output, i.e., an image-text fusion. The first output and the second output are then fused using a cross-attention mechanism to obtain a second fused feature map. The cross-attention mechanism can adaptively adjust the attention based on the attention weight between the first and second outputs, thereby organically combining the two.

[0072] Then, the second fused feature map can be processed by a preset decoder (for example, a transformer decoder) to obtain the segmentation result. At the same time, the difference between the segmentation result and the true mask can be calculated, that is, the second loss function. Finally, the parameters of the zero-sample image segmentation model can be updated according to the second loss function, and the performance of the model can be continuously optimized through iterative training. In this way, after multiple rounds of training, the trained zero-sample image segmentation model can be obtained.

[0073] It should be understood that when generating the segmentation results, the focal loss and dice loss can also be used to perform a linear combination of the masks, and the weight ratio can be 1:1. The second loss function here can also be obtained by calculating the intersection-over-union score. The technical means here can be consistent with the fully connected layer processing in the training process of the basic segmentation module, and will not be repeated here.

[0074] Figure 5 This is a flowchart of another method for training a zero-sample image segmentation model based on multimodality provided in an embodiment of the present application. Figure 5 The processing of the trained zero-shot image segmentation model is further explained.

[0075] In the trained zero-shot image segmentation model, the image to be segmented and the corresponding prompt word can first be input into the basic segmentation module. The image encoder configured with the adapter module encodes the image to be segmented to obtain the encoding features of the image, and the encoding features of the image are input into the mask decoder for decoding to obtain a mask corresponding to the image encoding features; at the same time, the prompt word encoder is used to encode the prompt word to obtain the prompt word encoding features. The prompt word may include coordinate points, detection boxes, text and initial masks. Then the prompt word encoding features are also processed by the mask decoder to obtain a mask corresponding to the prompt word encoding features. The mask obtained by the basic segmentation module is then input into the cross-attention fusion module for cross-attention fusion with the first fusion feature map obtained by the segmentation fine-tuning module.

[0076] In the segmentation fine-tuning module, the received image can be subjected to cross-modal processing and saliency detection processing respectively to obtain text embedding features and saliency feature maps, wherein the cross-modal processing can be processed using a language model and a CLIP text encoder to obtain text embedding features, and then the text embedding features and the saliency feature map are cross-attention fused to obtain a first fused feature map, namely, the image-text fusion feature. It should be understood that in the same segmented character, the image received in the segmentation fine-tuning module is the same as the image received in the basic segmentation module.

[0077] Furthermore, the cross-attention mechanism is used again to fuse the first fused feature map and the mask to obtain a second fused feature map, and then the second fused feature map is processed by the transformer decoder to obtain the result output of the zero-shot image segmentation model. The result output may include the final mask of the image segmentation and the IoU score. In this way, the performance of the zero-shot image segmentation model can be evaluated by the IoU score. That is, during the application of the zero-shot image segmentation model, the parameters of the model can also be updated based on the IoU score, thereby helping the model to improve the accuracy of image segmentation and also improve the model's understanding ability, thereby helping the model to accurately segment unseen object types and small objects.

[0078] It should be understood that the attention mechanisms mentioned in the embodiments of the present application can all be the same attention mechanism, that is, the cross-attention fusion process in the segmentation fine-tuning module can be consistent with another cross-attention fusion process in the zero-sample image segmentation model.

[0079] In this way, by training the zero-shot image segmentation model to obtain the trained zero-shot image segmentation model, the model can perform multimodal learning through prompt word encoding and image encoding in the absence of labeled samples, thereby improving the model's ability to learn unseen object types and small objects, thereby improving the performance of the zero-shot image segmentation model.

[0080] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0081] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0082] Figure 6 Schematic diagram of a zero-sample image segmentation model training device based on multimodality provided in an embodiment of the present application. Figure 6 As shown, the multimodal zero-shot image segmentation model training device includes:

[0083] An acquisition module 601 is configured to acquire images from an image training set and prompt words from a prompt word training set, where the prompt words are determined based on the content of the images;

[0084] The first processing module 602 is configured to input the image and the prompt word into the basic segmentation module, process the image using the image encoder to obtain image coding features, and process the prompt word using the prompt word encoder to obtain prompt word coding features;

[0085] A first training module 603 is configured to process the image encoding features and the prompt word encoding features through a mask decoder to obtain a corresponding mask, process the mask through a fully connected layer and calculate a first loss function, and train the basic segmentation module based on the first loss function to obtain a trained basic segmentation module;

[0086] The second processing module 604 is configured to input the image into the segmentation and fine-tuning module, perform cross-modal processing and saliency detection processing on the image to obtain text embedding features and salient feature maps, and perform cross-attention fusion on the text embedding features and the salient feature maps to obtain a first fused feature map;

[0087] A second training module 605 is configured to process the first fused feature map through a multi-layer perceptron and a fully connected layer and calculate a second loss function, and train the segmentation fine-tuning module based on the second loss function to obtain a trained segmentation fine-tuning module;

[0088] The third processing module 606 is configured to splice the trained basic segmentation module and the trained segmentation fine-tuning module, and connect the cross attention fusion module and the preset decoder to obtain a zero-shot image segmentation model;

[0089] The third training module 607 is configured to input the image and prompt word into the zero-sample image segmentation model for processing, calculate the third loss function, train the zero-sample image segmentation model based on the third loss function, and obtain a trained zero-sample image segmentation model.

[0090] In some embodiments, the first processing module 602 is specifically used to process the image using multiple converter modules to obtain image coding features, and the converter module includes a first normalization layer, a multi-head attention layer, an adapter module, a second normalization layer and a multi-layer perceptron; wherein, the output of the multi-head attention layer is processed according to the adapter module to obtain the input of the second normalization layer, and the adapter module is configured as a parameter update module of the image encoder.

[0091] In some embodiments, the first processing module 602 is specifically used to process the output of the multi-head attention layer in sequence using the first linear layer, the compression linear layer, the activation layer, the restoration linear layer and the second linear layer in the adapter module to obtain the input of the second normalization layer; wherein, the output of the first linear layer after being processed by the first activation function is multiplied by the output of the restoration linear layer to obtain the input of the second linear layer, the input of the adapter module is processed by the global average pooling result and the output of the second linear layer are spliced ​​to obtain the output of the adapter module, and the output of the adapter module is used as the input of the second normalization layer.

[0092] In some embodiments, the first processing module 602 is specifically used to determine the type of prompt word based on the image, and the prompt word includes a coordinate point, a detection box, text and an initial mask; if the prompt word includes a coordinate point, the coordinate point is mapped to a preset dimension vector to obtain the encoding feature of the coordinate point; if the prompt word includes a detection box, the upper left corner point and the lower right corner point of the detection box are respectively represented by vector features to obtain the encoding feature of the detection box; if the prompt word includes the text, the text feature of the text is extracted using a preset text encoder to obtain the encoding feature of the text; if the prompt word includes an initial mask, the initial mask is subjected to convolution processing and channel transformation to obtain the initial mask feature, and the initial mask feature is multiplied by the image feature to obtain the encoding feature of the initial mask, which is obtained based on a preset image feature extraction model.

[0093] In some embodiments, the first training module 603 is specifically used to process the mask through the first function and the second function to obtain the final mask of the image; calculate the first loss function based on the final mask and the true mask, and train the basic segmentation module based on the first loss function to obtain the trained basic segmentation module.

[0094] In some embodiments, the second processing module 604 is specifically used to use a language model and a text encoder to perform cross-modal processing on the image to obtain text embedding features; use a saliency detection module to extract saliency features of the image to obtain a saliency feature map; and perform cross-attention fusion on the text embedding features and the saliency feature map to obtain a first fused feature map.

[0095] In some embodiments, the third training module 607 is specifically used to process the image and the prompt word using the trained basic segmentation module in the zero-sample image segmentation model to obtain a first output result; use the trained segmentation fine-tuning module to process the image to obtain a second output result; cross-attentionally fuse the first output result with the second output result to obtain a second fused feature map; input the second fused feature map into a preset decoder for processing to obtain a segmentation result, and calculate a third loss function based on the segmentation result and the true mask; update the zero-sample image segmentation model parameters according to the third loss function to obtain a trained zero-sample image segmentation model.

[0096] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0097] Figure 7 Schematic diagram of the electronic device 7 provided in the embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable by the processor 701. When the processor 701 executes the computer program 703, the steps of the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0098] The electronic device 7 may be a desktop computer, a notebook, a PDA, a cloud server or other electronic device. The electronic device 7 may include but is not limited to a processor 701 and a memory 702. Those skilled in the art will understand that Figure 7 This is merely an example of the electronic device 7 and does not limit the electronic device 7 . The electronic device 7 may include more or fewer components than shown in the figure, or different components.

[0099] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0100] The memory 702 can be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 702 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 7. The memory 702 can also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0101] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0103] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multimodal zero-shot image segmentation model training method, characterized in that: include: Obtaining an image in an image training set and a prompt word in a prompt word training set, wherein the prompt word is determined based on the content of the image; Inputting the image and the prompt word into a basic segmentation module, processing the image using an image encoder to obtain image coding features, and processing the prompt word using a prompt word encoder to obtain prompt word coding features; Processing the image encoding features and the prompt word encoding features through a mask decoder to obtain a corresponding mask, processing the mask through a fully connected layer and calculating a first loss function, and training the basic segmentation module based on the first loss function to obtain a trained basic segmentation module; Inputting the image into a segmentation and fine-tuning module, performing cross-modal processing and saliency detection processing on the image to obtain text embedding features and a salient feature map, and performing cross-attention fusion on the text embedding features and the salient feature map to obtain a first fused feature map; Processing the first fusion feature map through a multi-layer perceptron and a fully connected layer and calculating a second loss function, training the segmentation fine-tuning module based on the second loss function to obtain a trained segmentation fine-tuning module; The trained basic segmentation module and the trained segmentation fine-tuning module are spliced ​​together, and a cross-attention fusion module and a preset decoder are connected to obtain a zero-shot image segmentation model; The image and the prompt word are input into the zero-sample image segmentation model for processing, and a third loss function is calculated. The zero-sample image segmentation model is trained based on the third loss function to obtain a trained zero-sample image segmentation model.

2. The method according to claim 1, characterized in that The processing of the image by using an image encoder to obtain image coding features includes: Processing the image using a plurality of converter modules to obtain the image encoding features, wherein the converter modules include a first normalization layer, a multi-head attention layer, an adapter module, a second normalization layer, and a multi-layer perceptron; The output of the multi-head attention layer is processed according to the adapter module to obtain the input of the second normalization layer, and the adapter module is configured as a parameter update module of the image encoder.

3. The method according to claim 2, characterized in that The processing of the output of the multi-head attention layer according to the adapter module to obtain the input of the second normalization layer includes: In the adapter module, the output of the multi-head attention layer is processed in sequence using a first linear layer, a compression linear layer, an activation layer, a recovery linear layer, and a second linear layer to obtain the input of the second normalization layer; The output of the first linear layer after being processed by the first activation function is multiplied by the output of the recovery linear layer to obtain the input of the second linear layer, the input of the adapter module after being processed by the global average pooling is concatenated with the output of the second linear layer to obtain the output of the adapter module, and the output of the adapter module is used as the input of the second normalization layer.

4. The method according to claim 1, wherein The prompt word is processed by the prompt word encoder to obtain the prompt word encoding feature, including: determining a type of the prompt word according to the image, wherein the prompt word includes a coordinate point, a detection box, text, and an initial mask; If the prompt word includes the coordinate point, mapping the coordinate point to a preset dimension vector to obtain a coding feature of the coordinate point; If the prompt word includes the detection box, the upper left corner point and the lower right corner point of the detection box are respectively represented by vector features to obtain the encoding features of the detection box; If the prompt word includes the text, extracting text features of the text using a preset text encoder to obtain encoding features of the text; If the prompt word includes the initial mask, the initial mask is subjected to convolution processing and channel transformation to obtain initial mask features, and the initial mask features are multiplied by image features to obtain encoding features of the initial mask, where the image features are obtained based on a preset image feature extraction model.

5. The method according to claim 1, wherein The step of processing the mask through a fully connected layer and calculating a first loss function, and training the basic segmentation module based on the first loss function to obtain a trained basic segmentation module includes: Processing the mask by a first function and a second function to obtain a final mask of the image; The first loss function is calculated according to the final mask and the true mask, and the basic segmentation module is trained based on the first loss function to obtain the trained basic segmentation module.

6. The method according to claim 1, wherein The image is input into the segmentation and fine-tuning module, cross-modal processing and saliency detection processing are performed on the image to obtain text embedding features and salient feature maps, and cross-attention fusion is performed on the text embedding features and the salient feature maps to obtain a first fused feature map, including: Performing cross-modal processing on the image using a language model and a text encoder to obtain the text embedding feature; Extracting salient features from the image using a saliency detection module to obtain the saliency feature map; The text embedding feature and the saliency feature map are cross-attentively fused to obtain the first fused feature map.

7. The method according to claim 1, characterized in that Inputting the image and the prompt word into the zero-shot image segmentation model for processing, calculating a third loss function, and training the zero-shot image segmentation model based on the third loss function to obtain a trained zero-shot image segmentation model, including: In the zero-shot image segmentation model, the image and the prompt word are processed using the trained basic segmentation module to obtain a first output result; Processing the image using the trained segmentation fine-tuning module to obtain a second output result; Performing cross-attention fusion on the first output result and the second output result to obtain a second fused feature map; Inputting the second fused feature map into the preset decoder for processing to obtain a segmentation result, and calculating the third loss function based on the segmentation result and the true mask; The zero-shot image segmentation model parameters are updated according to the third loss function to obtain a trained zero-shot image segmentation model.

8. A multimodal zero-shot image segmentation model training device, characterized in that: include: An acquisition module is configured to acquire an image from an image training set and a prompt word from a prompt word training set, wherein the prompt word is determined based on the content of the image; a first processing module configured to input the image and the prompt word into a basic segmentation module, process the image using an image encoder to obtain image coding features, and process the prompt word using a prompt word encoder to obtain prompt word coding features; a first training module configured to process the image encoding features and the prompt word encoding features through a mask decoder to obtain a corresponding mask, process the mask through a fully connected layer and calculate a first loss function, and train the basic segmentation module based on the first loss function to obtain a trained basic segmentation module; a second processing module configured to input the image into a segmentation and fine-tuning module, perform cross-modal processing and saliency detection processing on the image, obtain text embedding features and a salient feature map, and perform cross-attention fusion on the text embedding features and the salient feature map to obtain a first fused feature map; A second training module is configured to process the first fusion feature map through a multi-layer perceptron and a fully connected layer and calculate a second loss function, and train the segmentation fine-tuning module based on the second loss function to obtain a trained segmentation fine-tuning module; a third processing module, configured to concatenate the trained basic segmentation module and the trained segmentation fine-tuning module, and connect a cross-attention fusion module and a preset decoder to obtain a zero-shot image segmentation model; The third training module is configured to input the image and the prompt word into the zero-sample image segmentation model for processing, calculate a third loss function, train the zero-sample image segmentation model based on the third loss function, and obtain a trained zero-sample image segmentation model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images