Few-sample semantic segmentation method based on image-text fusion

Abstract

The invention relates to the field of computer vision, and discloses a few-sample semantic segmentation method based on image-text fusion, comprising the following steps: S1, collecting and preprocessing data sets, grouping the data sets according to categories, each group comprising a plurality of support images and query images; s2, extracting multi-scale image features by using a pre-trained deep neural network; s3, extracting text features prompted by a preset text by using a text encoder; s4, fusing the extracted text features with the image features; s5, mining the correlation between the support image features and the query image features by using an attention mechanism; and S6, performing up-sampling on the fused image features, and decoding to generate a segmentation prediction result. According to the method, semantic information is supplemented through image-text fusion and text prompt, and the accuracy and performance of few-sample semantic segmentation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a few-sample semantic segmentation method based on image-text fusion. Background Art

[0002] In the field of computer vision, few-shot semantic segmentation has always been a challenging research problem. Traditional semantic segmentation methods rely on a large amount of labeled data to train models to achieve accurate recognition and segmentation of target areas in images. However, in practical applications, obtaining a large amount of labeled data is often very difficult and time-consuming, especially in some specific fields or emerging scenarios where labeled samples are extremely limited. This makes the existing segmentation methods based on large amounts of data training perform poorly in the face of few-shot scenarios, making them difficult to effectively promote and apply.

[0003] To solve this problem, researchers have proposed few-shot semantic segmentation methods, attempting to maintain high segmentation performance with a small number of labeled samples. However, these existing methods mainly rely on extracting visual features from limited support images, but due to the small amount of data and lack of diversity, the model often fails to fully capture the details and semantic information of the target category when segmenting the query image. In addition, due to the spatial differences between the visual features and semantic information of the image, existing methods find it difficult to achieve efficient feature fusion, further limiting the segmentation performance of the model. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a few-sample semantic segmentation method based on image-text fusion. In the few-sample scenario, semantic information is supplemented by image-text fusion and text prompts to improve the accuracy and performance of semantic segmentation.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a few-sample semantic segmentation method based on image-text fusion, comprising the following steps:

[0006] Collect and preprocess a dataset, wherein the dataset is grouped by category, each group containing a number of support images and query images;

[0007] Extract multi-scale image features using pre-trained deep neural networks;

[0008] Using a text encoder to extract text features of a preset text prompt;

[0009] Fusion of extracted text features with image features;

[0010] Use the attention mechanism to mine the correlation between the supporting image features and the query image features;

[0011] The fused image features are upsampled and decoded to generate segmentation prediction results.

[0012] Preferably, the step of collecting and preprocessing the data set includes:

[0013] The dataset is grouped according to the total image category of the dataset, and the dataset is divided into four groups, each group is called a fold;

[0014] The data in each fold consists of support images and query images, where the support images are denoted as The query image is denoted as D q ={X,Y}, where X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value set in the few-shot scenario, and represents the number of supporting images for each category;

[0015] The image categories in the dataset are evenly distributed into four folds to ensure that the categories contained in each fold do not overlap with each other.

[0016] Preferably, the step of extracting multi-scale image features using a pre-trained deep neural network includes:

[0017] The input image X i Input into the pre-trained deep neural network to extract multi-scale image features;

[0018] Through different levels of the pre-trained deep neural network, four scales of image features are extracted in turn, which are denoted as X1, X2, X3, and X4, respectively, where X1, X2, X3, and X4 represent image features from shallow to deep layers in turn;

[0019] Perform global average pooling on the deep feature map to obtain the global features of the image The formula is:

[0020]

[0021] Global Features With deep features Figure X 4 are input into the multi-head self-attention mechanism together to enhance the feature expression ability of the image. The formula is:

[0022]

[0023] in, represents the enhanced global features, and Z represents the enhanced image features.

[0024] Preferably, the step of extracting text features of a preset text prompt using a text encoder comprises:

[0025] Preset text prompts, where the text prompts are related to the target category in the image and are used to indicate the target category to be segmented;

[0026] Inputting the text prompt into a text encoder Transformer to extract text features related to the prompt;

[0027] The text encoder Transformer processes the input text prompts, captures the contextual information in the text through the self-attention mechanism, and generates a feature vector t representing the semantics of the text. The formula is:

[0028] t=Transformer(text prompt)

[0029] Wherein, text prompt represents the preset text prompt, and t represents the text feature vector extracted from the text prompt.

[0030] Preferably, the step of fusing the extracted text features with the image features comprises:

[0031] Through a Transformer decoder consisting of three layers of Transformers, the image feature Z is fused with the text feature t to generate a text feature V containing visual information. The formula is:

[0032]

[0033] Among them, Z represents the multi-scale features extracted from the image, t represents the text features generated by the text prompt, and V represents the fused text features containing visual information;

[0034] Using the learnable scaling parameter α, the fused visual information V is weightedly combined with the original text feature t to generate the updated text feature t', which is:

[0035] t′=t+αV

[0036] Among them, α is a learnable parameter that controls the influence of visual information on text features, t' is the updated text feature, which contains semantic information from the image;

[0037] Calculate the correlation between the updated text feature t' and the image feature Z to generate the score map S. The formula is:

[0038] S=Zt′

[0039] The score map S is concatenated with the original deep image feature X4 to form the fused image feature X'4 for subsequent decoding and segmentation. The formula is:

[0040] X′4=[X4,S]

[0041] Among them, X4 is the original deep image feature, S is the score map, and X'4 is the fused image feature, which combines the multimodal information of image and text.

[0042] Preferably, the step of mining the correlation between the supporting image features and the query image features using the attention mechanism comprises:

[0043] The features F extracted from the support image s and the features F extracted from the query image q Flattening is performed to convert the multidimensional feature map into a one-dimensional vector. The formula is:

[0044] F′ s = flatten(F s )

[0045] F′ q = flatten(F q )

[0046] The flattened support image feature F' s and query image features F' q Perform linear mapping respectively to generate query vector Q, key vector K and value vector V. The formula is:

[0047] Q=linear(flatten(F' q ))

[0048] K,V=linear(flatten(F' s ))

[0049] The self-attention mechanism is used to calculate the correlation between the query vector Q and the key vector K. The correlation is calculated by the dot product of the query vector Q and the key vector K, and the attention weight is generated by the normalization operation, and then the value vector V is weighted. The formula is:

[0050]

[0051] Among them, softmax(·) represents the normalization operation, is the scaling factor, d is the dimension of the vector, and Atten(Q,K,V) is the weighted feature representation;

[0052] Through the above attention mechanism, we can mine the supporting image features F s and query image features F q The correlation between them is used to obtain enhanced query image features. The formula is:

[0053]

[0054] in, The query image features are enhanced by the attention mechanism and contain semantic information related to the supporting image features, which is further used to improve the accuracy of few-sample semantic segmentation.

[0055] Preferably, the step of upsampling the fused image features and decoding to generate the segmentation prediction result comprises:

[0056] The fused image feature X'4 is upsampled to restore the feature map to the same resolution as the input image. Upsampling can be achieved through bilinear interpolation or transposed convolution. The formula is:

[0057] X up =Upsample(X′4)

[0058] Among them, X'4 represents the fused image features, X up is the feature map after upsampling;

[0059] In the upsampling process, the upsampled features Figure X up The image features X of the corresponding scale extracted in the encoding stage i Fusion is performed to retain global semantic information and detail information. The formula is:

[0060] X fused =Concat(X up ,X i )

[0061] Among them, Concat(·) represents the feature concatenation operation, X i represents the multi-scale features extracted in the encoding stage, X fused Represents the fused feature map;

[0062] The fused features Figure X fused Input to the decoder, which consists of multiple convolutional layers or transposed convolutional layers to gradually restore and generate the final segmentation prediction results The specific expression is:

[0063]

[0064] Among them, Decoder(·) represents the decoder, represents the segmentation prediction mask output by the decoder;

[0065] The final segmentation prediction mask It has the same resolution as the original input image, and the value of each pixel represents the probability or classification label of the pixel belonging to a certain category, which is used for the prediction output of the few-shot semantic segmentation task.

[0066] The present invention also provides a few-sample semantic segmentation device based on image-text fusion, comprising:

[0067] Data preprocessing module, used to collect and preprocess data sets;

[0068] Image feature extraction module, which uses pre-trained deep neural network to extract multi-scale image features;

[0069] A text feature extraction module, which uses a text encoder to extract text features of a preset text prompt;

[0070] Feature fusion module, which fuses the extracted text features with image features;

[0071] The correlation mining module uses the attention mechanism to mine the correlation between the supporting image features and the query image features;

[0072] The decoding module upsamples the fused image features and decodes them to generate segmentation prediction results.

[0073] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0074] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0075] The present invention provides a few-sample semantic segmentation method based on image-text fusion, which has the following beneficial effects:

[0076] 1. The present invention proposes a few-sample semantic segmentation method based on image-text fusion, which supplements additional semantic information by adding text prompts and improves the existing few-sample semantic segmentation method by using the image-text fusion method, thereby significantly improving the segmentation performance. This method effectively overcomes the problem of poor segmentation effect caused by insufficient data in the case of few samples.

[0077] 2. When solving the problem of few-sample semantic segmentation, the present invention introduces textual hints related to image categories as a supplementary source of semantic information. Textual hints provide additional semantic guidance for the model, helping the model to better understand and identify the target categories in the image, so that semantic segmentation can still be performed accurately when data is limited.

[0078] 3. The present invention effectively alleviates the fusion problem of visual and semantic space due to spatial differences by aligning the image visual space and the text semantic space. Through the effective fusion of image and text features, the model can establish a closer connection between the semantic level and the visual level, improve the expressive power of the fused features, and thus enhance the segmentation effect.

[0079] 4. The proposed method shows good segmentation performance in the case of few samples and can significantly improve the accuracy of segmentation. Whether in the case of limited training data or in the face of new categories, the method can effectively generalize and provide high-precision segmentation results in various challenging tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0081] Figure 2 A flowchart of a new method for few-sample semantic segmentation based on prototype perception is provided for an embodiment of the present invention;

[0082] Figure 3 A schematic diagram of support images and query images in the training and testing process provided by an embodiment of the present invention;

[0083] Figure 4 An example diagram of extracting multi-scale image features using the ResNet50 network provided in an embodiment of the present invention;

[0084] Figure 5 A schematic diagram of text features updated using visual information provided by an embodiment of the present invention;

[0085] Figure 6 A schematic diagram of image features that introduce semantic information provided by an embodiment of the present invention;

[0086] Figure 7 A schematic diagram of performing relationship mining between a support image and a query image provided by an embodiment of the present invention;

[0087] Figure 8 The segmentation performance of the model provided by the embodiment of the present invention on the COCO dataset;

[0088] Fig. 9 It is a schematic diagram of the structure of the device of the present invention;

[0089] Fig.10 It is a schematic diagram of the computer device structure of the present invention.

[0090] Among them, 100, data preprocessing module; 200, image feature extraction module; 300, text feature extraction module; 400, feature fusion module; 500, correlation mining module; 600, decoding module; 40, computer equipment; 41, processor; 42, memory; 43, storage medium. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] Please refer to the attached Figure 1 The embodiment of the present invention provides a few-sample semantic segmentation method based on image-text fusion, comprising the following steps:

[0093] S1. Collect and preprocess datasets

[0094] First, we need to collect and preprocess the dataset. The task of few-shot semantic segmentation relies on the effective grouping and processing of the dataset. To ensure that the model can accurately perform semantic segmentation under few-shot conditions, we need to group the dataset reasonably by category and preprocess the image data in a standardized manner to provide high-quality input data for subsequent steps.

[0095] In this embodiment, the specific implementation steps are as follows:

[0096] 1. Dataset grouping:

[0097] The dataset is grouped according to the total image categories of the dataset, and the dataset is divided into multiple groups (folds). For example, for the COCO dataset, the dataset contains 80 image categories, which are evenly divided into four groups, each containing 20 categories, and the categories between different groups do not overlap. This grouping method ensures that the categories in different groups are independent of each other, which can provide diversity for model training in few-sample scenarios.

[0098] 2. Definition and division of support images and query images:

[0099] The data of each group is further divided into support image set D s and query image D q :

[0100] Support image set D s , used for model training, providing a small amount of annotation information to guide model learning:

[0101]

[0102] Query image D q , used for model testing and validation to evaluate the segmentation performance of the model in practical applications:

[0103] D q ={X,Y}

[0104] Among them, X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value set in the few-shot scenario (usually 1 or 5), representing the number of supporting images for each category.

[0105] 3. Data preprocessing:

[0106] Before model training, the image data must be preprocessed, including the following operations:

[0107] Resizing: Resizing images to the input size required by the network ensures that images fed into the neural network have a uniform resolution.

[0108] Data augmentation: Apply data augmentation techniques such as random cropping, rotation, flipping, and color jittering to increase the diversity of the data and improve the robustness of the model.

[0109] Normalization: Normalize the image pixel values ​​and scale the pixel values ​​to the range of [0, 1] or [-1, 1], which helps speed up the model training process and improve the training effect.

[0110] Through the above steps, this embodiment achieves effective grouping and preprocessing of the data set, laying a solid foundation for the few-sample semantic segmentation task of the model.

[0111] S2. Extract multi-scale image features using pre-trained deep neural networks

[0112] After completing the collection and preprocessing of the data set, the next step is to extract multi-scale image features through a pre-trained deep neural network. The goal of this step is to extract feature information at different levels from the image in order to provide rich visual feature support in the subsequent image-text fusion and semantic segmentation process.

[0113] In this embodiment, the specific implementation steps are as follows:

[0114] 1. Multi-scale feature extraction:

[0115] Input image X iThe data is processed by a pre-trained ResNet50 network. ResNet50 is a commonly used convolutional neural network that has been pre-trained on large-scale datasets (such as ImageNet) and has good feature extraction capabilities.

[0116] In the ResNet50 network, convolutional layers at different levels will extract feature maps of different scales. Specifically, in this embodiment, feature maps of four scales are extracted, which are respectively recorded as X1, X2, X3, and X4. Among them:

[0117] X1 represents the low-level features extracted by the shallow convolutional layer, which contains more edge and texture information;

[0118] X2 and X3 represent the intermediate layer features, which contain gradually abstracted semantic information;

[0119] X4 represents the high-level features extracted by the deep convolutional layers, which mainly contain global semantic information.

[0120] 2. Global feature extraction:

[0121] For deep features Figure X 4 Further processing, using global average pooling (Global Average Pooling) to compress the spatial dimension of the feature map into a vector, obtain the global feature The specific operation of global average pooling is to average all pixels in each channel of the feature map:

[0122]

[0123] The global feature It represents the global information of the entire image and can provide semantic support for multimodal feature fusion in subsequent steps.

[0124] 3. Enhanced multi-head self-attention mechanism:

[0125] In order to further improve the expressiveness of image features, the global features With deep features Figure X 4 is input to the Multi-Head Self-Attention (MHSA) mechanism for processing. The Multi-Head Self-Attention mechanism captures the relationship between different positions in the feature map through multiple parallel attention heads to generate enhanced global features. And the deep feature Z:

[0126]

[0127] in, represents the global features enhanced by the attention mechanism, and Z represents the enhanced deep image features.

[0128] Through the above steps, the present embodiment completes the extraction and enhancement of multi-scale image features, providing rich and diverse visual information for subsequent image-text fusion and segmentation prediction.

[0129] S3. Use the text encoder to extract text features of the preset text prompt

[0130] After completing the extraction of multi-scale image features, in order to further enhance the model's ability to understand the semantics of the target category, the present invention uses a text encoder to extract text features related to the preset text prompt. By combining image features with text features, the semantic information of the model can be enriched, thereby improving the accuracy of few-sample semantic segmentation.

[0131] In this embodiment, the specific implementation steps are as follows:

[0132] 1. Text prompt design:

[0133] First, design a text prompt related to the target category. This prompt is used to clarify the target category that the model should focus on. For example, for a target category of "cat", the text prompt can be set to "a photo of a cat". This prompt provides clear semantic guidance for the model, so that the text features extracted subsequently can more accurately reflect the semantic information of the target category.

[0134] 2. Use of text encoder:

[0135] Input the preset text prompt into the text encoder Transformer. Transformer is a model based on the self-attention mechanism, which is widely used in natural language processing tasks. Its self-attention mechanism can effectively capture the contextual information in the text, so that the extracted text features can better represent the semantic content of the text prompt.

[0136] 3. Extraction of text features:

[0137] Specifically, the text encoder Transformer processes the input text prompt and generates a high-dimensional text feature vector t. This vector encodes the semantic information of the text prompt and can be effectively combined with the image features in the subsequent feature fusion process. The text feature extraction process can be expressed as:

[0138] t=Transformer(text prompt)

[0139] Among them, text prompt is the input text prompt, and t is the text feature vector extracted from the text prompt.

[0140] Through the above steps, the text features related to the target category are successfully extracted in this embodiment, providing necessary semantic information support for the fusion of image and text features in subsequent steps.

[0141] S4. Fusion of extracted text features and image features

[0142] After the image features and text features are extracted, in order to make full use of the information of these two different modalities, the present invention fuses the extracted text features with the image features. Through this fusion method, the model can combine visual and semantic information, thereby enhancing the understanding and segmentation capabilities of the target category.

[0143] In this embodiment, the specific implementation steps are as follows:

[0144] 1. Visual information is integrated into text features:

[0145] First, the image feature Z extracted in step S2 is fused with the text feature t extracted in step S3. Specifically, the image feature Z and the text feature t are taken as input and sent to the Transformer decoder composed of three layers of Transformers. The decoder integrates the visual information into the text features through a multi-layer self-attention mechanism and a feedforward neural network to generate a new feature representation V. The process can be expressed as:

[0146]

[0147] Among them, Z represents the multi-scale features extracted from the image, t represents the text features generated by the text prompt, and V represents the text features integrated with visual information.

[0148] 2. Update text features:

[0149] In order to make the fused visual information better affect the text features, a learnable scaling parameter α is used to weight the visual information V and the original text features t to generate updated text features t'. This update operation adjusts the weight of the visual information to make the text features more suitable for the few-sample segmentation task. The specific operation is:

[0150] t′=t+αV

[0151] Among them, α is a learnable parameter that controls the influence of visual information, and t' is the updated text feature that combines the semantic information of the image.

[0152] 3. Integrate semantic information into image features:

[0153] Next, the updated text feature t' is further fused with the image feature Z. Specifically, the correlation between the text feature t' and the image feature Z is calculated to generate a score map S, which reflects the degree of semantic matching between the text and image features. The fusion operation can be expressed as:

[0154] S=Zt′

[0155] Finally, the generated score map S is concatenated with the deep image feature X4 to form a new fused image feature X'4 for subsequent upsampling and decoding steps. The concatenation operation can be expressed as:

[0156] X′4=[X4,S]

[0157] Among them, X4 is the original deep image feature, S is the semantic score map, and X'4 is the image feature combined with semantic information.

[0158] Through the above steps, the effective fusion of image features and text features is achieved in this embodiment. The generated fusion features contain both the details of visual information and the text semantic information, providing richer feature support for subsequent segmentation prediction.

[0159] S5. Use the attention mechanism to mine the correlation between supporting image features and query image features

[0160] After completing the fusion of image features and text features, in order to further improve the accuracy of few-sample semantic segmentation, the present invention mines the correlation between the supporting image features and the query image features through the attention mechanism. In this way, the semantic relationship between the supporting image and the query image can be better captured, so that the target category can be more accurately identified during the segmentation process.

[0161] In this embodiment, the specific implementation steps are as follows:

[0162] 1. Feature flattening and linear mapping:

[0163] First, the features F extracted from the support image are s and the features F extracted from the query image q Perform a flattening operation. The flattening operation converts the original multi-dimensional feature map into a one-dimensional vector for subsequent linear mapping processing. The expression of the flattening operation is:

[0164] F′ s = flatten(F s )

[0165] F′ q = flatten(F q )

[0166] Flattened support image features F' s and query image features F' q Through linear mapping, the query vector Q, key vector K and value vector V are generated respectively. The purpose of the linear mapping operation is to convert the feature vector into a suitable dimension for self-attention mechanism calculation. The specific expression is:

[0167] Q=linear(flatten(F' q ))

[0168] K,V=linear(flatten(F' s ))

[0169] Among them, Q represents the query vector for querying image features, K and V represent the key vector and value vector supporting image features.

[0170] 2. Self-attention mechanism calculation:

[0171] The self-attention mechanism is used to calculate the correlation between the query vector Q and the key vector K. The specific operation is to calculate the dot product of the query vector Q and the key vector K to obtain the similarity score between them. In order to avoid the value being too large or too small, the calculation will be scaled and the similarity score will be normalized by the softmax function to generate the attention weight. The calculation formula of the self-attention mechanism is:

[0172]

[0173] in, is the scaling factor, d is the dimension of the vector, and Atten(Q,K,V) represents the feature representation after attention weighting.

[0174] 3. Feature enhancement:

[0175] Use the generated attention weights to weight the value vector V to obtain the weighted support image features. Then, the weighted support image features are compared with the query image features F q Further combination is performed to form enhanced query image features This combination helps the model better utilize the semantic information in the support image when segmenting the query image. The expression of the enhanced query image feature is:

[0176]

[0177] in, The query image features are processed by the self-attention mechanism and combined with relevant semantic information from the support images.

[0178] Through the above steps, this embodiment achieves effective mining of the correlation between supporting image features and query image features, and the generated enhanced features combine the semantic information of both, which helps to improve the accuracy and robustness of few-sample semantic segmentation tasks.

[0179] S6. Upsample the fused image features and decode them to generate segmentation prediction results

[0180] After completing the correlation mining between the supporting image features and the query image features, in order to obtain the final segmentation result, the present invention upsamples the fused image features and generates the final segmentation prediction mask through the decoder. The upsampling and decoding process can restore the spatial resolution of the image while retaining and utilizing feature information of different scales to achieve high-precision semantic segmentation.

[0181] In this embodiment, the specific implementation steps are as follows:

[0182] 1. Upsampling to restore spatial resolution:

[0183] First, the fused image feature X'4 generated in the previous step is upsampled. The purpose of upsampling is to restore the low-resolution feature map to the spatial resolution of the original image so that the output segmentation mask can accurately match the size of the input image. Upsampling can usually be achieved through methods such as bilinear interpolation or transposed convolution. The specific operation expression is:

[0184] X up =Upsample(X′4)

[0185] Among them, X'4 is the fused image feature, X up It is the upsampled feature map with higher spatial resolution.

[0186] 2. Multi-scale feature fusion:

[0187] In the upsampling process, the upsampled features Figure X up The features of the corresponding scale extracted in the encoding stage Figure X i Through the combination of this multi-scale feature, the global semantic information and detail information can be retained at the same time, thereby enhancing the accuracy of the segmentation result. The specific expression of the fusion operation is:

[0188] X fused =Concat(X up ,X i )

[0189] Among them, X i represents the multi-scale features extracted in the encoding stage, X fusedIt is the fused feature map, which contains multi-scale visual information.

[0190] 3. Decode and generate segmentation mask:

[0191] The fused features Figure X fused The decoder is usually composed of multiple convolutional layers or transposed convolutional layers, which gradually restore the spatial resolution of the feature map to generate the final segmentation prediction result. The decoding process restores the feature map to a segmentation mask of the same size as the original input image, and the value of each pixel represents the probability or classification label of the location belonging to a certain category. The expression of the decoding process is:

[0192]

[0193] Among them, Decoder(·) represents the decoder, Represents the segmentation prediction mask output by the decoder.

[0194] 4. Accuracy verification (mIoU):

[0195] In order to verify the accuracy of the segmentation results, the generated segmentation mask is evaluated using the mIoU (Mean Intersection over Union) metric. mIoU is a commonly used semantic segmentation evaluation metric that measures the accuracy of segmentation by calculating the intersection over union ratio between the predicted result and the true label.

[0196] Specifically, for each category i, calculate its IoU value:

[0197]

[0198] Among them, TP i Represents the number of true positives for category i, FP i Indicates the false positive number, FN i Represents the number of false negatives.

[0199] Then, we average the IoU of all categories to get the mIoU value:

[0200]

[0201] Among them, K is the total number of categories, and mIoU is the average segmentation accuracy over all categories.

[0202] Through the above steps, this embodiment not only realizes the upsampling and decoding process of the fused features and generates high-precision segmentation prediction results, but also quantitatively evaluates the segmentation results through the mIoU indicator to ensure the accuracy and effectiveness of the model in the few-sample semantic segmentation task.

[0203] Example:

[0204] The embodiment of the present invention provides a prototype-aware few-sample semantic segmentation method, which combines image features with text features and utilizes a multi-head self-attention mechanism and a cross-attention mechanism to achieve high-precision semantic segmentation in a few-sample scenario. The following is a detailed description of the embodiments of the present invention through the accompanying drawings.

[0205] Figure 2 The figure shows an overall flow chart of a new method for semantic segmentation of few samples based on prototype perception provided by an embodiment of the present invention. The process includes the following main steps:

[0206] S1. Collect and download the datasets required for the experiment and perform preprocessing.

[0207] S2. Extract multi-scale image features using deep neural networks.

[0208] S3. Use text encoder to extract text features.

[0209] S4. Perform image and text feature fusion.

[0210] S5. Perform correlation mining on supporting image features and query image features.

[0211] S6. Decode the fused image features to obtain prediction results.

[0212] S7. Input the test set to evaluate the overall segmentation accuracy of the model.

[0213] Figure 3 The figure shows a schematic diagram of the support images and query images in the training and testing process of an embodiment of the present invention. The support image set contains N*K images, where N represents the number of categories (as shown in the figure, the categories include Dog, Goose, Bus, etc.), and K represents the number of support images corresponding to each category (as shown in the figure, K is 1, that is, each category has one support image). The query image consists of the original image and the corresponding image mask. During the training process, the model learns the features of each category through the support images, and applies these features to the segmentation task of the query image.

[0214] Figure 4 The figure shows an example of extracting multi-scale image features using the ResNet50 network provided by an embodiment of the present invention. The ResNet50 network consists of stacked convolutional layers, pooling layers, and activation layers, and is divided into four blocks. Different blocks contain different numbers of convolutional layers and can extract image features of four different scales. The shallow feature map retains more spatial detail information, while the deep feature map contains higher-level semantic information.

[0215] Figure 5 The figure shows a schematic diagram of the text feature update process provided by an embodiment of the present invention. First, a text prompt "a photo of [CLS]" is preset, where CLS represents the target category to be segmented. Then, the text prompt is encoded using a Transformer encoder to obtain initial text features. Next, the visual information extracted from the Transformer Decoder is residually updated with the text features to achieve the fusion of image and text information, thereby enhancing the semantic expression ability of the text features.

[0216] Figure 6 The figure shows a schematic diagram of the fusion process of image features and text features in an embodiment of the present invention. First, the multi-head self-attention mechanism (MHSA) is used to obtain the global features. And feature map Z. Then, the feature map Z is multiplied by the text feature t to obtain the corresponding relationship between the image feature and the text feature, and the image and text features are initially fused through this relationship, thereby introducing the semantic information of the text into the image feature.

[0217] Figure 7 The figure shows a schematic diagram of using a cross-attention mechanism to mine the relationship between a support image and a query image provided by an embodiment of the present invention. The features of the query image are used as Query (Q), and the features of the support image are used as Key (K) and Value (V). Through the cross-attention mechanism, the class-related information in the query image is mapped to the support image, thereby further emphasizing the target category in the query image. This process effectively enhances the segmentation ability of the model in the case of few samples.

[0218] Figure 8 The figure shows the segmentation performance comparison of the embodiments of the present invention on the COCO dataset. The figure shows that the segmentation accuracy of the model of the present invention on fold2 and fold3 is better than that of the other three methods, and the overall segmentation performance is better than the other two methods, second only to CWTd. The evaluation results of mIoU show the superior performance of the model of the present invention in the few-sample semantic segmentation task.

[0219] The present invention combines the multimodal features of images and texts and uses multi-head self-attention and cross-attention mechanisms to achieve high-precision semantic segmentation in few-sample scenarios. ResNet50 extracts multi-scale image features, and the Transformer encoder extracts text features, and effectively enhances the semantic understanding ability of the model in the process of feature fusion and relationship mining. Finally, accurate segmentation results are generated through upsampling and decoding, and performance evaluation is performed through mIoU, which verifies the superiority of the present invention in few-sample semantic segmentation tasks.

[0220] The few-sample semantic segmentation device based on image-text fusion described below and the few-sample semantic segmentation method based on image-text fusion described above can refer to each other.

[0221] Please refer to the attached Fig. 9 The present invention also provides a few-sample semantic segmentation device based on image-text fusion, comprising:

[0222] A data preprocessing module 100, for collecting and preprocessing data sets;

[0223] An image feature extraction module 200 extracts multi-scale image features using a pre-trained deep neural network;

[0224] A text feature extraction module 300 extracts text features of a preset text prompt using a text encoder;

[0225] A feature fusion module 400, which fuses the extracted text features with the image features;

[0226] A correlation mining module 500 uses an attention mechanism to mine the correlation between the supporting image features and the query image features;

[0227] The decoding module 600 upsamples the fused image features and decodes them to generate segmentation prediction results.

[0228] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0229] Please refer to the attached Fig.10 The present invention further provides a computer device 40, comprising: a processor 41 and a memory 42, wherein the memory 42 stores a computer program executable by the processor, and when the computer program is executed by the processor, the above method is executed.

[0230] The present invention further provides a storage medium 43 on which a computer program is stored. When the computer program is run by the processor 41, the above method is executed.

[0231] Among them, the storage medium 43 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0232] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A few-sample semantic segmentation method based on image-text fusion, characterized in that: The following steps are involved: Collect and preprocess a dataset, wherein the dataset is grouped by category, each group containing a number of support images and query images; Extract multi-scale image features using pre-trained deep neural networks; Using a text encoder to extract text features of a preset text prompt; Fusion of extracted text features with image features; Use the attention mechanism to mine the correlation between the supporting image features and the query image features; The fused image features are upsampled and decoded to generate segmentation prediction results.

2. The method for few-sample semantic segmentation based on image-text fusion according to claim 1, characterized in that: The steps of collecting and preprocessing the data set include: The dataset is grouped according to the total image category of the dataset, and the dataset is divided into four groups, each group is called a fold; The data in each fold consists of support images and query images, where the support images are denoted as The query image is denoted as D q ={X,Y}, where X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value set in the few-shot scenario, and represents the number of supporting images for each category; The image categories in the dataset are evenly distributed into four folds to ensure that the categories contained in each fold do not overlap with each other.

3. The few-sample semantic segmentation method based on image-text fusion according to claim 1, characterized in that: The step of extracting multi-scale image features using a pre-trained deep neural network includes: The input image X i Input into the pre-trained deep neural network to extract multi-scale image features; Through different levels of the pre-trained deep neural network, four scales of image features are extracted in turn, which are denoted as X1, X2, X3, and X4, respectively, where X1, X2, X3, and X4 represent image features from shallow to deep layers in turn; Perform global average pooling on the deep feature map to obtain the global features of the image The formula is: Global Features Together with the deep feature map X4, it is input into the multi-head self-attention mechanism to enhance the feature expression ability of the image. The formula is: in, represents the enhanced global features, and Z represents the enhanced image features.

4. The method for few-sample semantic segmentation based on image-text fusion according to claim 1, characterized in that: The step of using a text encoder to extract text features of a preset text prompt comprises: Preset text prompts, where the text prompts are related to the target category in the image and are used to indicate the target category to be segmented; Inputting the text prompt into a text encoder Transformer to extract text features related to the prompt; The text encoder Transformer processes the input text prompts, captures the contextual information in the text through the self-attention mechanism, and generates a feature vector t representing the semantics of the text. The formula is: t=Transformer(text prompt) Wherein, text prompt represents the preset text prompt, and t represents the text feature vector extracted from the text prompt.

5. The method for few-sample semantic segmentation based on image-text fusion according to claim 1, characterized in that: The step of fusing the extracted text features with the image features comprises: Through a Transformer decoder consisting of three layers of Transformers, the image feature Z is fused with the text feature t to generate a text feature V containing visual information. The formula is: Among them, Z represents the multi-scale features extracted from the image, t represents the text features generated by the text prompt, and V represents the fused text features containing visual information; Using the learnable scaling parameter α, the fused visual information V is weightedly combined with the original text feature t to generate the updated text feature t', which is: t′=t+αV Among them, α is a learnable parameter that controls the influence of visual information on text features, t' is the updated text feature, which contains semantic information from the image; Calculate the correlation between the updated text feature t' and the image feature Z to generate the score map S. The formula is: S=Zt′ The score map S is concatenated with the original deep image feature X4 to form the fused image feature X'4 for subsequent decoding and segmentation. The formula is: X′4=[X4,S] Among them, X4 is the original deep image feature, S is the score map, and X'4 is the fused image feature, which combines the multimodal information of image and text.

6. The method for few-sample semantic segmentation based on image-text fusion according to claim 1, characterized in that: The step of using the attention mechanism to mine the correlation between the supporting image features and the query image features comprises: The features F extracted from the support image s and the features F extracted from the query image q Flattening is performed to convert the multidimensional feature map into a one-dimensional vector. The formula is: F′ s =flatten(F s ) F′ q =flatten(F q ) The flattened support image feature F' s and query image features F' q Perform linear mapping respectively to generate query vector Q, key vector K and value vector V. The formula is: Q=linear(flatten(F' q )) K,V=linear(flatten(F' s )) The self-attention mechanism is used to calculate the correlation between the query vector Q and the key vector K. The correlation is calculated by the dot product of the query vector Q and the key vector K, and the attention weight is generated by the normalization operation, and then the value vector V is weighted. The formula is: Among them, softmax(·) represents the normalization operation, is the scaling factor, d is the dimension of the vector, and Atten(Q,K,V) is the weighted feature representation; Through the above attention mechanism, we can mine the supporting image features F s and query image features F q The correlation between them is used to obtain enhanced query image features. The formula is: in, The query image features are enhanced by the attention mechanism and contain semantic information related to the supporting image features, which are further used to improve the accuracy of few-shot semantic segmentation.

7. The few-sample semantic segmentation method based on image-text fusion according to claim 1, characterized in that: The steps of upsampling the fused image features and decoding to generate segmentation prediction results include: The fused image feature X'4 is upsampled to restore the feature map to the same resolution as the input image. Upsampling can be achieved through bilinear interpolation or transposed convolution. The formula is: X up =Upsample(X′4) Among them, X'4 represents the fused image features, X up is the feature map after upsampling; In the upsampling process, the upsampled feature map X up The image features X of the corresponding scale extracted in the encoding stage i Fusion is performed to retain global semantic information and detail information. The formula is: X fused =Concat(X up ,X i ) Among them, Concata(·) represents the feature concatenation operation, X i represents the multi-scale features extracted in the encoding stage, X fused Represents the fused feature map; The fused feature map X fused Input to the decoder, which consists of multiple convolutional layers or transposed convolutional layers to gradually restore and generate the final segmentation prediction results The specific expression is: Among them, Decoder(·) represents the decoder, represents the segmentation prediction mask output by the decoder; The final segmentation prediction mask It has the same resolution as the original input image, and the value of each pixel represents the probability or classification label of the pixel belonging to a certain category, which is used for the prediction output of the few-shot semantic segmentation task.

8. A few-sample semantic segmentation device based on image-text fusion, used to implement the few-sample semantic segmentation method based on image-text fusion according to any one of claims 1 to 7, characterized in that: include: Data preprocessing module, used to collect and preprocess data sets; Image feature extraction module, which uses pre-trained deep neural network to extract multi-scale image features; A text feature extraction module, which uses a text encoder to extract text features of a preset text prompt; Feature fusion module, which fuses the extracted text features with image features; The correlation mining module uses the attention mechanism to mine the correlation between the supporting image features and the query image features; The decoding module upsamples the fused image features and decodes them to generate segmentation prediction results.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images