A small sample semantic segmentation method and system fusing class label semantics

Abstract

The application relates to the field of image processing, and proposes a small sample semantic segmentation method and system fusing class label semantics, wherein a prior information generation module is designed, multi-modal data fusion of image data information and text data information serving as class labels is realized, a semantic segmentation model can more accurately understand image content, a multi-scale fusion module is further designed, original detail information of an image is further protected, the calculation performance and channel fusion capability of the semantic segmentation model are greatly improved, the parameter quantity of a decoder is greatly reduced while ensuring the speech segmentation precision, and the accuracy of target recognition and target positioning of semantic segmentation is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a few-sample semantic segmentation method and system that integrates category label semantics. Background Technology

[0002] With the rapid development of computer vision technology, semantic segmentation technology, as an extremely important part of computer vision technology, has gradually appeared in more application scenarios. The key point of semantic segmentation technology is how to accurately identify target regions and target objects through semantic segmentation models.

[0003] In existing technologies, semantic segmentation models are typically trained solely on image data. Different types of image data are used as guide images to enhance the semantic understanding capabilities of the semantic segmentation model, enabling it to understand image content more accurately. However, this is still a single-modal semantic segmentation model, which is insufficient in terms of the breadth and richness of data information.

[0004] Therefore, how to design a semantic segmentation model that can integrate multimodal data information, so as to enable the model to understand the image content more accurately and improve the accuracy of target recognition and target localization in semantic segmentation, has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a few-sample semantic segmentation method and system that integrates category label semantics. This is achieved by designing a prior information generation module, which includes a text-image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text-image adaptation submodule adds weights to each channel of the image features using text features. The prior generation submodule generates prior information for the query image using image and text information. The cross-attention submodule establishes dense attention between image information to obtain the relationship between the target category in the supporting image and the query image. The assembly submodule performs feature fusion on the generated information through TE blocks and CBAM blocks, thereby achieving multimodal data fusion of image data information and text data information as category labels. This enables the semantic segmentation model to more accurately understand the image content. Furthermore, by designing a multi-scale fusion module, the original detailed information of the image is further preserved, greatly improving the computational performance and channel fusion capability of the semantic segmentation model. While ensuring the accuracy of speech segmentation, the number of decoder parameters is also greatly reduced. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation.

[0006] This invention proposes a few-sample semantic segmentation method that integrates category label semantics, comprising:

[0007] The query image, supporting images, supporting masks, and text category labels are obtained and preprocessed to obtain multi-layer query features, multi-layer supporting features, and text features.

[0008] The high-level, mid-level, and low-level features in the multi-layer query features and multi-layer support features are respectively input into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation operation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information.

[0009] Prior information from high-level, mid-level, and low-level query images is input into a multi-scale fusion module to obtain the final segmentation prediction.

[0010] In summary, based on the aforementioned few-sample semantic segmentation method that integrates category label semantics, this invention designs a prior information generation module. This module includes: a text-image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text-image adaptation submodule adds weights to each channel of the image features using text features. The prior generation submodule generates prior information for the query image using image and text information. The cross-attention submodule establishes dense attention between image information to obtain the relationship between the target category in the supporting image and the query image. The assembly submodule performs feature fusion on the generated information through TE blocks and CBAM blocks, thereby achieving multimodal data fusion of image data information and text data information as category labels. This enables the semantic segmentation model to more accurately understand the image content. Furthermore, by designing a multi-scale fusion module, the original detailed information of the image is further preserved, greatly improving the computational performance and channel fusion capability of the semantic segmentation model. While ensuring the accuracy of speech segmentation, the number of decoder parameters is also greatly reduced. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation. Specifically, the process involves acquiring and preprocessing query images, supporting images, supporting masks, and text category labels to obtain multi-layer query features, multi-layer supporting features, and text features. The high-level, mid-level, and low-level features from the multi-layer query features and multi-layer supporting features are then input into a prior information generation module to generate high-level, mid-level, and low-level query image prior information. This prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation to obtain multimodal prior information. The prior information generation submodule performs basic prior information generation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information. Finally, the high-level, mid-level, and low-level query image prior information are input into a multi-scale fusion module to obtain the final segmentation prediction. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation.

[0011] Furthermore, the step of obtaining the query image, support image, support mask, and text category labels and performing preprocessing to obtain multi-layer query features, multi-layer support features, and text features includes:

[0012] Get the query image Supports images Support for masks and text category labels Then, the text category labels Input the text encoder of the CLIP model to obtain fixed-dimensional text features. ;

[0013] The query image and supporting images Input the backbone network for feature extraction, and obtain the query features respectively. and supporting features , ,in Indicates the height of the feature map, Indicates the width of the feature map. The query feature represents the number of channels in the feature vector. and the supporting features Including low-level query features Low-level support features Mid-level query features Mid-layer support features High-level query features High-level support characteristics .

[0014] Furthermore, the text-image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information, including the following steps:

[0015] The fully connected layer in the text-image adaptation submodule applies text features A linear projection operation is performed, and the formula for the linear projection operation is as follows:

[0016]

[0017] in, Indicates a fully connected layer. Text encoder representing the CLIP model, text features , It is the dimension of the embedding space;

[0018] Convolutional blocks in the text-image adaptation submodule for query features and supporting features Perform channel transformation operation to transform the query features and the supporting features The scaling transformation is The dimension transformation block in the text-image adaptation submodule further transforms the query features after channel transformation. and the supporting features A dimensional transformation operation is performed, and the formula for the dimensional transformation operation is as follows:

[0019]

[0020]

[0021] in, This represents the query features after dimensional transformation. This represents the supporting features after dimensionality transformation. , This represents the dimension transformation function. Represents the convolution function;

[0022] The text features The query features and the supporting features A multimodal fusion operation is performed, and the formula for the multimodal fusion operation is as follows:

[0023]

[0024]

[0025] in, This represents the query features after multimodal fusion. This represents the support features after multimodal fusion. Representing text features The transpose of the matrix;

[0026] Support for mask A bilinear interpolation downsampling operation is performed, and the formula for the bilinear interpolation downsampling operation is as follows:

[0027]

[0028] in, This represents the support mask after downsampling. The downsampling function is represented by the support mask. Size and the supporting features same;

[0029] The support mask Query features and supporting features In the multi-head attention block of the input text image adaptation submodule, the support mask... Query features and supporting features Each as a multi-head attention block matrix, Matrix and Matrix, to obtain multimodal prior information The acquisition of multimodal prior information The formula is as follows:

[0030]

[0031] in, This represents a feedforward neural network layer. Presentation layer normalization layer, This represents a multi-head attention layer, and the mechanism function of the multi-head attention layer is as follows:

[0032]

[0033] in, Let Q be the dimension of the Q matrix and K matrix. express The transpose of a matrix.

[0034] Furthermore, the prior information generation submodule performs basic prior information generation operations to obtain basic prior information, including the following steps:

[0035] Based on query characteristics and text features Generate the first rough basic prior information The generation of the first rough basic prior information The formula is as follows:

[0036]

[0037] in, Represents the cosine similarity function. ;

[0038] For the first rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows:

[0039]

[0040] in, for , This indicates retrieving the maximum value. This indicates obtaining the minimum value. ;

[0041] Based on the downsampled support mask Query features and supporting features Generate second coarse basic prior information The generation of the second coarse basic prior information The formula is as follows:

[0042]

[0043] in, , Indicate query characteristics The transpose of the matrix, express Norm, This indicates element-wise multiplication;

[0044] For the second rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows:

[0045]

[0046] in, ;

[0047] Based on the first rough foundation prior information and the second rough foundational prior information Obtain basic prior information The acquisition of basic prior information The formula is as follows:

[0048]

[0049] in, This is a hyperparameter, set to 0.5.

[0050] Furthermore, the step of the cross-attention submodule performing the attention-weighted prior information generation operation to obtain attention-weighted prior information includes:

[0051] query features Support features Support for masks The first multi-head attention block of the input cross-attention submodule, the query feature The supporting features and the support mask Each serves as the first multi-head attention block matrix, Matrix and Matrix, to obtain coarse attention-weighted prior information The acquisition of coarse attention-weighted prior information The formula is as follows:

[0052]

[0053] in, , ;

[0054] The query features and the coarse attention-weighted prior information The second multi-head attention block of the input cross-attention submodule, the query feature As the second multi-head attention block Matrix, the coarse attention-weighted prior information As the second multi-head attention block Matrix and Matrix, to obtain attention-weighted prior information The acquisition of attention-weighted prior information The formula is as follows:

[0055]

[0056] in, This indicates weighted aggregation.

[0057] Furthermore, the step of inputting the multimodal prior information, the basic prior information, and the attention-weighted prior information into the assembly submodule to obtain the prior information of the query image includes:

[0058] Multimodal prior information Basic prior information Attention-weighted prior information In the input assembly submodule, the assembly submodule is a residual structure, and the main path of the residual structure will contain the original multimodal prior information. The aforementioned basic prior information and the attention-weighted prior information In the input splicing block, the residual path of the residual structure includes a TE encoder block and a CBAM block. The CBAM block includes a channel attention mechanism block and a spatial attention mechanism block. Both the channel attention mechanism block and the spatial attention mechanism block are residual structures.

[0059] The multimodal prior information The aforementioned basic prior information and the attention-weighted prior information To perform fusion in order to obtain prior information about fusion. The acquisition of fusion prior information The formula is as follows:

[0060]

[0061] The fusion prior information Input into the splicing block;

[0062] Then the fused prior information Input into the TE encoder block to obtain first prior information. The acquisition of the first prior information The formula is as follows:

[0063]

[0064] in, This represents a stacked Transformer encoder that transmits the first prior information. Input into the splicing block;

[0065] The fused prior information is then... Input into the CBAM block to obtain the second prior information. The acquisition of the second prior information The formula is as follows:

[0066]

[0067] in, This indicates a CBAM block, which includes the second prior information. Input into the splicing block;

[0068] The multimodal prior information The aforementioned basic prior information The attention-weighted prior information The fusion of prior information The first prior information and the second prior information The splicing is performed along the channel dimension, and the splicing formula is as follows:

[0069]

[0070] in, This indicates a query for prior information about an image. , This represents the concatenation function.

[0071] Furthermore, the step of inputting the prior information of the query images at high, middle, and low levels into the multi-scale fusion module to obtain the final segmentation prediction includes:

[0072] Prior information of query images at high, middle, and low levels is input into a multi-scale fusion module. The multi-scale fusion module includes a dual-channel convolutional block and a final decoder. The dual-channel convolutional block includes a first convolutional path and a second convolutional path. The first convolutional path includes a 3x3 convolutional layer, a batch normalization layer, and an ELU activation layer. The second convolutional path includes a 1x1 convolutional layer, a group normalization layer, and a ReLU activation layer. The final decoder includes a convolutional layer, a batch normalization layer, an ELU activation layer, and a CBAM block.

[0073] After channel discarding processing, the prior information of the query images in the high-level, mid-level, and low-level layers is input into dual-channel convolutional blocks for feature aggregation. The processed prior information of the query images in the high-level layers is upsampled and then fused with the processed prior information of the query images in the mid-level layers using element-wise addition. The fused prior information of the query images is then input into dual-channel convolutional blocks for feature aggregation. The fused prior information of the query images is then upsampled and then fused with the processed prior information of the query images in the low-level layers using element-wise addition, and then input into dual-channel convolutional blocks for feature aggregation. After upsampling, it is input into the final decoder. The final decoder obtains the final segmentation prediction and performs loss calculation. The formula for loss calculation is as follows:

[0074]

[0075]

[0076]

[0077] in, This represents the predicted mask of the query image and its corresponding ground truth mask. The Dice loss between them Indicates CE loss, This indicates the total loss.

[0078] This invention proposes a few-sample semantic segmentation system that integrates category label semantics, comprising:

[0079] The feature acquisition module is used to acquire the query image, support image, support mask and text category label and preprocess them to obtain multi-layer query features, multi-layer support features and text features;

[0080] The prior information generation module is used to input the high-level, mid-level, and low-level features from the multi-layer query features and the multi-layer support features into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs a basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs an attention-weighted prior information generation operation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information.

[0081] The multi-scale fusion module is used to input prior information of high-level, mid-level, and low-level query images into the multi-scale fusion module to obtain the final segmentation prediction.

[0082] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the few-sample semantic segmentation method as described above that integrates category label semantics.

[0083] The present invention also provides a computer device, the computer device including a memory and a processor, wherein:

[0084] The memory is used to store computer programs;

[0085] When the processor executes the computer program stored in the memory, it implements the few-sample semantic segmentation method that integrates category label semantics as described above. Attached Figure Description

[0086] Figure 1 This is a flowchart of the few-sample semantic segmentation method that integrates category label semantics proposed in the first embodiment of the present invention;

[0087] Figure 2 This is a flowchart of the few-sample semantic segmentation method that integrates category label semantics proposed in the second embodiment of the present invention;

[0088] Figure 3 This is a schematic diagram of the structure of the few-sample semantic segmentation system that integrates category label semantics proposed in the third embodiment of the present invention.

[0089] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0090] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0091] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0092] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0093] Please see Figure 1 The diagram shows a flowchart of a few-sample semantic segmentation method that integrates category label semantics according to a first embodiment of the present invention. This few-sample semantic segmentation method that integrates category label semantics includes steps S01 to S03, wherein:

[0094] Step S01: Obtain the query image, supporting images, supporting masks, and text category labels, and perform preprocessing to obtain multi-layer query features, multi-layer supporting features, and text features;

[0095] It should be noted that the query image is obtained in this embodiment. Supports images Support for masks and text category labels Then, the text category labels Input the text encoder of the CLIP model to obtain fixed-dimensional text features. The query image and supporting images The input is processed by the backbone network for feature extraction. In this embodiment, the backbone networks selected are ResNet50 and ResNet101, which are pre-trained on ImageNet1K, to obtain the query features. and supporting features , ,in Indicates the height of the feature map, Indicates the width of the feature map. The query feature represents the number of channels in the feature vector. and the supporting features Including low-level query features Low-level support features Mid-level query features Mid-layer support features High-level query features High-level support characteristics In this embodiment, the low-level, middle-level, and high-level layers correspond to the 3rd, 6th, and 9th layers in the multi-layer feature, respectively. The image sizes of the low-level, middle-level, and high-level layers are respectively [missing information - likely a percentage of the original image's dimensions]. , and .

[0096] Step S02: Input the high-level, mid-level, and low-level features of the multi-layer query features and the multi-layer support features into the prior information generation module to generate prior information for the query image at the high, mid, and low levels. The prior information generation module includes a text image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation to obtain multimodal prior information. The prior generation submodule performs basic prior information generation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation to obtain attention-weighted prior information. Input the multimodal prior information, basic prior information, and attention-weighted prior information into the assembly submodule to obtain the query image prior information.

[0097] It should be noted that in this embodiment, the text image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information. This is achieved by first processing text features by the fully connected layer in the text image adaptation submodule. A linear projection operation is performed, and the formula for the linear projection operation is as follows:

[0098]

[0099] in, Indicates a fully connected layer. Text encoder representing the CLIP model, text features , It is the dimension of the embedding space, and then the convolutional blocks in the text image adaptation submodule are used to match the query features. and supporting features Perform channel transformation operation to transform the query features and the supporting features The scaling transformation is The dimension transformation block in the text-image adaptation submodule further transforms the query features after channel transformation. and the supporting features A dimensional transformation operation is performed, and the formula for the dimensional transformation operation is as follows:

[0100]

[0101]

[0102] in, This represents the query features after dimensional transformation. This represents the supporting features after dimensionality transformation. , This represents the dimension transformation function. Represents the convolution function, and then the text features are... The query features and the supporting features A multimodal fusion operation is performed, and the formula for the multimodal fusion operation is as follows:

[0103]

[0104]

[0105] in, This represents the query features after multimodal fusion. This represents the support features after multimodal fusion. Representing text features The transpose matrix is ​​used to add weights to each channel of the image features based on text features, making the model pay more attention to the features of the channels with higher weights, and also supporting masking. A bilinear interpolation downsampling operation is performed, and the formula for the bilinear interpolation downsampling operation is as follows:

[0106]

[0107] in, This represents the support mask after downsampling. The downsampling function is represented by the support mask. Size and the supporting features Similarly, finally, the support mask will be used. Query features and supporting features In the multi-head attention block of the input text image adaptation submodule, the support mask... Query features and supporting features Each as a multi-head attention block matrix, Matrix and Matrix, to obtain multimodal prior information The acquisition of multimodal prior information The formula is as follows:

[0108]

[0109] in, This represents a feedforward neural network layer. Presentation layer normalization layer, This represents a multi-head attention layer, and the mechanism function of the multi-head attention layer is as follows:

[0110]

[0111] in, Let Q be the dimension of the Q matrix and K matrix. express The transpose of a matrix.

[0112] It should be noted that in this embodiment, the prior information generation submodule performs basic prior information generation operations to obtain basic prior information, which is based on query features. and text features Generate the first rough basic prior information The generation of the first rough basic prior information The formula is as follows:

[0113]

[0114] in, Represents the cosine similarity function. For the first rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows:

[0115]

[0116] in, for , This indicates retrieving the maximum value. This indicates obtaining the minimum value. Based on the downsampled support mask Query features and supporting features Generate second coarse basic prior information The generation of the second coarse basic prior information The formula is as follows:

[0117]

[0118] in, , Indicate query characteristics The transpose of the matrix, express Norm, This represents element-wise multiplication. Through this operation, only foreground information is retained in the image features, eliminating background interference, and improving the second coarse foundational prior information. The deviations are standardized, and the formula for deviation standardization is as follows:

[0119]

[0120] in, Based on the first rough foundational prior information and the second rough foundational prior information Obtain basic prior information The acquisition of basic prior information The formula is as follows:

[0121]

[0122] in, This is a hyperparameter, set to 0.5.

[0123] It should be noted that in this embodiment, the cross-attention submodule performs an attention-weighted prior information generation operation to obtain attention-weighted prior information. This is done by first generating the query features... Support features Support for masks The first multi-head attention block of the input cross-attention submodule, the query feature The supporting features and the support mask Each serves as the first multi-head attention block matrix, Matrix and Matrix, to obtain coarse attention-weighted prior information The acquisition of coarse attention-weighted prior information The formula is as follows:

[0124]

[0125] in, , Then the query features and the coarse attention-weighted prior information The second multi-head attention block of the input cross-attention submodule, the query feature As the second multi-head attention block Matrix, the coarse attention-weighted prior information As the second multi-head attention block Matrix and Matrix, to obtain attention-weighted prior information The acquisition of attention-weighted prior information The formula is as follows:

[0126]

[0127] in, This indicates weighted aggregation. This module performs weighted aggregation on features in the query image and supporting images, identifying regions with high attention scores as prior information of the foreground and regions with low attention scores as prior information of the background.

[0128] It should be noted that, in this embodiment, the rotor module obtains the prior information of the query image by first processing the multimodal prior information. Basic prior information Attention-weighted prior information In the input assembly submodule, the assembly submodule is a residual structure, and the main path of the residual structure will contain the original multimodal prior information. The aforementioned basic prior information and the attention-weighted prior information In the input splicing block, the residual path of the residual structure includes a TE encoder block and a CBAM block. The CBAM block includes a channel attention mechanism block and a spatial attention mechanism block. Both the channel attention mechanism block and the spatial attention mechanism block are residual structures that incorporate the multimodal prior information. The aforementioned basic prior information and the attention-weighted prior information To perform fusion in order to obtain prior information about fusion. The acquisition of fusion prior information The formula is as follows:

[0129]

[0130] The fusion prior information The input is then incorporated into the splicing block, and the fusion prior information is then... Input into the TE encoder block to obtain first prior information. The acquisition of the first prior information The formula is as follows:

[0131]

[0132] in, This represents a stacked Transformer encoder that transmits the first prior information. The first prior information in the input concatenation block This includes the local and global dependencies of each element in the sequence, as well as the interactions between elements. Through a self-attention mechanism, the model can capture long-range dependencies in the sequence, which is crucial for understanding complex sequence data. Furthermore, it integrates prior information. Input into the CBAM block to obtain the second prior information. The acquisition of the second prior information The formula is as follows:

[0133]

[0134] in, This represents a CBAM block, which infers attention maps sequentially along the channel and spatial dimensions of the feature map. These attention maps are then multiplied with the input feature map to achieve adaptive feature optimization, incorporating the second prior information. The multimodal prior information is included in the input splicing block. The aforementioned basic prior information The attention-weighted prior information The fusion of prior information The first prior information and the second prior information The splicing is performed along the channel dimension, and the splicing formula is as follows:

[0135]

[0136] in, This indicates a query for prior information about an image. , This represents the concatenation function.

[0137] Step S03: Input the prior information of the query images at high, middle and low levels into the multi-scale fusion module to obtain the final segmentation prediction;

[0138] It should be noted that the multi-scale fusion module in this embodiment includes a dual-channel convolutional block and a final decoder. The dual-channel convolutional block includes a first convolutional path and a second convolutional path. The first convolutional path includes a 3x3 convolutional layer, a batch normalization layer, and an ELU activation layer. The second convolutional path includes a 1x1 convolutional layer, a group normalization layer, and a ReLU activation layer. The final decoder includes a convolutional layer, a batch normalization layer, an ELU activation layer, and a CBAM block. The prior information of the query images at high, middle, and low levels is input into the multi-scale fusion module. After channel discarding processing, the prior information of the query images at high, middle, and low levels is input into the dual channels respectively. In the convolutional block, feature aggregation processing is performed. The processed prior information of the query image in the higher layers is upsampled and then fused element-wise with the processed prior information of the query image in the middle layers. The fused prior information of the query image is then input into a dual-channel convolutional block for feature aggregation processing. The processed and fused prior information of the query image in the lower layers is then upsampled and fused element-wise with the processed prior information of the query image in the lower layers. This fused prior information is then input into a dual-channel convolutional block for feature aggregation processing and upsampling before being input into the final decoder. The final decoder obtains the final segmentation prediction and performs loss calculation. The formula for loss calculation is as follows:

[0139]

[0140]

[0141]

[0142] in, This represents the predicted mask of the query image and its corresponding ground truth mask. The Dice loss between them Indicates CE loss, The total loss is represented by the output of the three different levels of prior information generation modules. After channel dropping, convolutional blocks and upsampling, the output features are compared with the real mask of the downsampled query image to calculate the multi-scale loss, which further increases the generalization ability of the model.

[0143] In summary, based on the aforementioned few-sample semantic segmentation method that integrates category label semantics, this invention designs a prior information generation module. This module includes: a text-image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text-image adaptation submodule adds weights to each channel of the image features using text features. The prior generation submodule generates prior information for the query image using image and text information. The cross-attention submodule establishes dense attention between image information to obtain the relationship between the target category in the supporting image and the query image. The assembly submodule performs feature fusion on the generated information through TE blocks and CBAM blocks, thereby achieving multimodal data fusion of image data information and text data information as category labels. This enables the semantic segmentation model to more accurately understand the image content. Furthermore, by designing a multi-scale fusion module, the original detailed information of the image is further preserved, greatly improving the computational performance and channel fusion capability of the semantic segmentation model. While ensuring the accuracy of speech segmentation, the number of decoder parameters is also greatly reduced. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation. Specifically, the process involves acquiring and preprocessing query images, supporting images, supporting masks, and text category labels to obtain multi-layer query features, multi-layer supporting features, and text features. The high-level, mid-level, and low-level features from the multi-layer query features and multi-layer supporting features are then input into a prior information generation module to generate high-level, mid-level, and low-level query image prior information. This prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation to obtain multimodal prior information. The prior information generation submodule performs basic prior information generation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information. Finally, the high-level, mid-level, and low-level query image prior information are input into a multi-scale fusion module to obtain the final segmentation prediction. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation.

[0144] Please see Figure 2 The diagram shows a flowchart of a few-sample semantic segmentation method that integrates category label semantics according to a second embodiment of the present invention. This few-sample semantic segmentation method that integrates category label semantics includes steps S11 to S17, wherein:

[0145] Step S11: After obtaining the query image, support image, support mask and text category label, input the text category label into the text encoder of the CLIP model to obtain fixed-dimensional text features, input the query image and support image into the backbone network for feature extraction, and obtain multi-layer query features and multi-layer support features respectively.

[0146] Step S12: Input the high-level, mid-level, and low-level features from the multi-layer query features and multi-layer support features into the prior information generation module to generate prior information for the query images at the high, mid, and low levels.

[0147] Step S13: The fully connected layer in the text image adaptation submodule performs linear projection operation on the text features; the convolutional block in the text image adaptation submodule performs channel transformation operation on the query features and support features; the dimension transformation block in the text image adaptation submodule then performs dimension transformation operation on the channel-transformed query features and support features; the text features, query features and support features are fused into a multimodal model; bilinear interpolation downsampling operation is performed on the support mask; and the support mask, query features and support features are input into the multi-head attention block of the text image adaptation submodule to obtain multimodal prior information.

[0148] Step S14: Generate first coarse basic prior information based on query features and text features, perform deviation standardization on the first coarse basic prior information, generate second coarse basic prior information based on downsampled support mask, query features and support features, perform deviation standardization on the second coarse basic prior information, and obtain basic prior information based on the first coarse basic prior information and the second coarse basic prior information.

[0149] Step S15: Input the query feature, supporting feature, and supporting mask into the first multi-head attention block of the cross-attention submodule to obtain coarse attention weighted prior information; input the query feature and the coarse attention weighted prior information into the second multi-head attention block of the cross-attention submodule to obtain attention weighted prior information.

[0150] Step S16: Input the multimodal prior information, basic prior information, and attention-weighted prior information into the assembly submodule, fuse the multimodal prior information, basic prior information, and attention-weighted prior information to obtain fused prior information, input the fused prior information into the stitching block, then input the fused prior information into the TE encoder block to obtain the first prior information, then input the fused prior information into the CBAM block to obtain the second prior information, and stitch the multimodal prior information, basic prior information, attention-weighted prior information, fused prior information, first prior information, and second prior information together in the channel dimension;

[0151] Step S17: Input the prior information of the query images of high-level, mid-level, and low-level layers into the multi-scale fusion module. After channel discarding processing, the prior information of the query images of high-level, mid-level, and low-level layers is input into the dual-channel convolutional block for feature aggregation processing. The processed prior information of the query images of high-level layers is upsampled and then fused with the processed prior information of the query images of mid-level layers by element-wise addition. The fused prior information of the query images is input into the dual-channel convolutional block for feature aggregation processing. The processed and fused prior information of the query images is upsampled and then fused with the processed prior information of the query images of low-level layers by element-wise addition and input into the dual-channel convolutional block for feature aggregation processing and upsampling. The input is then input into the final decoder. The final decoder obtains the final segmentation prediction and performs loss calculation.

[0152] It should be noted that, in the comparative experiment between this invention and existing semantic segmentation models, ResNet101 was used as the backbone network, and a 1-shot configuration was adopted. The comparison results in the dataset are shown in Table 1 below. The comparison results in the dataset are shown in Table 2 below:

[0153] Table 1

[0154]

[0155] Table 2

[0156]

[0157] In Table 1 above, dataset 1, dataset 2, dataset 3, and dataset 4 represent respectively... , , and In Table 2 above, dataset 1, dataset 2, dataset 3, and dataset 4 respectively represent , , and As can be seen from Tables 1 and 2 above, the present invention has made significant progress in all evaluation indicators and experimental groups, especially in the average intersection-union ratio, which has an excellent effect and greatly improves the performance of the semantic segmentation model.

[0158] In summary, based on the aforementioned few-sample semantic segmentation method that integrates category label semantics, this invention designs a prior information generation module. This module includes: a text-image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text-image adaptation submodule adds weights to each channel of the image features using text features. The prior generation submodule generates prior information for the query image using image and text information. The cross-attention submodule establishes dense attention between image information to obtain the relationship between the target category in the supporting image and the query image. The assembly submodule performs feature fusion on the generated information through TE blocks and CBAM blocks, thereby achieving multimodal data fusion of image data information and text data information as category labels. This enables the semantic segmentation model to more accurately understand the image content. Furthermore, by designing a multi-scale fusion module, the original detailed information of the image is further preserved, greatly improving the computational performance and channel fusion capability of the semantic segmentation model. While ensuring the accuracy of speech segmentation, the number of decoder parameters is also greatly reduced. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation. Specifically, the process involves acquiring and preprocessing query images, supporting images, supporting masks, and text category labels to obtain multi-layer query features, multi-layer supporting features, and text features. The high-level, mid-level, and low-level features from the multi-layer query features and multi-layer supporting features are then input into a prior information generation module to generate high-level, mid-level, and low-level query image prior information. This prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation to obtain multimodal prior information. The prior information generation submodule performs basic prior information generation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information. Finally, the high-level, mid-level, and low-level query image prior information are input into a multi-scale fusion module to obtain the final segmentation prediction. This invention significantly improves the accuracy of target recognition and target localization in semantic segmentation.

[0159] Please see Figure 3 The figure shown is a schematic diagram of the structure of the few-sample semantic segmentation system that integrates category label semantics proposed in the third embodiment of the present invention. The system includes:

[0160] The feature acquisition module 10 is used to acquire the query image, support image, support mask and text category label and perform preprocessing to obtain multi-layer query features, multi-layer support features and text features;

[0161] The prior information generation module 20 is used to input the high-level, mid-level, and low-level features of the multi-layer query features and the multi-layer support features into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs a basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs an attention-weighted prior information generation operation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information.

[0162] The multi-scale fusion module 30 is used to input the prior information of the query images at high, medium and low levels into the multi-scale fusion module to obtain the final segmentation prediction.

[0163] Furthermore, the feature acquisition module 10 includes:

[0164] The feature acquisition unit 101 is used to acquire the query image, support image, support mask and text category label and preprocess them to obtain multi-layer query features, multi-layer support features and text features.

[0165] Furthermore, the prior information generation module 20 includes:

[0166] The generation unit 201 is used to input the high-level, mid-level, and low-level features of the multi-layer query features and the multi-layer support features into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs a basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs an attention-weighted prior information generation operation to obtain attention-weighted prior information.

[0167] The splicing unit 202 is used to input the multimodal prior information, the basic prior information and the attention-weighted prior information into the assembly submodule to obtain the query image prior information.

[0168] Furthermore, the multi-scale fusion module 30 includes:

[0169] The fusion unit 301 is used to input the prior information of the query images at high, medium and low levels into the multi-scale fusion module to obtain the final segmentation prediction.

[0170] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the aforementioned few-sample semantic segmentation method that integrates category label semantics.

[0171] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned few-sample semantic segmentation method that integrates category label semantics.

[0172] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0173] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0174] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0175] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0176] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A few-sample semantic segmentation method that integrates category label semantics, characterized in that, include: The query image, supporting images, supporting masks, and text category labels are obtained and preprocessed to obtain multi-layer query features, multi-layer supporting features, and text features. The high-level, mid-level, and low-level features in the multi-layer query features and multi-layer support features are respectively input into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs attention-weighted prior information generation operation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information. The text-image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information, including the following steps: The fully connected layer in the text-image adaptation submodule applies text features A linear projection operation is performed, and the formula for the linear projection operation is as follows: in, Indicates a fully connected layer. Text encoder representing the CLIP model, text features , It is the dimension of the embedding space; Convolutional blocks in the text-image adaptation submodule for query features and supporting features Perform channel transformation operation to transform the query features and the supporting features The scaling transformation is The dimension transformation block in the text-image adaptation submodule further transforms the query features after channel transformation. and the supporting features A dimensional transformation operation is performed, and the formula for the dimensional transformation operation is as follows: in, This represents the query features after dimensional transformation. This represents the supporting features after dimensionality transformation. , This represents the dimension transformation function. Represents the convolution function; The text features The query features and the supporting features A multimodal fusion operation is performed, and the formula for the multimodal fusion operation is as follows: in, This represents the query features after multimodal fusion. This represents the support features after multimodal fusion. Representing text features The transpose of the matrix; Support for mask A bilinear interpolation downsampling operation is performed, and the formula for the bilinear interpolation downsampling operation is as follows: in, This represents the support mask after downsampling. The downsampling function is represented by the support mask. Size and the supporting features same; The support mask Query features and supporting features In the multi-head attention block of the input text image adaptation submodule, the support mask... Query features and supporting features Each as a multi-head attention block matrix, Matrix and Matrix, to obtain multimodal prior information The acquisition of multimodal prior information The formula is as follows: in, This represents a feedforward neural network layer. Presentation layer normalization layer, This represents a multi-head attention layer, and the mechanism function of the multi-head attention layer is as follows: in, Let Q be the dimension of the Q matrix and K matrix. express The transpose of a matrix; The prior information generation submodule performs basic prior information generation operations to obtain basic prior information, including the following steps: Based on query characteristics and text features Generate the first rough basic prior information The generation of the first rough basic prior information The formula is as follows: in, Represents the cosine similarity function. ; For the first rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows: in, for , This indicates retrieving the maximum value. This indicates obtaining the minimum value. ; Based on the downsampled support mask Query features and supporting features Generate second coarse basic prior information The generation of the second coarse basic prior information The formula is as follows: in, , Indicate query characteristics The transpose of the matrix, express Norm, This indicates element-wise multiplication; For the second rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows: in, ; Based on the first rough foundation prior information and the second rough foundational prior information Obtain basic prior information The acquisition of basic prior information The formula is as follows: in, This is a hyperparameter, set to 0.5; The steps of the cross-attention submodule in generating attention-weighted prior information to obtain attention-weighted prior information include: query features Support features Support for masks The first multi-head attention block of the input cross-attention submodule, the query feature The supporting features and the support mask Each serves as the first multi-head attention block matrix, Matrix and Matrix, to obtain coarse attention-weighted prior information The acquisition of coarse attention-weighted prior information The formula is as follows: in, , ; The query features and the coarse attention-weighted prior information The second multi-head attention block of the input cross-attention submodule, the query feature As the second multi-head attention block Matrix, the coarse attention-weighted prior information As the second multi-head attention block Matrix and Matrix, to obtain attention-weighted prior information The acquisition of attention-weighted prior information The formula is as follows: in, Indicates weighted aggregation; Prior information from high-level, mid-level, and low-level query images is input into a multi-scale fusion module to obtain the final segmentation prediction.

2. The few-sample semantic segmentation method that integrates category label semantics according to claim 1, characterized in that, The step of acquiring the query image, support image, support mask, and text category labels, and preprocessing them to obtain multi-layer query features, multi-layer support features, and text features includes: Get the query image Supports images Support for masks and text category labels Then, the text category labels Input the text encoder of the CLIP model to obtain fixed-dimensional text features. ; The query image and supporting images Input the backbone network for feature extraction, and obtain the query features respectively. and supporting features , ,in Indicates the height of the feature map, Indicates the width of the feature map. The query feature represents the number of channels in the feature vector. and the supporting features Including low-level query features Low-level support features Mid-level query features Mid-layer support features High-level query features High-level support characteristics .

3. The few-sample semantic segmentation method integrating category label semantics according to claim 1, characterized in that, The step of inputting the multimodal prior information, the basic prior information, and the attention-weighted prior information into the assembly submodule to obtain the prior information of the query image includes: Multimodal prior information Basic prior information Attention-weighted prior information In the input assembly submodule, the assembly submodule is a residual structure, and the main path of the residual structure will contain the original multimodal prior information. The aforementioned basic prior information and the attention-weighted prior information In the input splicing block, the residual path of the residual structure includes a TE encoder block and a CBAM block. The CBAM block includes a channel attention mechanism block and a spatial attention mechanism block. Both the channel attention mechanism block and the spatial attention mechanism block are residual structures. The multimodal prior information The aforementioned basic prior information and the attention-weighted prior information To perform fusion in order to obtain prior information about fusion. The acquisition of fusion prior information The formula is as follows: The fusion prior information Input into the splicing block; Then the fused prior information Input into the TE encoder block to obtain first prior information. The acquisition of the first prior information The formula is as follows: in, This represents a stacked Transformer encoder that transmits the first prior information. Input into the splicing block; The fused prior information Input into the CBAM block to obtain the second prior information. The acquisition of the second prior information The formula is as follows: in, This indicates a CBAM block, which includes the second prior information. Input into the splicing block; The multimodal prior information The aforementioned basic prior information The attention-weighted prior information The fusion of prior information The first prior information and the second prior information The splicing is performed along the channel dimension, and the splicing formula is as follows: in, This indicates a query for prior information about an image. , This represents the concatenation function.

4. The few-sample semantic segmentation method that integrates category label semantics according to claim 1, characterized in that, The step of inputting prior information from high-level, mid-level, and low-level query images into a multi-scale fusion module to obtain the final segmentation prediction includes: Prior information of query images at high, middle, and low levels is input into a multi-scale fusion module. The multi-scale fusion module includes a dual-channel convolutional block and a final decoder. The dual-channel convolutional block includes a first convolutional path and a second convolutional path. The first convolutional path includes a 3x3 convolutional layer, a batch normalization layer, and an ELU activation layer. The second convolutional path includes a 1x1 convolutional layer, a group normalization layer, and a ReLU activation layer. The final decoder includes a convolutional layer, a batch normalization layer, an ELU activation layer, and a CBAM block. After channel discarding processing, the prior information of the query images in the high-level, mid-level, and low-level layers is input into dual-channel convolutional blocks for feature aggregation. The processed prior information of the query images in the high-level layers is upsampled and then fused with the processed prior information of the query images in the mid-level layers using element-wise addition. The fused prior information of the query images is then input into dual-channel convolutional blocks for feature aggregation. The fused prior information of the query images is then upsampled and then fused with the processed prior information of the query images in the low-level layers using element-wise addition, and then input into dual-channel convolutional blocks for feature aggregation. After upsampling, it is input into the final decoder. The final decoder obtains the final segmentation prediction and performs loss calculation. The formula for loss calculation is as follows: in, This represents the predicted mask of the query image and its corresponding ground truth mask. The Dice loss between them Indicates CE loss, This indicates the total loss.

5. A few-sample semantic segmentation system that integrates category label semantics, characterized in that, include: The feature acquisition module is used to acquire the query image, support image, support mask and text category label and preprocess them to obtain multi-layer query features, multi-layer support features and text features; The prior information generation module is used to input the high-level, mid-level, and low-level features from the multi-layer query features and the multi-layer support features into the prior information generation module for high-level, mid-level, and low-level query image prior information generation processing. The prior information generation module includes a text image adaptation submodule, a prior information generation submodule, a cross-attention submodule, and an assembly submodule. The text image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information. The prior information generation submodule performs a basic prior information generation operation to obtain basic prior information. The cross-attention submodule performs an attention-weighted prior information generation operation to obtain attention-weighted prior information. The multimodal prior information, the basic prior information, and the attention-weighted prior information are input into the assembly submodule to obtain query image prior information. The text-image adaptation submodule performs a multimodal prior information generation operation to obtain multimodal prior information, including the following steps: The fully connected layer in the text-image adaptation submodule supports text features. A linear projection operation is performed, and the formula for the linear projection operation is as follows: in, Indicates a fully connected layer. Text encoder representing the CLIP model, text features , It is the dimension of the embedding space; Convolutional blocks in the text-image adaptation submodule for query features and supporting features Perform channel transformation operation to transform the query features and the supporting features The scaling transformation is The dimension transformation block in the text-image adaptation submodule further transforms the query features after channel transformation. and the supporting features A dimensional transformation operation is performed, and the formula for the dimensional transformation operation is as follows: in, This represents the query features after dimensional transformation. This represents the supporting features after dimensionality transformation. , This represents the dimension transformation function. Represents the convolution function; The text features The query features and the supporting features A multimodal fusion operation is performed, and the formula for the multimodal fusion operation is as follows: in, This represents the query features after multimodal fusion. This represents the support features after multimodal fusion. Representing text features The transpose of the matrix; Support for mask A bilinear interpolation downsampling operation is performed, and the formula for the bilinear interpolation downsampling operation is as follows: in, This represents the support mask after downsampling. The downsampling function is represented by the support mask. Size and the supporting features same; The support mask Query features and supporting features In the multi-head attention block of the input text image adaptation submodule, the support mask... Query features and supporting features Each as a multi-head attention block matrix, Matrix and Matrix, to obtain multimodal prior information The acquisition of multimodal prior information The formula is as follows: in, This represents a feedforward neural network layer. Presentation layer normalization layer, This represents a multi-head attention layer, and the mechanism function of the multi-head attention layer is as follows: in, Let Q be the dimension of the Q matrix and K matrix. express The transpose of a matrix; The prior information generation submodule performs basic prior information generation operations to obtain basic prior information, including the following steps: Based on query characteristics and text features Generate the first rough basic prior information The generation of the first rough basic prior information The formula is as follows: in, Represents the cosine similarity function. ; For the first rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows: in, for , This indicates retrieving the maximum value. This indicates obtaining the minimum value. ; Based on the downsampled support mask Query features and supporting features Generate second coarse basic prior information The generation of the second coarse basic prior information The formula is as follows: in, , Indicate query characteristics The transpose of the matrix, express Norm, This indicates element-wise multiplication; For the second rough foundational prior information The deviations are standardized, and the formula for deviation standardization is as follows: in, ; Based on the first rough foundation prior information and the second rough foundational prior information Obtain basic prior information The acquisition of basic prior information The formula is as follows: in, This is a hyperparameter, set to 0.5; The steps of the cross-attention submodule in generating attention-weighted prior information to obtain attention-weighted prior information include: query features Support features Support for masks The first multi-head attention block of the input cross-attention submodule, the query feature The supporting features and the support mask Each serves as the first multi-head attention block matrix, Matrix and Matrix, to obtain coarse attention-weighted prior information The acquisition of coarse attention-weighted prior information The formula is as follows: in, , ; The query features and the coarse attention-weighted prior information The second multi-head attention block of the input cross-attention submodule, the query feature As the second multi-head attention block Matrix, the coarse attention-weighted prior information As the second multi-head attention block Matrix and Matrix, to obtain attention-weighted prior information The acquisition of attention-weighted prior information The formula is as follows: in, Indicates weighted aggregation; The multi-scale fusion module is used to input prior information of high-level, mid-level, and low-level query images into the multi-scale fusion module to obtain the final segmentation prediction.

6. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the few-sample semantic segmentation method that integrates category label semantics as described in any one of claims 1-4.

7. A computer device, characterized in that, The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the few-sample semantic segmentation method that integrates category label semantics as described in any one of claims 1-4.

Images