A few-labeled remote sensing image semantic segmentation method based on visual text guidance

Abstract

The application relates to a few-labeled remote sensing image semantic segmentation method and device based on visual text guidance, comprising the following steps: processing image data by using a visual feature encoder of a visual text model to obtain visual coding features; processing support image coding features and query image coding features by using a visual text prior decoupling model to obtain support image visual text priors and query image visual text priors; and obtaining mixed multi-level support image coding features by using a high-confidence visual feature model; obtaining visual relation prior values by using a multi-level prior calculation model; and decoding the priors by using a multi-level prior decoding network to obtain a segmentation result of the input image. In the application, visual text priors are introduced in remote sensing image small sample semantic segmentation, and the universality of the visual text model is used to solve the problem of large intra-class differences of remote sensing images in small sample semantic segmentation.

Description

Technical Field

[0001] The embodiments of this application relate to the field of computer vision, and in particular to a method and apparatus for semantic segmentation of remote sensing images with few annotations based on visual text guidance. Background Technology

[0002] Few-sample semantic segmentation of remote sensing images aims to achieve pixel-level differentiation of a category using a small number of labeled images. However, due to the significant differences between remote sensing targets within the same category, a small amount of labeled data cannot cover all semantic information within the category, resulting in poor segmentation results. This application's semantic segmentation method using a small number of labeled samples aims to extract generalizable visual text priors using a visual text model to alleviate the intra-class difference problem. To further improve the robustness of the guidance, a high-confidence feature fusion method is proposed, which mixes high-confidence query features with corresponding hierarchical support image encoding features and uses Euclidean distance normalization to reduce cumulative gain, thereby improving the performance of parametric visual feature metrics. This technology has important applications in defense technology, urban planning, and natural disaster prevention and control.

[0003] Existing semantic segmentation techniques for a small number of labeled images can be divided into prototype-based methods and affinity-based methods. While these methods improve the performance of semantic segmentation for small samples to some extent, their effectiveness in alleviating the problem of class discrepancies remains limited. Summary of the Invention

[0004] In view of this, embodiments of this application propose a method and apparatus for semantic segmentation of remote sensing images with few annotations based on visual text guidance, aiming to solve the problem of the impact of large intra-class differences on segmentation performance in small sample semantic segmentation.

[0005] To achieve the above objectives, embodiments of this application provide a semantic segmentation method for minimally labeled remote sensing images based on visual text guidance, comprising: inputting pre-acquired image data and text data into a semantic segmentation model, wherein the image data includes a support image, support image labels, and a query image, and the text data includes the name of the category to be segmented and the name of the background category; the semantic segmentation model includes a visual text model and a ResNet network; processing the image data using the visual feature encoder of the visual text model to obtain visual encoding features, wherein the visual encoding features include support image encoding features and query image encoding features, and the support image encoding features include foreground features; processing the support image encoding features and query image encoding features respectively using a pre-constructed visual text prior decoupling model to obtain the support image visual text prior and query image visual text prior regarding the category to be segmented; and using a pre-constructed high-resolution model... A hybrid model of visual features is used to process the visual text prior of the query image, obtaining high-confidence features in the visual features of the query image. Based on the high-confidence features and multi-level query image coding features, the query prototype is determined. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the hybrid multi-level support image coding features. The multi-level support image coding features and multi-level query image coding features are obtained by processing the query image and support images using the ResNet network. A preset multi-level prior calculation model is used to process the visual features of the query image and the hybrid multi-level support image coding features to obtain multi-level cosine affinity priors and multi-level Euclidean distance normalized loss cumulative gain priors. A preset multi-level prior decoding network is used to decode the visual text priors, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors to obtain the segmentation result of the query image.

[0006] Optionally, the background class name in the text data is processed using the visual text model to obtain the background class text encoding features. The feature is that the pre-constructed visual text prior decoupling model is used to process the supporting image encoding features and the query image encoding features respectively, correspondingly obtaining the supporting image visual text prior and the query image visual text prior regarding the category to be segmented. This includes: calculating the supporting image visual text similarity and the query image text similarity based on the cosine similarity between the supporting image encoding features and the query image encoding features and the background class text encoding features; and then... A visual text prior decoupling model is used to input the encoding features of the supporting image and the encoding features of the query image, thereby obtaining the visual text prior of the supporting image and the visual text prior of the query image. The visual text prior decoupling model includes: a first 2D transposed convolutional layer, a first normalization layer, a first activation layer, a second 2D transposed convolutional layer, a second normalization layer, a second activation layer, a first 2D convolutional layer, a third normalization layer, a third activation layer, a first max pooling layer, a second 2D convolutional layer, a fourth normalization layer, a second max pooling layer, a third 2D convolutional layer, and a sigmoid activation function layer, connected in sequence.

[0007] Optionally, the method further includes: training the visual-text prior decoupling model using the binary cross-entropy loss between the visual-text prior of the supporting image and the label of the supporting image, to obtain the final visual-text prior decoupling model.

[0008] Optionally, the step of processing the visual text prior of the query image using a pre-constructed high-confidence visual feature fusion model to determine high-confidence features in the visual features of the query image, and determining the query prototype based on the high-confidence features and multi-level query image coding features, and then weighting and summing the query prototype and the foreground features in the multi-level support image coding features to obtain the fused multi-level support image coding features, includes: inputting the support image and the query image into the ResNet network respectively to obtain the corresponding multi-level support image coding features and multi-level query image coding features; aligning the multi-level support image coding features and multi-level query image coding features with the spatial resolution of the input image to obtain the aligned multi-level query image coding features. The query image coding features and aligned multi-level support image coding features are used. For each level of the aligned multi-level query image coding features and aligned multi-level support image coding features, the following operations are performed to obtain hybrid multi-level support image coding features: The query image visual text prior is normalized to obtain a normalized prior; a threshold is calculated on the normalized prior to obtain a confidence level, setting confidence levels above a preset threshold to 1 and confidence levels below or equal to the preset threshold to 0; a query prototype is obtained based on the confidence level and the corresponding level of query image coding features; and the hybrid corresponding level support image coding features are determined based on the support image labels, the query prototype, and the corresponding level support image coding features.

[0009] Optionally, determining the hybrid hierarchical support image coding features based on query image labels, query prototypes, and support image coding features includes: finding the target class region of the support image coding features of the corresponding level according to the support image labels, and performing a weighted hybrid mixing of the target class region and the query prototype to obtain the hybrid support image coding features of that level.

[0010] Optionally, before processing the visual features of the query image and the hybrid multi-level support image coding features using a preset multi-level prior calculation model to obtain the visual relationship prior value, the method includes: interpolating and flattening the support image labels to obtain corrected support image labels with spatial resolution consistent with the visual features; performing a Haldman product operation on the hybrid multi-level support image coding features and the corrected support image labels respectively to obtain the mask features of the corresponding level; calculating the cosine similarity between the mask features and the query image coding features of each level respectively, and then performing mask mean calculation and dimension transformation on the cosine similarity along the dimension of the corrected support image labels to obtain the cosine affinity prior of each level.

[0011] Optionally, before processing the query image visual features and the mixed multi-level support image coding features using a preset multi-level prior calculation model to obtain the visual relationship prior value, the method further includes: calculating the pixel-by-pixel Euclidean distance between the mixed multi-level support image coding features and the query image coding features respectively to obtain the Euclidean distance at each level; calculating the number of foreground pixels of the corrected support image label; determining the coordinate subset of the mixed multi-level support image coding features that has the closest Euclidean distance at each level to the query image coding features, wherein the number of coordinate subsets is equal to the number of foreground pixels; performing a relevance score on the coordinate subset, and calculating the loss cumulative gain and the ideal loss cumulative gain for the query image visual features as the target class based on the relevance score; obtaining the multi-level Euclidean distance normalized loss cumulative gain prior based on the loss cumulative gain and the ideal loss cumulative gain for the query image visual features as the target class.

[0012] Optionally, the preset multi-level prior decoding network includes 2D convolutional layers, group normalization layers, and... The function layer utilizes a pre-defined multi-level prior decoding network to decode the visual text prior, multi-level cosine affinity prior, and multi-level Euclidean distance normalized loss cumulative gain prior to obtain the segmentation result of the query image. This includes: using 2D convolutional layers, group normalization layers, and... The function layer upsamples and performs cross-layer connections on visual text priors at different levels, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors to obtain the corresponding dual-channel query image prediction probability map.

[0013] Optionally, the method further includes: calculating the scale-aware cross-entropy segmentation loss by calculating the predicted probability map of the query image and the corrected support image labels; obtaining the total loss function based on the sum of the binary cross-entropy loss and the scale-aware cross-entropy segmentation loss; and training the ResNet network and the visual text model using the total loss function to obtain a semantic segmentation model with determined network parameters.

[0014] To achieve the above objectives, embodiments of this application also provide a semantic segmentation device for remote sensing images with few annotations based on visual text guidance, comprising: an input module for inputting pre-acquired image data and text data into a semantic segmentation model, wherein the image data includes a support image, support image labels, and a query image, and the text data includes the name of the category to be segmented and the name of the background category; the semantic segmentation model includes a visual text model and a ResNet network; an encoding module for processing the image data using a visual feature encoder of the visual text model to obtain visual encoding features, wherein the visual encoding features include support image encoding features and query image encoding features, and the support image encoding features include foreground features; a first prior calculation module for processing the support image encoding features and query image encoding features respectively using a pre-constructed visual text prior decoupling model, correspondingly obtaining the support image visual text prior and query image visual text prior regarding the category to be segmented; and a feature mixing module for... A pre-constructed high-confidence visual feature fusion model is used to process the visual text prior of the query image, obtaining high-confidence features in the visual features of the query image. Based on the high-confidence features and multi-level query image coding features, a query prototype is determined. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features. The multi-level support image coding features and multi-level query image coding features are obtained by processing the query image and support images using a ResNet network. A second prior calculation module is used to process the visual features of the query image and the fused multi-level support image coding features using a pre-defined multi-level prior calculation model, obtaining multi-level cosine affinity priors and multi-level Euclidean distance normalized loss cumulative gain priors. An image segmentation module is used to decode the visual text priors, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors using a pre-defined multi-level prior decoding network to obtain the segmentation result of the query image.

[0015] The embodiments of this application propose a method and apparatus for semantic segmentation of remote sensing images with few annotations based on visual text guidance, comprising: using pre-acquired image data and text data as input to a semantic segmentation model, wherein the image data includes a support image, support image labels, and a query image, and the text data includes the name of the category to be segmented and the name of the background category; the semantic segmentation model includes a visual text model and a ResNet network; processing the image data using a visual feature encoder of the visual text model to obtain visual encoded features, wherein the visual encoded features include support image encoded features and query image encoded features, the support image encoded features being... The features include foreground features; a pre-built visual-text prior decoupling model is used to process the support image encoding features and query image encoding features respectively, resulting in support image visual-text priors and query image visual-text priors for the categories to be segmented; a pre-built high-confidence visual feature fusion model is used to process the query image visual-text prior, obtaining high-confidence features in the query image visual features, and the query prototype is determined based on the high-confidence features and multi-level query image encoding features. Then, the query prototype and the foreground features in the multi-level support image encoding features are weighted and summed to obtain the fused multi-level support image. The encoding features, including multi-level support image encoding features and multi-level query image encoding features, are obtained by processing query images and support images using a ResNet network. A pre-defined multi-level prior calculation model is used to process the visual features of the query image and the mixed multi-level support image encoding features to obtain multi-level cosine affinity priors and multi-level Euclidean distance normalized loss cumulative gain priors. A pre-defined multi-level prior decoding network is used to decode the visual text priors, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors to obtain the segmentation result of the query image. This application is applicable to small samples of remote sensing images. This paper introduces visual text priors into semantic segmentation, leveraging the universality of visual text models to alleviate the problem of large intra-class differences in remote sensing images during small-sample semantic segmentation, thereby improving segmentation performance. Furthermore, the visual text priors designed in this application overcome the deficiency of visual text models in lacking target localization capabilities. The high-confidence visual feature fusion model mixes the high-confidence portion of the query features with the corresponding hierarchical support image coding features, improving the robustness of the measurement process of the corresponding hierarchical support image coding features and query features. Compared with existing remote sensing small-sample semantic segmentation methods, this application has higher segmentation accuracy and high utilization value in local security tasks. Attached Figure Description

[0016] Figure 1 This is a flowchart of a visual text-guided semantic segmentation method for remote sensing images with few annotations, provided in one embodiment of this application. Figure 1 ;

[0017] Figure 2This is a flowchart of a visual text-guided semantic segmentation method for remote sensing images with few annotations, provided in one embodiment of this application. Figure 2 ;

[0018] Figure 3 This is a structural block diagram of a remote sensing image semantic segmentation device based on visual text guidance with minimal annotation, provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this application to enable readers to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0020] One embodiment of this application proposes a visual text-guided semantic segmentation method for minimally labeled remote sensing images, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for illustration. The implementation details of the visual text-guided semantic segmentation method for minimally labeled remote sensing images proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0021] It should be noted first that this application designs a visual-text prior decoupling module, which utilizes the visual feature encoder of a contrastive language-image pre-training (CLIP) model. and text feature encoder To uncover potential visual text priors with spatial positioning capabilities.

[0022] The specific process of the visual text-guided, minimally labeled remote sensing image semantic segmentation method proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0023] S101, Input pre-acquired image data and text data into the semantic segmentation model, wherein the image data includes supporting images, supporting image labels and query images, the text data includes the category name to be segmented and the background category name, and the semantic segmentation model includes a visual text model and a ResNet network;

[0024] The input to the semantic segmentation model includes supporting images. and its labels (Target area labeled) and query image (Image to be segmented), and the category name to be segmented (e.g., "buildings") and a set of background class names used for prompts .

[0025] S102, the image data is processed using the visual feature encoder of the visual text model to obtain visual coding features, wherein the visual coding features include supporting image coding features and query image coding features, and the supporting image coding features include foreground features;

[0026] The output of the visual text model includes pixel-level visual encoded features. and instance-level text encoding features Encode background text features The sequence is obfuscated to avoid introducing semantically relevant information about the background class that could lead to overfitting.

[0027] S103 uses a pre-built visual-text prior decoupling model to process the coding features of the supporting image and the coding features of the query image respectively, and obtains the visual-text priors of the supporting image and the query image with respect to the category to be segmented.

[0028] The purpose of this step is to mine the visual-text correlation of the target class and generate a pixel-level prior map.

[0029] refer to Figure 1 In the embodiments of this application, the background class name in the text data is processed using the visual text model to obtain the background class text encoding features. S103 may include the following execution process:

[0030] S1031, calculate the visual text similarity of the supporting image and the text similarity of the query image based on the cosine similarity between the supporting image coding features and the query image coding features and the background text coding features, respectively;

[0031] S1032, Input the supporting image coding features and the query image coding features into the visual text prior decoupling model to obtain the supporting image visual text prior and the query image visual text prior.

[0032] The visual-text prior decoupling model includes: a first 2D transposed convolutional layer, a first normalization layer, a first activation layer, a second 2D transposed convolutional layer, a second normalization layer, a second activation layer, a first 2D convolutional layer, a third normalization layer, a third activation layer, a first max pooling layer, a second 2D convolutional layer, a fourth normalization layer, a second max pooling layer, a third 2D convolutional layer, and a Sigmoid activation function layer, connected in sequence.

[0033] In embodiments of this application, the method further includes: training the visual text prior decoupling model using the binary cross-entropy loss between the visual text prior of the supporting image and the supporting image label, to obtain the final visual text prior decoupling model.

[0034] To ensure Higher values ​​reflect high-confidence classification as the target class; otherwise, it is considered background. Using supporting image labels as constraints, the binary cross-entropy loss between the supporting visual text prior and the labels is calculated:

[0035] (3)

[0036] in This is the binary cross-entropy loss.

[0037] Specifically, the processor generates cue text (such as "buildings in satellite image") for category name C, which is encoded as text features. T C For the background class collection { B 1 ,...,B n Randomly shuffle the text and then concatenate the pieces to generate obfuscated background text features. T B Features of splicing target and background text: T p =[ T C ；T B ], to obtain the features of the prompt text .

[0038] First, the processor calculates visual text similarity, which can be: The specific calculation formula is as follows:

[0039] (1)

[0040] in The transformation function representing the feature , for , This is the cosine similarity metric function. Fvision This includes multi-level support image coding features and multi-level query image coding features. These features are obtained by processing query and support images using the ResNet network.

[0041] After obtaining the visual text similarity, the processor needs to calculate from This application extracts prior visual text information about the target class to be segmented. A prior decoupling network is designed to perform prior decoupling, specifically consisting of an upsampling part and a downsampling part. Visual text similarity is then calculated. After mapping, the visual text prior of the target class is obtained. The specific formula is as follows:

[0042] (2)

[0043] in, , , These represent 2D transposed convolution, 2D convolution, and max pooling, respectively. For instance normalization, It is the ReLU activation function. Use the Sigmoid activation function to ensure that the output value is within... Within the scope, the visual text priors for the target class include the supporting image visual text priors and the query image visual text priors.

[0044] For example, the upsampling path can be ConvTranspose2D→InstanceNorm→ReLU→ConvTranspose2D→InstanceNorm→ReLU; the downsampling path can be Conv2D→InstanceNorm→ReLU→MaxPool→Conv2D→InstanceNorm→MaxPool→Conv2D; and the output layer can be Sigmoid activated.

[0045] S104 uses a pre-constructed high-confidence visual feature fusion model to process the visual text prior of the query image, obtains high-confidence features in the visual features of the query image, and determines the query prototype based on the high-confidence features and multi-level query image coding features. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features. The multi-level support image coding features and multi-level query image coding features are obtained by processing the query image and support image based on the ResNet network.

[0046] The purpose of this step is to enhance the robustness of the corresponding layer's supporting image coding features and alleviate intra-class differences.

[0047] This step may include the following execution process:

[0048] S1041, Input the support image and query image into the ResNet network respectively to obtain the multi-level support image coding features and multi-level query image coding features;

[0049] S1042, Align the multi-level support image coding features and the multi-level query image coding features with the spatial resolution of the input image to obtain the aligned multi-level query image coding features and the aligned multi-level support image coding features.

[0050] S1043, Perform the following operation on each level of the aligned multi-level query image coding features and the aligned multi-level support image coding features to obtain the hybrid multi-level support image coding features:

[0051] S1044, Normalize the visual text prior of the query image to obtain the normalized prior;

[0052] S1045, perform threshold calculation on the normalized prior to obtain the confidence level, set the confidence level above the preset threshold to 1, and set the confidence level below or equal to the preset threshold to 0.

[0053] S1046. Based on the confidence level and the corresponding level of query image encoding features, the query prototype is obtained.

[0054] S1047, Based on the supporting image label, query prototype and corresponding level supporting image coding features, determine the mixed corresponding level supporting image coding features.

[0055] Specifically, the high-confidence visual feature fusion model in this application utilizes a weighted fusion of visual features from the high-confidence prior of the query image visual text with the query features to improve the robustness of the subsequent visual affinity prior. The module's input is visual features. Query visual text prior with spatial resolution consistent with visual features after interpolation and supporting image tags .

[0056] First of all Perform max-min normalization operation:

[0057]

[0058] Thresholds are calculated based on the normalized priors. Quantities above the threshold are considered to have high confidence and are set to 1; otherwise, they are set to 0.

[0059]

[0060] here As the confidence threshold, this application sets it to 0.7, according to... and Calculate the query prototype

[0061] (6)

[0062] in This is for mask average pooling operation.

[0063] In the embodiments of this application, the target class region of the corresponding level of support image coding features is found according to the support image label, and the target class region and the query prototype are weighted and mixed to obtain the hybrid support image coding features of that level.

[0064] Specifically, the processor is based on the supporting image tags. Find the target class region of the corresponding hierarchical support image coding features, and perform a weighted mixture of the features of this region and the query prototype to obtain the final multi-level support image coding features:

[0065] (7)

[0066] As a mixed weight, this application sets it to 0.5.

[0067] S105, using a pre-defined multi-level prior calculation model to process the visual features of the query image and the hybrid multi-level support image coding features, to obtain the multi-level cosine affinity prior and the multi-level Euclidean distance normalized loss cumulative gain prior.

[0068] In embodiments of this application, prior to step S105, the method further includes:

[0069] S1051, After interpolating and flattening the support image label, a corrected support image label with spatial resolution consistent with visual features is obtained.

[0070] S1052, perform a Hadman product operation on the hybrid multi-level support image coding features and the corrected support image labels respectively to obtain the mask features of the corresponding level.

[0071] S1053, calculate the cosine similarity of the mask features and the query image coding features at each level respectively, and then perform mask mean calculation and dimension transformation on the cosine similarity along the dimension of the corrected support image label to obtain the cosine affinity prior at each level.

[0072] Specifically, in addition to visual text priors, this application utilizes cosine affinity priors based on visual relationships and Euclidean distance normalized loss cumulative gain priors, with the input being flattened visual features. After interpolating and flattening the supporting image labels, we obtain corrected supporting image labels with spatial resolution consistent with visual features. .

[0073] For the cosine affinity prior, a Hadman product operation is first performed on the final multi-level support image encoded features and the corrected support image labels to obtain the mask features:

[0074] (8)

[0075] in Represent the Hadman product, and then calculate their cosine similarity:

[0076] (9)

[0077] Then to The cosine affinity prior is obtained by calculating the mask mean along the support dimension and performing a dimension transformation. .

[0078] (10)

[0079] In the embodiments of this application, before processing the visual features of the query image and the mixed multi-level support image coding features using a preset multi-level prior calculation model to obtain the visual relationship prior value, the method further includes:

[0080] Calculate the pixel-wise Euclidean distance between the hybrid multi-level support image coding features and the query image coding features to obtain the Euclidean distance for each level.

[0081] Calculate the number of foreground pixels for the corrected support image label;

[0082] Determine the coordinate subset of the mixed multi-level supporting image coding features that has the closest Euclidean distance to each level of the query image coding features, where the number of coordinate subsets is equal to the number of foreground pixels;

[0083] The relevance scores are applied to the coordinate subsets, and the cumulative loss gain and the ideal cumulative loss gain for the visual features of the query image as the target class are calculated based on the relevance scores.

[0084] Based on the loss cumulative gain and the ideal loss cumulative gain of the query image visual features as the target class, a multi-level Euclidean distance normalized loss cumulative gain prior is obtained.

[0085] Specifically, in addition to visual text priors, this application utilizes cosine affinity priors based on visual relationships and Euclidean distance normalized loss cumulative gain priors, with the input being flattened visual features. After interpolating and flattening the supporting image labels, we obtain corrected supporting image labels with spatial resolution consistent with visual features. .

[0086] For the Euclidean distance-normalized loss cumulative gain prior, the pixel-wise Euclidean distance between the support features and the query features is first calculated:

[0087] (11)

[0088] Then calculate to get Number of foreground pixels:

[0089] (12)

[0090] for Get the nearest neighbor. Coordinates of each supporting feature:

[0091] (13)

[0092] The function is to find the previous The coordinates of the largest element, for Take the negative value to find the nearest element. Then, obtain a relevance score based on whether it falls within the foreground. If the value falls within the foreground, the relevance score is 1; otherwise, it is zero. Considering the influence of ranking position, calculate the cumulative gain of the loss:

[0093]

[0094] The ideal cumulative gain for the target class is:

[0095] (15)

[0096] Representative before The nearest matching features all fall within the foreground position. Therefore, the normalized cumulative loss gain is calculated:

[0097] (16)

[0098] The closer the value is to 1, the stronger the characteristic. The more likely it is to be the target class. This is obtained based on the algorithm. Euclidean distance normalized loss cumulative gain prior .

[0099] S106, the visual text prior, multi-level cosine affinity prior, and multi-level Euclidean distance normalized loss cumulative gain prior are decoded using a preset multi-level prior decoding network to obtain the segmentation result of the query image.

[0100] In the embodiments of this application, the preset multi-level prior decoding network includes 2D convolutional layers, group normalization layers, and function layers. Step S106 may include the following execution process:

[0101] S1061 utilizes 2D convolutional layers, group normalization layers, and function layers to upsample and connect different levels of visual text priors, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors to obtain the corresponding dual-channel query image prediction probability map.

[0102] In embodiments of this application, the method further includes:

[0103] Calculate the scale-aware cross-entropy segmentation loss by calculating the predicted probability map of the query image and the corrected support image labels;

[0104] The total loss function is obtained by summing the binary cross-entropy loss and the scale-aware cross-entropy segmentation loss.

[0105] The ResNet network and the visual text model are trained using the overall loss function to obtain a semantic segmentation model with determined network parameters.

[0106] In this embodiment, the multi-level prior decoder consists of 2D convolution, group normalization, and... The function is constructed by upsampling and cross-layer connections of priors at different levels, ultimately outputting a dual-channel query image prediction probability map. Combining probability maps with image labels Compute the scale-aware cross-entropy segmentation loss:

[0107]

[0108] Therefore, the final total loss function is:

[0109]

[0110] By backpropagating the loss function, the network parameters can be optimized.

[0111] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0112] Compared with other few-shot semantic segmentation techniques, this application has the following advantages:

[0113] 1. The method in this application introduces visual text priors into the semantic segmentation of remote sensing images with few samples. By leveraging the universality of visual text models, it alleviates the problem of large intra-class differences in remote sensing images during semantic segmentation with few samples, thereby improving segmentation performance. Furthermore, the visual text priors designed in this application overcome the deficiency of visual text models in lacking target localization capabilities.

[0114] 2. Enhanced algorithm robustness. The high-confidence visual feature fusion module designed in this application blends the high-confidence portion of the query features with the supporting features, thereby improving the robustness of the measurement process for both supporting and query features.

[0115] 3. Visual guidance is utilized more fully. The visual relationship prior designed in this application not only utilizes the cosine similarity between features, but also designs a Euclidean distance normalized loss cumulative gain prior, which improves the utilization of visual feature relationships.

[0116] 4. Higher segmentation accuracy. Compared with existing remote sensing small-sample semantic segmentation methods, this application has higher segmentation accuracy and is of high value in local security tasks.

[0117] Reference Figure 2 Another embodiment of this application provides a method for semantic segmentation of remote sensing images with few annotations based on visual text guidance, and the implementation steps are as follows:

[0118] Step 1, Feature extraction process.

[0119] First, the input consists of a supporting image, a query image, and prompt text. Feature encoding is then performed on both the image and the prompt text. For the image, the image encoder includes the visual encoder part of the visual text model and a multi-level visual backbone part, ultimately yielding one CLIP visual feature and a set of multi-level visual features for each image. The prompt text is encoded by the text encoder part of the visual text model to obtain CLIP text features.

[0120] Step 2, visual-text prior decoupling process.

[0121] The CLIP visual features of the support image and query image, and the CLIP textual features of the prompt text, are input into the visual-text prior decoupling module. First, the visual-text similarity between the support image and query image is calculated separately. Then, the prior decoupling network is used to decouple the visual-text similarity between the two images, obtaining the visual-text priors of the support image and query image regarding the category to be segmented. For the visual-text prior of the support image, a binary cross-entropy calculation is performed between it and the ground truth label of the support image to obtain the visual-text prior decoupling loss.

[0122] Step 3, high-confidence visual feature mixing process.

[0123] Based on the prior visual text of the query image, the high-confidence portion of the query visual features is identified. A query prototype is then calculated using masked mean pooling. Finally, the query prototype and the foreground portion of the supporting visual features are weighted and summed to obtain the hybrid supporting visual features.

[0124] Step 4, the visual relationship prior calculation process.

[0125] Visual relationship priors are calculated using query visual features and the blended visual text features. These include a cosine affinity prior based on cosine similarity and a Euclidean distance-normalized loss cumulative gain prior.

[0126] Step 5, multi-level prior decoding process.

[0127] The visual text prior, multi-level cosine affinity prior, and Euclidean distance normalized loss cumulative gain prior are decoded, upsampled, and connected across layers to obtain the final segmentation result. The segmentation loss is calculated by combining the loss with the real label, and the overall loss is obtained by summing the loss with the decoupled loss of the visual text prior, thereby optimizing the model.

[0128] This application also provides simulation conditions and simulation content:

[0129] This application describes a simulation performed using PyTorch on an Intel(R) Xeon(R) Silver4210R CPU @ 2.40GHz, with 128GB of memory and a Linux operating system. The data used in the simulation is a publicly available dataset.

[0130] The simulation data is the iSAID-5i remote sensing image dataset, which contains 15 categories and images with a size of 256×256 pixels. Ten classes are used for training, and the remaining five classes are used for validation, including three partitioning methods: fold0, fold1, and fold2. Simulations were performed under two different conditions: 1-shot and 5-shot (using one and five support images respectively) to verify the superior performance of this application. The visual backbone is a ResNet50 pre-trained on ImageNet, and the visual text model is RemoteCLIP-ViT-L-14.

[0131] To demonstrate the effectiveness of our method, we compared it with other state-of-the-art few-shot semantic segmentation methods. The evaluation metric was mIoU, and the comparison results are shown in Table 1.

[0132] Table 1

[0133]

[0134] As can be seen from Table 1, this application achieves better results in the mIoU index compared to other methods.

[0135] refer to Figure 3 Based on the above embodiments, this application also proposes a semantic segmentation device for remote sensing images with few annotations based on visual text guidance. The remote sensing image semantic segmentation device 1000 includes:

[0136] The input module 1001 is used to input pre-acquired image data and text data into the semantic segmentation model. The image data includes supporting images, supporting image labels, and query images. The text data includes the name of the category to be segmented and the name of the background category. The semantic segmentation model includes a visual text model and a ResNet network.

[0137] The encoding module 1002 is used to process image data using a visual feature encoder of a visual text model to obtain visual encoding features, wherein the visual encoding features include supporting image encoding features and query image encoding features, and the supporting image encoding features include foreground features.

[0138] The first prior calculation module 1003 is used to process the supporting image coding features and the query image coding features respectively using a pre-built visual text prior decoupling model, and obtain the supporting image visual text prior and the query image visual text prior with respect to the category to be segmented.

[0139] The feature fusion module 1004 is used to process the visual text prior of the query image using a pre-built high-confidence visual feature fusion model to obtain high-confidence features in the visual features of the query image, and to determine the query prototype based on the high-confidence features and multi-level query image coding features. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features. The multi-level support image coding features and the multi-level query image coding features are obtained by processing the query image and support image based on the ResNet network.

[0140] The second prior calculation module 1005 is used to process the visual features of the query image and the mixed multi-level support image coding features using a preset multi-level prior calculation model to obtain the multi-level cosine affinity prior and the multi-level Euclidean distance normalized loss cumulative gain prior.

[0141] The image segmentation module 1006 is used to decode the visual text prior, the multi-level cosine affinity prior, and the multi-level Euclidean distance normalized loss cumulative gain prior using a preset multi-level prior decoding network to obtain the segmentation result of the query image.

[0142] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0143] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent from this embodiment.

[0144] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A semantic segmentation method for remote sensing images with few annotations based on visual text guidance, characterized in that, include: The semantic segmentation model is input with pre-acquired image data and text data. The image data includes supporting images, supporting image labels, and query images. The text data includes the category name to be segmented and the background category name. The semantic segmentation model includes a visual text model and a ResNet network. Image data is processed using a visual feature encoder of a visual text model to obtain visual coding features, wherein the visual coding features include supporting image coding features and query image coding features, and the supporting image coding features include foreground features; The pre-built visual-text prior decoupling model is used to process the coding features of the support image and the coding features of the query image respectively, thereby obtaining the visual-text priors of the support image and the query image with respect to the category to be segmented. A pre-constructed high-confidence visual feature fusion model is used to process the visual text prior of the query image to obtain high-confidence features in the visual features of the query image. The query prototype is determined based on the high-confidence features and the multi-level query image coding features. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features. The multi-level support image coding features and the multi-level query image coding features are obtained by processing the query image and support image based on the ResNet network. The visual features of the query image and the hybrid multi-level support image coding features are processed using a pre-defined multi-level prior calculation model to obtain multi-level cosine affinity prior and multi-level Euclidean distance normalized loss cumulative gain prior. The visual text prior, multi-level cosine affinity prior, and multi-level Euclidean distance normalized loss cumulative gain prior are decoded using a pre-defined multi-level prior decoding network to obtain the segmentation result of the input image.

2. The semantic segmentation method for remote sensing images with few annotations based on visual text guidance according to claim 1, characterized in that, The method utilizes the visual text model to process background class names in text data, obtaining background class text encoding features. The method is characterized by using a pre-constructed visual text prior decoupling model to process the supporting image encoding features and the query image encoding features respectively, thereby obtaining the supporting image visual text prior and the query image visual text prior regarding the category to be segmented, including: The visual text similarity of the supporting image and the text similarity of the query image are calculated based on the cosine similarity between the supporting image encoding features, the query image encoding features and the background text encoding features, respectively. By inputting the supporting image coding features and the query image coding features into the visual text prior decoupling model, the corresponding visual text priors for the supporting image and the visual text prior for the query image are obtained. The visual-text prior decoupling model includes: a first 2D transposed convolutional layer, a first normalization layer, a first activation layer, a second 2D transposed convolutional layer, a second normalization layer, a second activation layer, a first 2D convolutional layer, a third normalization layer, a third activation layer, a first max pooling layer, a second 2D convolutional layer, a fourth normalization layer, a second max pooling layer, a third 2D convolutional layer, and a Sigmoid activation function layer, connected in sequence.

3. The semantic segmentation method for minimally labeled remote sensing images based on visual text guidance according to claim 2, characterized in that, The method further includes: The visual-text prior decoupling model is trained using the binary cross-entropy loss between the visual-text prior of the supporting image and the supporting image label, resulting in the final visual-text prior decoupling model.

4. The semantic segmentation method for minimally labeled remote sensing images based on visual text guidance according to claim 1, characterized in that, The process utilizes a pre-constructed high-confidence visual feature fusion model to process the visual text prior of the query image, determines the high-confidence features in the query image visual features, and determines the query prototype based on the high-confidence features and multi-level query image coding features. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features, including: The support image and the query image are input into the ResNet network respectively to obtain the multi-level support image coding features and the multi-level query image coding features. Align the multi-level support image coding features and the multi-level query image coding features with the spatial resolution of the input image to obtain the aligned multi-level query image coding features and the aligned multi-level support image coding features. Perform the following operation on each level of the aligned multi-level query image coding features and the aligned multi-level support image coding features to obtain the hybrid multi-level support image coding features: The visual text prior of the query image is normalized to obtain the normalized prior; A threshold is calculated on the normalized prior to obtain the confidence level. The confidence level above the preset threshold is set to 1, and the confidence level below or equal to the preset threshold is set to 0. Based on the confidence level and the corresponding level of query image encoding features, the query prototype is obtained; The hybrid corresponding layer support image coding features are determined based on the support image labels, query prototypes, and corresponding layer support image coding features.

5. The semantic segmentation method for remote sensing images with few annotations based on visual text guidance according to claim 4, characterized in that, The process of determining the hybrid corresponding-level support image coding features based on support image labels, query prototypes, and corresponding-level support image coding features includes: Based on the supporting image labels, the target class region of the corresponding level of supporting image coding features is found, and the target class region and the query prototype are weighted and mixed to obtain the hybrid supporting image coding features of that level.

6. The semantic segmentation method for remote sensing images with few annotations based on visual text guidance according to claim 4, characterized in that, Before processing the visual features of the query image and the hybrid multi-level support image coding features using a preset multi-level prior calculation model to obtain the multi-level cosine affinity prior and the multi-level Euclidean distance normalized loss cumulative gain prior, the method includes: After interpolating and flattening the support image labels, we obtain corrected support image labels with spatial resolution consistent with visual features. The Haldman product operation is performed on the hybrid multi-level support image coding features and the corrected support image labels respectively to obtain the mask features of the corresponding levels. The cosine similarity between the mask features and the query image coding features at each level is calculated separately. Then, the mask mean and dimension transformation are performed sequentially along the dimensions of the corrected support image labels to obtain the cosine affinity prior at each level.

7. The semantic segmentation method for minimally labeled remote sensing images based on visual text guidance according to claim 6, characterized in that, Before processing the visual features of the query image and the hybrid multi-level support image coding features using a preset multi-level prior calculation model to obtain the multi-level cosine affinity prior and the multi-level Euclidean distance normalized loss cumulative gain prior, the method further includes: Calculate the pixel-wise Euclidean distance between the hybrid multi-level support image coding features and the query image coding features to obtain the Euclidean distance for each level. Calculate the number of foreground pixels for the corrected support image label; Determine the coordinate subset of the mixed multi-level supporting image coding features that has the closest Euclidean distance to each level of the query image coding features, where the number of coordinate subsets is equal to the number of foreground pixels; The relevance scores are applied to the coordinate subsets, and the cumulative loss gain and the ideal cumulative loss gain for the visual features of the query image as the target class are calculated based on the relevance scores. Based on the loss cumulative gain and the ideal loss cumulative gain of the query image visual features as the target class, a multi-level Euclidean distance normalized loss cumulative gain prior is obtained.

8. The semantic segmentation method for minimally labeled remote sensing images based on visual text guidance according to claim 3, characterized in that, The pre-defined multi-level prior decoding network includes 2D convolutional layers, group normalization layers, and... The function layer utilizes a pre-defined multi-level prior decoding network to decode the visual text prior, the multi-level cosine affinity prior, and the multi-level Euclidean distance normalized loss cumulative gain prior to obtain the segmentation result of the input image, including: By using a multi-level prior decoding network, upsampling and cross-level connections are performed on visual text priors at different levels, multi-level cosine affinity priors, and multi-level Euclidean distance normalized loss cumulative gain priors to obtain the corresponding dual-channel query image prediction probability map.

9. The semantic segmentation method for remote sensing images with few annotations based on visual text guidance according to claim 8, characterized in that, The method further includes: Calculate the scale-aware cross-entropy segmentation loss by calculating the predicted probability map of the query image and the corrected support image labels; The total loss function is obtained by summing the binary cross-entropy loss and the scale-aware cross-entropy segmentation loss. The ResNet network and the visual text model are trained using the overall loss function to obtain a semantic segmentation model with determined network parameters.

10. A semantic segmentation device for remote sensing images with few annotations based on visual text guidance, characterized in that, include: The input module is used to input pre-acquired image data and text data into the semantic segmentation model. The image data includes supporting images, supporting image labels, and query images. The text data includes the name of the category to be segmented and the name of the background category. The semantic segmentation model includes a visual text model and a ResNet network. The encoding module is used to process image data using a visual feature encoder of a visual text model to obtain visual encoding features, wherein the visual encoding features include supporting image encoding features and query image encoding features, and the supporting image encoding features include foreground features; The first prior calculation module is used to process the encoding features of the supporting image and the encoding features of the query image respectively using a pre-built visual-text prior decoupling model, and obtain the visual-text prior of the supporting image and the query image with respect to the category to be segmented. The feature fusion module is used to process the visual text prior of the query image using a pre-built high-confidence visual feature fusion model to obtain high-confidence features in the visual features of the query image. Based on the high-confidence features and multi-level query image coding features, the query prototype is determined. Then, the query prototype and the foreground features in the multi-level support image coding features are weighted and summed to obtain the fused multi-level support image coding features. The multi-level support image coding features and multi-level query image coding features are obtained by processing the query image and support image based on the ResNet network. The second prior calculation module is used to process the visual features of the query image and the hybrid multi-level support image coding features using a preset multi-level prior calculation model to obtain the multi-level cosine affinity prior and the multi-level Euclidean distance normalized loss cumulative gain prior. The image segmentation module is used to decode the visual text prior, the multi-level cosine affinity prior, and the multi-level Euclidean distance normalized loss cumulative gain prior using a preset multi-level prior decoding network to obtain the segmentation result of the input image.

Images