Remote sensing image semantic segmentation prediction method and system based on vision-language pre-training model

Abstract

The invention discloses a remote sensing image semantic segmentation prediction method and system based on a vision-language pre-training model. The method comprises the following steps: acquiring a source domain image with a label and a target domain image without a label; adding a pseudo tag to the target domain image without the tag by using a teacher network; performing style conversion on the labeled source domain image to obtain a style migration image with a target domain image visual style; extracting text embedding features of the labeled source domain image by using a pre-trained vision-language pre-training model to obtain text embedding features of a semantic category; fusing the tagged source domain image, the style migration image and the text embedding features of the semantic category to generate an intermediate domain fusion image containing double-domain information and language priori knowledge; performing random mask processing on the intermediate domain fusion image; and inputting the multi-scale context features and the text embedding features of the student network extraction mask image into a vision-language decoder, and then carrying out semantic segmentation on a prediction result.

Description

Technical Field

[0001] This invention relates to a method and system for semantic segmentation and prediction of remote sensing images based on a vision-language pre-trained model, belonging to the fields of computer vision and remote sensing image processing technology. Background Technology

[0002] With the rapid development of deep learning technology, semantic segmentation methods based on Convolutional Neural Networks (CNN) and Transformers have made significant progress in remote sensing image interpretation. However, the performance of these methods is highly dependent on a large amount of pixel-level labeled data, while the manual annotation of remote sensing images is extremely costly, and the annotation cost is even higher due to the greater complexity and size of remote sensing images. In addition, there is a significant domain shift problem among remote sensing images acquired from different geographical regions, at different times, and with different sensors, making it difficult to directly apply models trained on one dataset to other datasets. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for semantic segmentation prediction of remote sensing images based on a vision-language pre-trained model. By using a language-guided dual-domain fusion strategy, unsupervised domain adaptive semantic segmentation of remote sensing images is achieved, which significantly reduces the dependence on labeled data, effectively mitigates the impact of domain shift, and improves the model's generalization ability and segmentation accuracy on unseen remote sensing data.

[0004] To achieve the above objectives, the present invention is implemented using the following technical solution.

[0005] On one hand, this invention provides a method for semantic segmentation and prediction of remote sensing images based on a vision-language pre-trained model, comprising: Obtain labeled source domain images and unlabeled target domain images; The teacher network is used to add pseudo-labels to the unlabeled target domain image, providing a supervision signal for the student network. The labeled source domain image is style-transferred to obtain a style-transferred image with the visual style of the target domain image; Text embedding features of the labeled source domain image are extracted using a vision-language pre-trained model to obtain text embedding features of semantic categories; The labeled source domain image, style transfer image, and semantic category text embedding features are fused to generate an intermediate domain fusion image containing dual domain information and language prior knowledge. The intermediate domain fused image is subjected to random masking to obtain a masked image; The student network extracts multi-scale contextual features of the masked image and text embedding features of the semantic category. The two types of features are input into a pre-constructed visual-language decoder, which then outputs the semantic segmentation prediction results of the remote sensing image. The student network updates the teacher network using an exponential moving average.

[0006] Furthermore, the step of using the teacher network to add pseudo-labels to the unlabeled target domain image to provide supervision signals for the student network includes: Based on the text embedding features of the semantic categories, calculate the semantic similarity between each semantic category, and identify semantic category pairs whose semantic similarity is higher than a preset threshold; Based on the unlabeled target domain image, boundary regions are identified using a superpixel clustering algorithm to obtain the boundary mask; The proportion of pixels whose predicted probability from the teacher network exceeds the high confidence threshold is used as the overall confidence level. Based on the semantic category pair, the boundary mask, and the overall confidence, the pseudo-label is dynamically weighted and trained to obtain the final target domain pseudo-label. The final target domain pseudo-labels are used to provide supervisory signals for student networks.

[0007] Furthermore, the vision-language pre-trained model adopts the OpenCLIP model, which includes an image encoder and a text encoder; The image encoder is used to map the labeled source domain image to an image feature vector; The text encoder is used to map the text description corresponding to the labeled source domain image into a text feature vector with the same dimension as the image feature vector, and generate text embedding features for each semantic category.

[0008] Furthermore, before mapping the text description corresponding to the labeled source domain image to a text feature vector with the same dimension as the image feature vector, the semantic category description is optimized using cue word engineering techniques, thereby increasing the feature distance between similar semantic categories in the text embedding space.

[0009] Furthermore, a bidirectional generative adversarial network is used to perform style transfer on the labeled source domain image, including: A generator using a bidirectional generative adversarial network transforms the labeled source domain image into a style-transferred image with the visual style of the target domain image, while preserving the original semantic content through cyclic consistency constraints. The discriminator of a bidirectional generative adversarial network is used to evaluate the realism of the images generated by the generator, and the generator is optimized through adversarial training so that the style-transferred images generated by the generator are visually close to the target domain images, but semantically consistent with the source domain images.

[0010] Further, the process of fusing the labeled source domain image, style transfer image, and semantic category text embedding features to generate an intermediate domain fusion image containing dual-domain information and language prior knowledge includes: Extract image features from the source domain image and the style transfer image respectively; The image features and the text embedding features are respectively subjected to L2 normalization. Based on the normalized image features and text embedding features, the cosine similarity between the image features and text embedding features at each pixel location is calculated. Based on the cosine similarity calculation results, cross-domain semantic prior information is added to each pixel location to obtain the associated feature map. The source domain image, style transfer image, and associated feature map are used to generate a smooth intermediate domain fusion image through a learnable convolutional neural network.

[0011] Further, the step of performing random masking processing on the intermediate domain fused image to obtain a masked image includes: A block masking strategy is used to divide the intermediate domain fused image into... Blocks of varying sizes, randomly selected at a ratio of [missing information]. r The block is masked, and all pixel values ​​of the masked block are set to 0.

[0012] Furthermore, the training method for the student network includes: The student network is trained using labeled data from the source domain image, with the optimization objective being to minimize the cross-entropy loss, as shown in the formula: (1); In the formula, The first segmentation loss is obtained by calculating the segmentation prediction result of the student network on the labeled source domain data and its true label. For the true annotation of the source domain, The predicted probability of the student network; (i,k) are the pixel index and category index in the labeled source domain data; The student network is used to make predictions on the mask-enhanced target domain image, and the cross-entropy loss is calculated using weighted pseudo-labels, as shown in the formula: (2); In the formula, The second segmentation loss is calculated by comparing the student network's segmentation prediction results for unlabeled target domain data with high-quality target domain pseudo-labels; i,j,k ) represents the pixel position and category index in the unlabeled target domain data. For the student network to pixel ( i,j ) belongs to category k The predicted probability, Dynamic weights for corresponding pixels and categories; according to and The total loss function is obtained by weighted summation. The total loss function The calculation expression is as follows: (3), In the formula, and These are preset weighting coefficients; Based on the total loss function The parameters of the student network are optimized through backpropagation.

[0013] Furthermore, the student network adopts a hierarchical Transformer architecture, which includes multiple consecutively connected encoding stages to sequentially output feature maps with decreasing resolution, so as to capture both local details and global information.

[0014] On the other hand, the present invention provides a remote sensing image segmentation system based on a vision-language pre-trained model, comprising: The data acquisition module is configured to acquire labeled source domain images and unlabeled target domain images; The pseudo-label generation module is configured to use the teacher network to add pseudo-labels to the unlabeled target domain image, providing a supervision signal to the student network. The style transfer module is configured to perform style transfer on the labeled source domain image to obtain a style-transferred image with the visual style of the target domain image; The text embedding extraction module is configured to extract text embedding features of the labeled source domain image using a vision-language pre-trained model to obtain text embedding features of semantic categories. The dual-domain fusion module is configured to fuse the labeled source domain image, style transfer image, and semantic category text embedding features to generate an intermediate domain fusion image containing dual-domain information and language prior knowledge. The masking module is configured to perform random masking on the intermediate domain fused image to obtain a masked image; The segmentation prediction module is configured to extract multi-scale contextual features of the masked image through the student network and call the text embedding features of the semantic category, input the two types of features into a pre-constructed visual-language decoder, and then output the semantic segmentation prediction result of the remote sensing image.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention employs a self-training method based on a teacher-student model framework to address unsupervised domain adaptation tasks. By generating high-quality pseudo-labels for unlabeled target domain images, it significantly reduces reliance on manually labeled data, thereby effectively lowering labeling costs. The random masking technique used in this invention guides the model to learn the intrinsic structure and spatial relationships of the data, rather than relying on labeled supervision signals. This enables the model to obtain stronger domain-invariant feature representations, thereby improving generalization performance and effectively mitigating domain bias. This invention introduces a vision-language pre-trained model as an intermediary. The text embeddings generated by this model serve as stable and invariant semantic prior knowledge, maintaining consistency across different data domains. This effectively bridges the semantic spaces of the source and target domains, providing crucial support for achieving domain adaptation. Attached Figure Description

[0016] Figure 1 The diagram shows a module schematic of a remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model. Figure 2 The diagram shows a flowchart of a remote sensing image semantic segmentation prediction method based on a vision-language pre-trained model. Detailed Implementation

[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0019] Example 1 See Figure 1 and Figure 2 This embodiment introduces a method for semantic segmentation and prediction of remote sensing images based on a vision-language pre-trained model. The method specifically includes: Step S1: Obtain labeled source domain data and unlabeled target domain data; In this embodiment, the ISPRS Posdam remote sensing image deep learning semantic segmentation dataset can be selected as the source domain data. This dataset contains high-resolution aerial images with a resolution of 6000×6000 pixels, which have been labeled with six semantic tags: "impermeable surface, building, low vegetation, tree, car, and background". After acquisition, the images are cropped and normalized to a size of 512×512 pixels and divided into training and validation sets in a 7:3 ratio.

[0020] In this embodiment, the preprocessing method for the target domain image is similar to that for the source domain data, that is, it is uniformly cropped to 512×512 pixels and the pixel value is normalized. However, the key difference is that the target domain image does not need to provide any manual annotation.

[0021] For the six semantic segmentation target categories of "impermeable surface, building, low vegetation, tree, car, and background" in this embodiment, standardized text descriptions are constructed as follows: Impermeable surfaces include roads, squares, and other areas, and are in the form of continuous sheets or long strips, mostly dark gray or grayish-black in color; The city's buildings are mostly densely distributed in block-like areas with clear outlines and colors mainly in light gray, off-white, and earth yellow. Low vegetation includes lawns and low shrubs, which are evenly covered and relatively low in height, and are light green and yellowish-green in color. The trees are mostly distributed in clusters or clumps, with a relatively rough texture and a dark green or blackish-green color. In remote sensing images, cars appear as tiny dots or short strips, and are mostly black, white, or silver-gray in color. Background refers to areas other than those mentioned above, lacking obvious target semantic features, and mostly consisting of image edge transitions or clutter areas; precise visual feature descriptions enhance the semantic association between text and remote sensing images, providing reliable linguistic guidance for cross-domain segmentation.

[0022] Step S2: Use the teacher network to add pseudo-labels to the unlabeled target domain image to provide supervision signals for the student network; This embodiment employs a self-training method based on a teacher-student architecture. Its core idea is to generate pseudo-labels using the model's predictions on unlabeled data, and then use these pseudo-labels to retrain the model, thereby fully extracting the effective information from the unlabeled data. In this architecture, the teacher network and student network have the same structure but different parameter update methods: The student network serves as the training subject, and its parameters are directly optimized through backpropagation and gradient descent. The teacher network does not participate in gradient updates; its main responsibility is to generate pseudo-labels on the target domain data, providing supervision signals to the student network. The parameters of the teacher network are obtained from the parameters of the student network through an exponential moving average, calculated as follows: (4), In the formula, For the parameters of the teacher network, For the parameters of the student network, This is the momentum coefficient, which is usually set between 0.99 and 0.999.

[0023] In this embodiment, the teacher-student model is based on the SegFormer-B4 network, which adopts a hierarchical Transformer structure and can output feature maps at four different scales, taking into account both local details and global semantic information.

[0024] Training on source domain data enables the student network to acquire basic semantic segmentation capabilities. However, due to the distributional differences between the source and target domains, directly applying the model to the target domain leads to performance degradation. To adapt the model to the target domain, it is necessary to train it using unlabeled data from the target domain. The teacher network generates pseudo-labels on the target domain images, calculated as follows: (5), In the formula, pixel position T The generated pseudo-labels, i.e., the predicted categories; This refers to the local image features or image patch at pixel location T in the target domain image; For teacher network models; For teacher networks regarding input features The predicted output; For all categories k In the middle, select the index of the category with the highest predicted score.

[0025] The teacher network generates initial pseudo-labels for unlabeled target domain images, enabling it to initially master the ability to segment remote sensing images of six categories: "impermeable surfaces, buildings, low vegetation, trees, cars, and background." Specific training parameters are shown in Table 1. Parameter name Parameter Indicators Iteration rounds 50 rounds Optimizer AdamW Initial learning rate 1e-4 Weight decay 1e-5 Table 1 The teacher network outputs the predicted probability distribution of each pixel for six categories: "impermeable surface, building, low vegetation, tree, car, and background", and selects the category corresponding to the maximum probability as the initial pseudo-label of the pixel.

[0026] After training is completed, the parameters of the student network are initialized as a copy of the parameters of the teacher network. This serves as the starting point for subsequent adaptive training, and the parameters are updated synchronously during the training process using an exponential moving average (EMA) strategy.

[0027] The total training loss is a weighted sum of the source domain supervision loss and the target domain pseudo-label loss. This self-training mechanism allows the model to gradually adapt to the data distribution in the target domain, improving its generalization ability in that domain. However, traditional self-training methods have a key problem: the quality of pseudo-labels directly affects model performance. Since the target domain data is unlabeled, the pseudo-labels generated by the teacher network inevitably contain errors. If incorrect pseudo-labels are used to train the student network, errors will accumulate, causing model performance to degrade or even crash. To mitigate this problem, a confidence filtering strategy is usually adopted, using only pixels with a predicted probability exceeding a certain threshold as pseudo-labels. Furthermore, pseudo-labels can be quality-evaluated and reweighted, giving higher weight to high-quality pseudo-labels and reducing the impact of low-quality pseudo-labels.

[0028] Therefore, in this embodiment, the proportion of pixels with predicted probabilities exceeding the high confidence threshold to the total number of pixels in each initial pseudo-label is counted and recorded as the overall confidence of the initial pseudo-label.

[0029] Step S3: Based on the semantic category text data, perform text embedding extraction processing using a vision-language pre-trained model to obtain category text embeddings; This embodiment uses OpenCLIP as the vision-language pre-training model. OpenCLIP is a large-scale CLIP model proposed by the LAION team in 2022. Self-supervised training of OpenCLIP is performed using the LAION-5B dataset, which contains 5.8 billion closely related image-text pairs, covering an extremely wide range of visual concepts and semantic relationships. This allows the text embeddings learned by the model to possess strong semantic universality and cross-domain generalization ability. Based on this characteristic, the text embeddings provided by OpenCLIP can serve as domain-invariant semantic prior knowledge, maintaining semantic consistency across remote sensing data domains from different sources and with different distributions. This lays the foundation for effective unsupervised domain adaptation in this embodiment.

[0030] Before performing text embedding and extraction, the semantic category text data is optimized using cue word engineering techniques. For example, the basic term "building" is optimized to "a regular geometric man-made structure with roofs and shadows from an aerial view"; "low vegetation" is optimized to "a homogeneous, low-lying textured vegetation area, such as a lawn, from an aerial view." In this way, the text description more accurately corresponds to the visual features of the remote sensing image.

[0031] In this embodiment, OpenCLIP employs a dual-branch structure of "text encoder-image encoder." The text encoder is a 12-layer Transformer structure with 768 hidden layers and 12 attention heads, responsible for converting text sequences into semantic features. The image encoder uses a ViT-B / 16 (Vision Transformer-Base / 16) structure, segmenting the input image into 16×16 pixel blocks and then encoding the visual features using a Transformer. The features output by the dual encoders are aligned through contrastive learning to construct cross-modal semantic associations.

[0032] The texts of the six semantic segmentation target categories "impermeable surface, building, low vegetation, tree, car, and background" in step S1 are input into the text encoder in OpenCLIP. During processing, the Byte-Pair Encoding strategy is used to segment the text first, converting the text into a text tensor of length 77. After calculation by the text encoder, the output feature vector with dimensions [1, 768] is obtained, which is the category text embedding corresponding to each semantic category. Finally, the embedding matrix formed by the six category text embeddings is obtained, which is used for subsequent cross-modal guidance tasks.

[0033] Using the six categories of text embeddings generated in step S3, the semantic similarity between each category is calculated. The formula for calculating the cosine similarity of the normalized text embeddings is as follows: (6), In the formula, and Any two of the six categories, and This is the output of the text encoder for the corresponding category in OpenCLIP. A higher cosine similarity indicates that the directions of the text embedding vectors of the two categories are closer, meaning the two categories are semantically similar and visually more easily confused. In this embodiment, a preset semantic similarity threshold of 0.7 is set. When the semantic similarity between two categories exceeds this threshold (0.7), the category pair is marked as an easily confused category pair, providing a basis for subsequently enhancing the model's discriminative ability.

[0034] A superpixel clustering algorithm is used to over-segment the preprocessed unlabeled target domain data from step S1, dividing it into multiple superpixel blocks with similar visual features. For each superpixel block, the category distribution of its initial pseudo-labels is statistically analyzed, and the entropy value of this distribution is calculated. Superpixel blocks with higher entropy values ​​typically correspond to regions containing multiple categories, i.e., the boundary regions of semantic objects. Due to the inherent ambiguity of annotation, the reliability of pseudo-labels in boundary regions is usually low. The boundary regions identified through the above process are encoded into a binary boundary mask, providing spatial location information for the subsequent weighted confidence of pseudo-labels.

[0035] This embodiment uses the initial pseudo-labels as a basis to construct a dynamically weighted loss function to achieve pseudo-label optimization. The specific strategy is as follows: 1. Confidence weighting: Set a high confidence threshold of 0.9. When the prediction probability > 0.9, increase the training weight of the image pseudo-label to 1.2; otherwise, decrease the training weight to 0.8. 2. Introducing a penalty coefficient for easily confused category pairs: For pixels corresponding to easily confused category pairs, a dynamic semantic penalty term based on text similarity is introduced; where the penalty coefficient is dynamically determined by the semantic similarity mapping of the category pairs: that is, the closer the semantic association, the higher the cosine similarity, and the higher the penalty coefficient. 3. The pseudo-label loss of pixels in the boundary region is weighted based on the boundary mask, and the weight of the boundary region is increased by 0.3 to improve the segmentation accuracy of the boundary region.

[0036] The aforementioned weighting strategy was integrated into the joint training of the teacher-student model. The model was iteratively trained for 50 rounds with the teacher network generating pseudo-labels, the student network learning and optimizing, and EMA updating the teacher network parameters. After each round, the overall confidence and semantic similarity were recalculated, and the weight parameters were dynamically adjusted to finally obtain high-quality target domain pseudo-labels.

[0037] Step S4: Based on the labeled source domain data, the unlabeled target domain data, and the category text embedding, perform language-guided dual-domain image fusion to obtain an intermediate domain image; This embodiment achieves deep fusion of source domain semantic information and target domain style features through style transfer and visual-linguistic feature association, generating intermediate domain images for model training. The specific process is divided into three stages: style transfer, feature association calculation, and intermediate domain fusion. This embodiment employs a Bidirectional Generative Adversarial Network (Bidirectional GAN) for style transfer from the source domain to the target domain. The generator uses a residual network to process the source domain image... Convert to an image with the visual style of the target domain Simultaneously, the original semantic content is preserved through cycle consistency constraints. The discriminator evaluates the realism of the generated images, and the generator is optimized through adversarial training so that the style-transfer images it produces are visually close to the target domain, but semantically consistent with the source domain. This process effectively reduces the appearance differences between the source and target domains, laying the foundation for subsequent dual-domain fusion. The style transfer network is trained for 30 epochs, with the optimizer being Adam, the learning rate being 2e-4, and the batch size being 4.

[0038] In step S1, an embedding matrix is ​​obtained by forming six categories of text embeddings. The adaptation layer maps the image to the same dimension as the image features, preparing for subsequent text-image association.

[0039] The encoder extracts features from both the original source domain image and source domain data with the visual style of the target domain to obtain image features. and .

[0040] To make text embeddings compatible with image features, an adaptation layer is typically needed to map the dimensions of the text embeddings to the same dimensions as the image features. The calculation expression is as follows: (7), in, This is the text embedding feature vector after mapping through the adaptation layer; This is the embedding vector for the k-th text. The weight matrix for the adaptation layer is a learnable parameter matrix. Its function is to linearly transform the dimension of the text embedding vector to the target dimension (i.e., the dimension of the image features). The bias vector for the adaptation layer is also a learnable parameter used to add an offset after transformation, thereby enhancing the model's expressive power.

[0041] To establish the correlation between image features and text features, L2 normalization is first applied to both to obtain normalized features. , and text embedding Then calculate the position of each pixel ( ij The image features of ) and the first k The cosine similarity formula for text embeddings of each category is as follows: (8), The first and second association feature maps are obtained respectively: the first association feature map M1 is used to represent the semantic association between the source domain image and the text; the second association feature map M2 is used to capture the association between the image and the text after style transfer.

[0042] An intermediate domain fusion strategy combining feature concatenation and convolutional fusion is employed. The labeled source domain data, the source domain data with the visual style of the target domain, the first association feature map, and the second association feature map are concatenated along the channel dimension. This embodiment uses a learnable convolutional network to generate a smooth intermediate domain image. The convolutional neural network uses a three-layer structure, and its calculation expression is as follows: 1. Feature encoding layer: 512 3×3 convolutional kernels are used for feature extraction, and the ReLU activation function is used to introduce non-linearity. (9); 2. Feature Compression and Normalization Layer: 256 3×3 convolutional kernels are used to further fuse features, and batch normalization is introduced to stabilize the training process. (10); 3. Output layer: A 1×1 convolutional kernel is used to map the number of channels to 3 channels, consistent with the original RGB image, and the final output is the fused intermediate domain image: (11); Step S5: Based on the category text embedding, high-quality target domain pseudo-labels, and intermediate domain images, train the text conditional mask segmentation network and output the segmentation results; In this embodiment, the text conditional mask segmentation network consists of three parts: a random mask module, a student network, and a visual-language decoder.

[0043] The random masking module performs random masking on the intermediate domain image. Mask generation employs a block masking strategy, dividing the image into... Blocks of varying sizes, randomly selected at a ratio of [missing information]. r The network masks the blocks of pixels, setting all pixel values ​​in the masked blocks to 0, while retaining the original pixel information in the unmasked areas, thus generating a masked image. This masking strategy forces the network to use information from the unmasked areas to infer the semantics of the masked areas, thereby learning the spatial context of the image. Compared to traditional full-image training, masked image modeling encourages the network to focus on the spatial dependencies between pixels, learning more robust feature representations.

[0044] The student network encoder employs a hierarchical Transformer architecture for multi-scale contextual feature extraction from masked images. This encoder comprises four consecutively connected encoding stages: Stage 1 outputs a 512×512 resolution feature map to capture local details of small targets such as cars; Stage 2 outputs a 256×256 resolution feature map; Stage 3 outputs a 128×128 resolution feature map; and Stage 4 outputs a 64×64 resolution feature map to capture global information of large targets such as impermeable surfaces. Features are transferred between stages via residual connections, enabling the collaborative capture of both local details and global information.

[0045] The visual-language decoder first upsamples and aligns the multi-scale features output by the encoder, and then fuses them to obtain unified image features. The association matrix between image features and text embeddings is calculated, the similarity between each pixel location and each semantic category is measured, and an association feature map is generated. S The associated feature map and image features are concatenated along the channel dimension to form a joint feature representation containing both visual and linguistic information. The joint features are then mapped to... (The sentence is incomplete and requires further context to translate accurately.) K The predicted output for each semantic category is calculated using the following expression: (12) This design allows the decoder to rely not only on visual features but also on semantic prior knowledge contained in text embeddings when performing semantic segmentation. Text embeddings are pre-trained on large-scale image-text pairs and contain rich cross-domain semantic information, which helps the network better understand and distinguish semantically similar categories.

[0046] Step S5, the specific training process, includes: The intermediate domain image generated in step S4 is input into the random masking module to obtain the mask image. At the same time, the high-quality target domain pseudo-label from step S3 and the labeled source domain data from step S1 are called to form a "mask image-label" training pair. After the masked image is processed by the student network encoder to extract multi-scale contextual features, it is input into the visual-language decoder and fused with the category text embedding to generate segmentation prediction results for six semantic categories. Constructing a hybrid loss function: The student network's segmentation prediction results for the labeled source domain data are compared with their true labels, and the resulting loss is the first segmentation loss. The student network's segmentation prediction results for unlabeled target domain data are compared with high-quality target domain pseudo-labels to calculate the second segmentation loss. ; Among them, the first segmentation loss Second segmentation loss It is calculated using the cross-entropy loss function, and the calculation expression is as follows: (1), In the formula, Predict the probability for the source domain, ( i,k () represents the pixel index and category index in the labeled source domain data. For one-hot encoding that is actually labeled in the source domain; (2), In the formula, ( i,j,k () represents the pixel position and category index in the unlabeled target domain data. For student networks to pixel ( i,j ) belongs to category k The predicted probability, Dynamic weights for corresponding pixels and categories; According to the first segmentation loss Second segmentation loss The total loss function is obtained by weighted summation. Total loss function The calculation expression is as follows: (3).

[0047] In each training iteration, a labeled image is sampled from the source domain, fused into a fused image, and then masked and propagated forward to calculate the source domain supervision loss. An unlabeled image is sampled from the target domain, augmented and masked, and then input into the student network for prediction. The teacher network generates pseudo-labels on the original image, and a weighted strategy is used to calculate the target domain pseudo-label loss. The two losses are jointly used to optimize the student network parameters, while the teacher network parameters are updated using an exponential moving average.

[0048] For test images in the target domain, there is no need for style transfer and masking. The semantic segmentation prediction results are obtained directly through forward propagation using the trained student encoder and vision-language decoder.

[0049] Example 2 Based on the same inventive concept as Embodiment 1, this embodiment introduces a remote sensing image segmentation system based on a vision-language pre-trained model, comprising: The data acquisition module is configured to acquire labeled source domain images and unlabeled target domain images; The text embedding extraction module is configured to extract text embedding features of semantic categories using a vision-language pre-trained model; The pseudo-label generation module is configured to generate pseudo-labels for the unlabeled target domain image using a teacher network. The style transfer module is configured to perform style transfer on the labeled source domain image to obtain a style-transferred image with the visual style of the target domain. The dual-domain fusion module is configured to fuse the labeled source domain image, style transfer image, and semantic category text embedding features to generate an intermediate domain fusion image containing dual-domain information and language prior knowledge. The masking module is configured to perform random masking on the intermediate domain fused image to obtain a masked image; The segmentation prediction module is configured to extract multi-scale contextual features of the masked image through a student network, combine them with text embedding features of the semantic category, input them into a visual-language decoder, and output semantic segmentation prediction results.

[0050] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0051] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0055] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for semantic segmentation and prediction of remote sensing images based on a vision-language pre-trained model, characterized in that, include: Obtain labeled source domain images and unlabeled target domain images; The teacher network is used to add pseudo-labels to the unlabeled target domain image, providing a supervision signal for the student network. The labeled source domain image is style-transferred to obtain a style-transferred image with the visual style of the target domain image; Text embedding features of the labeled source domain image are extracted using a vision-language pre-trained model to obtain text embedding features of semantic categories; The labeled source domain image, style transfer image, and semantic category text embedding features are fused to generate an intermediate domain fusion image containing dual domain information and language prior knowledge. The intermediate domain fused image is subjected to random masking to obtain a masked image; The student network extracts multi-scale contextual features of the masked image and text embedding features of the semantic category. The two types of features are input into a pre-constructed visual-language decoder, which then outputs the semantic segmentation prediction results of the remote sensing image. The student network updates the teacher network using an exponential moving average.

2. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 1, characterized in that, The step of adding pseudo-labels to the unlabeled target domain image using the teacher network to provide supervision signals to the student network includes: Based on the text embedding features of the semantic categories, calculate the semantic similarity between each semantic category, and identify semantic category pairs whose semantic similarity is higher than a preset threshold; Based on the unlabeled target domain image, boundary regions are identified using a superpixel clustering algorithm to obtain the boundary mask; The proportion of pixels whose predicted probability from the teacher network exceeds the high confidence threshold is used as the overall confidence level. Based on the semantic category pair, the boundary mask, and the overall confidence, the pseudo-label is dynamically weighted and trained to obtain the final target domain pseudo-label. The final target domain pseudo-labels are used to provide supervisory signals for student networks.

3. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 1, characterized in that, The vision-language pre-trained model uses the OpenCLIP model, which includes an image encoder and a text encoder; The image encoder is used to map the labeled source domain image to an image feature vector; The text encoder is used to map the text description corresponding to the labeled source domain image into a text feature vector with the same dimension as the image feature vector, and generate text embedding features for each semantic category.

4. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 3, characterized in that, Before mapping the text description corresponding to the labeled source domain image to a text feature vector with the same dimension as the image feature vector, the semantic category description is optimized using cue word engineering techniques, thereby increasing the feature distance between similar semantic categories in the text embedding space.

5. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 1, characterized in that, Style transfer of the labeled source domain images is performed using a bidirectional generative adversarial network, including: A generator using a bidirectional generative adversarial network transforms the labeled source domain image into a style-transferred image with the visual style of the target domain image, while preserving the original semantic content through cyclic consistency constraints. The discriminator of a bidirectional generative adversarial network is used to evaluate the realism of the images generated by the generator, and the generator is optimized through adversarial training so that the style-transferred images generated by the generator are visually close to the target domain images, but semantically consistent with the source domain images.

6. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 1, characterized in that, The process of fusing the labeled source domain image, style transfer image, and semantic category text embedding features to generate an intermediate domain fused image containing dual-domain information and language prior knowledge includes: Extract image features from the source domain image and the style transfer image respectively; The image features and the text embedding features are respectively subjected to L2 normalization. Based on the normalized image features and text embedding features, the cosine similarity between the image features and text embedding features at each pixel location is calculated. Based on the cosine similarity calculation results, cross-domain semantic prior information is added to each pixel location to obtain the associated feature map. The source domain image, style transfer image, and associated feature map are used to generate a smooth intermediate domain fusion image through a learnable convolutional neural network.

7. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 1, characterized in that, The step of performing random masking processing on the intermediate domain fused image to obtain a masked image includes: A block masking strategy is used to divide the intermediate domain fused image into... Blocks of varying sizes, randomly selected at a ratio of [missing information]. r The block is masked, and all pixel values ​​of the masked block are set to 0.

8. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 3, characterized in that, The training method for the student network includes: The student network is trained using labeled data from the source domain image, with the optimization objective being to minimize the cross-entropy loss, as shown in the formula: （1）； In the formula, The first segmentation loss is obtained by calculating the segmentation prediction result of the student network on the labeled source domain data and its true label. For the true annotation of the source domain, The predicted probability of the student network; (i,k) are the pixel index and category index in the labeled source domain data; The student network is used to make predictions on the mask-enhanced target domain image, and the cross-entropy loss is calculated using weighted pseudo-labels, as shown in the formula: （2）； In the formula, The second segmentation loss is calculated by comparing the student network's segmentation prediction results for unlabeled target domain data with high-quality target domain pseudo-labels; i,j,k ) represents the pixel position and category index in the unlabeled target domain data. For the student network to pixel ( i,j ) belongs to category k The predicted probability, Dynamic weights for corresponding pixels and categories; according to and The total loss function is obtained by weighted summation. The total loss function The calculation expression is as follows: （3）， In the formula, and These are preset weighting coefficients; Based on the total loss function The parameters of the student network are optimized through backpropagation.

9. The remote sensing image semantic segmentation and prediction method based on a vision-language pre-trained model according to claim 8, characterized in that, The student network adopts a hierarchical Transformer architecture, which contains multiple consecutively connected encoding stages to sequentially output feature maps with decreasing resolution, so as to capture both local details and global information.

10. A remote sensing image segmentation system based on a vision-language pre-trained model, characterized in that, include: The data acquisition module is configured to acquire labeled source domain images and unlabeled target domain images; The pseudo-label generation module is configured to use the teacher network to add pseudo-labels to the unlabeled target domain image, providing a supervision signal to the student network. The style transfer module is configured to perform style transfer on the labeled source domain image to obtain a style-transferred image with the visual style of the target domain image; The text embedding extraction module is configured to extract text embedding features of the labeled source domain image using a vision-language pre-trained model to obtain text embedding features of semantic categories. The dual-domain fusion module is configured to fuse the labeled source domain image, style transfer image, and semantic category text embedding features to generate an intermediate domain fusion image containing dual-domain information and language prior knowledge. The masking module is configured to perform random masking on the intermediate domain fused image to obtain a masked image; The segmentation prediction module is configured to extract multi-scale contextual features of the masked image through the student network and call the text embedding features of the semantic category, input the two types of features into a pre-constructed visual-language decoder, and then output the semantic segmentation prediction result of the remote sensing image.

Images