A semantic segmentation system for buildings in remote sensing images based on visual language model

Abstract

The present invention provides a remote sensing image building semantic segmentation system based on a visual language model. The remote sensing image building semantic segmentation system includes an initial building mask generation module and a pseudo-label optimization and model iteration enhancement module. The initial building mask generation module includes a query text set construction unit, a feature extraction unit, a cross-modal attention interaction unit, an initial building instance generation unit, and an instance fusion and mask generation unit. The pseudo-label optimization and model iteration enhancement module includes a pseudo-label screening unit, a model iteration training unit, and a dynamic parameter adjustment unit. The present invention can improve the quality of building segmentation pseudo-labels generated by VLMs, enhance the performance of weakly supervised semantic segmentation algorithms for remote sensing images, solve the problem that building labels generated by VLMs pre-trained on natural scene images cannot be directly applied to remote sensing images, and reduce the reliance of remote sensing image building semantic segmentation on manual annotation.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a remote sensing image building semantic segmentation system based on a visual language model. Background Art

[0002] Building semantic segmentation, a key task in computer vision for remote sensing image analysis, is dedicated to accurately classifying each pixel in a remote sensing image as a building or non-building category, laying a solid foundation for subsequent geographic information analysis. It has indispensable applications in numerous fields, including urban planning, disaster monitoring, and environmental assessment. In urban planning, accurate building segmentation results can help planners clarify urban spatial layout and rationally plan land use and infrastructure construction. In disaster monitoring scenarios, building damage in affected areas can be quickly identified, providing critical information for rescue operations. In environmental assessments, it helps analyze the impact of building distribution on the ecological environment.

[0003] Traditional building semantic segmentation methods rely heavily on large-scale, manually annotated data. However, manual annotation not only requires significant human, material, and time resources, but is also susceptible to subjective interference, leading to inconsistent results. Furthermore, traditional methods lack segmentation accuracy in complex and ever-changing real-world scenarios, such as varying architectural styles, scales, complex backgrounds, and occlusions.

[0004] With breakthroughs in cross-modal semantic alignment and zero-shot reasoning, visual language models (VLMs) have shown great potential in image semantic understanding. Currently, several high-performing VLMs have been developed, often pre-trained using natural scene images. However, due to significant domain differences between natural images and remote sensing images (such as imaging perspective, resolution, and feature distribution), VLMs pre-trained on natural scene images cannot be directly applied to building segmentation tasks in remote sensing images. Summary of the Invention

[0005] In view of this, the present invention provides a remote sensing image building semantic segmentation system based on a visual language model to solve the problem that VLMs pre-trained on natural scene images cannot be directly applied to the building segmentation task of remote sensing images, improve the model's segmentation accuracy for buildings in remote sensing images, and reduce the dependence on manual labeling of remote sensing images.

[0006] A remote sensing image building semantic segmentation system based on a visual language model includes an initial building mask generation module and a pseudo-label optimization and model iteration enhancement module. The initial building mask generation module includes a query text set construction unit, a feature extraction unit, a cross-modal attention interaction unit, an initial building instance generation unit, and an instance fusion and mask generation unit. The pseudo-label optimization and model iteration enhancement module includes a pseudo-label screening unit, a model iterative training unit, and a dynamic parameter adjustment unit.

[0007] The query text set construction unit is used to construct a diverse query text set consisting of multiple prompt words;

[0008] The feature extraction unit is used to extract text features of the query text set using an alignment hint encoder and to extract multi-scale visual features of remote sensing images;

[0009] The cross-modal attention interaction unit is used to use an adaptive cross-modal attention mechanism to interact text features and multi-scale visual features to obtain visual features that integrate text information;

[0010] The initial building instance generation unit is used to perform threshold processing, connected region analysis and de-overlapping processing on the visual features of the fused text information to generate an initial building instance;

[0011] The instance fusion and mask generation unit is used to fuse the initial building instances obtained based on different query texts to obtain an initial building mask, and obtain an initial building segmentation pseudo label based on the initial building mask;

[0012] The pseudo-label filtering unit is used to filter out reliable pseudo-labels from the initial building segmentation pseudo-labels based on uncertainty quantification, cross-validation consistency, and geometric constraint filtering;

[0013] The model iterative training unit is used to gradually enhance the generalization ability of the system using a two-stage training method of weak supervision training and fine-tuning optimization;

[0014] The dynamic parameter adjustment unit is used to perform dynamic parameter adjustment by adopting threshold adaptive update and query text weight redistribution.

[0015] The visual language model-based remote sensing image building semantic segmentation system provided by the present invention constructs a diverse query text collection, generates text features using a text encoder called an alignment and prompting encoder (APE), and extracts multi-scale features of dual-temporal remote sensing images through a visual encoder. An adaptive cross-modal attention mechanism is then used to deeply fuse text features with visual features, enabling precise feature alignment. The fused features are then subjected to threshold processing, connected region analysis, and de-overlapping processing to generate initial building instances. Initial building masks are then obtained by weighted summation of instance confidences from different query texts. Initial building segmentation pseudo-labels are then obtained through binarization. Pseudo-label screening, iterative model training, and dynamic parameter adjustment are then used to gradually improve the reliability of pseudo-labels and the system's adaptability to remote sensing scenes. The system effectively utilizes initial pseudo-labels and reduces the impact of noise, improving the accuracy of building segmentation in remote sensing images. This addresses the issue that building labels generated by VLMs pre-trained on natural scene images cannot be directly applied to remote sensing images, reducing reliance on manual annotation of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a structural block diagram of a remote sensing image building semantic segmentation system based on a visual language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0018] See also Figure 1 An embodiment of the present invention provides a remote sensing image building semantic segmentation system based on a visual language model, including an initial building mask generation module and a pseudo-label optimization and model iteration enhancement module; the initial building mask generation module includes a query text set construction unit, a feature extraction unit, a cross-modal attention interaction unit, an initial building instance generation unit, and an instance fusion and mask generation unit; the pseudo-label optimization and model iteration enhancement module includes a pseudo-label screening unit, a model iterative training unit, and a dynamic parameter adjustment unit.

[0019] Among them, in order to generate the initial building mask more accurately, the initial building mask generation module generates the initial building mask from the training set. No. Remote sensing images ( ) for in-depth processing to generate building masks.

[0020] The query text set construction unit is used to construct a diversified query text set consisting of multiple prompt words.

[0021] Among them, this application first constructs a k A diverse set of query texts consisting of prompt words , ,in, 、 、 The first, second, and query texts. The prompt words can be words or phrases, such as "building", "residential building", "commercial building", "skyscraper", "villa" and other building-related words and phrases. Compared with the existing technology, which only uses a single word (such as "house" or "building", etc.) as a prompt word query, the diversified query text set of this application can make full use of the powerful text understanding ability of the visual base model to capture the building features in the image from different angles. Different query texts can cover buildings of different types, sizes and uses, thereby improving the recognition ability of various buildings.

[0022] The feature extraction unit is used to extract text features of the query text set using an alignment hint encoder, and to extract multi-scale visual features of remote sensing images.

[0023] The feature extraction unit is specifically used to input each query text in the query text set into the text encoder in the alignment hint encoder, and the text encoder converts the query text into corresponding text features. query text , this application inputs it into the text encoder of APE. The text encoder converts the query text into corresponding text features , text features Contains the semantic information of the query text.

[0024] The feature extraction unit is also used to process the remote sensing image using the visual encoder in the alignment hint encoder to extract multi-scale visual features. Remote sensing images , extracting multi-scale visual features , , represents the number of visual feature scales extracted, 、 、 They are the feature maps of the first scale, the feature maps of the second scale, and the feature maps of the Feature maps of different scales.

[0025] The cross-modal attention interaction unit is used to interact text features and multi-scale visual features using an adaptive cross-modal attention mechanism to obtain visual features that integrate text information.

[0026] In order to better adapt to the matching of different query texts and image features in the query text set, this application adopts an adaptive cross-modal attention mechanism, which includes four main processes: similarity calculation, attention weight calculation, adaptive adjustment of attention matrix weights and feature fusion.

[0027] Among them, the cross-modal attention interaction unit is specifically used for:

[0028] Based on the cosine similarity, the similarity between the text features of each query text in the query text set and the feature maps of each scale in the multi-scale visual features is calculated. Remote sensing images No. Feature maps of different scales ,Will Flattened into a sequence of vectors , then calculate the Text features and The dot product of each element in , we get the similarity score matrix ;

[0029] Then use the softmax function to normalize the similarity score matrix to obtain the normalized attention weight matrix, which is expressed as:

[0030]

[0031] in, express The elements, express The elements, is the sequence length of the similarity score matrix, Represents the normalized attention weight matrix The elements;

[0032] In order to further enhance the adaptive capability, this application then introduces the adaptive adjustment factor This adaptive adjustment factor is dynamically adjusted based on the query text's recognition accuracy for feature maps at that scale in historical experiments. If a query text demonstrates a high recognition accuracy for a feature map at a certain scale, the corresponding adaptive adjustment factor will be larger; otherwise, it will be smaller.

[0033] pass Adjust the normalized attention weight matrix, the expression is:

[0034]

[0035] in, Represents the final attention weight matrix The elements;

[0036] Finally, use right Perform weighted summation to obtain the visual features of fused text information , the expression is:

[0037]

[0038] in, express The elements.

[0039] Through this adaptive cross-modal attention mechanism, the system can automatically focus on image regions that are more semantically relevant to the query text, thereby capturing the characteristics of the building more accurately.

[0040] The initial building instance generation unit is used to perform threshold processing, connected region analysis and de-overlapping processing on the visual features of the fused text information to generate an initial building instance.

[0041] The initial building instance generation unit is specifically used to:

[0042] right Perform threshold processing and set the threshold ,Will Medium to large Elements as part of the building, Less than or equal to The elements are used as background to obtain a binary feature map ;

[0043] Use a connected component analysis algorithm (such as the flood fill algorithm) to analyze the binary feature map The algorithm will find A set of interconnected foreground pixels in , where each connected region represents a potential building instance;

[0044] To ensure that building instances do not overlap, the detected connected areas are post-processed. When two or more connected areas overlap, they are screened based on the confidence of the connected areas (such as the sum of pixel values ​​in the area, average similarity score, etc.). The connected area with the highest confidence is retained, and the other connected areas are removed.

[0045] After the above processing, a set of non-overlapping initial building instances can be obtained.

[0046] The instance fusion and mask generation unit is used to fuse initial building instances obtained based on different query texts to obtain an initial building mask, and obtain an initial building segmentation pseudo label based on the initial building mask.

[0047] Among them, the instance fusion and mask generation unit fuses the initial building instances obtained based on different query texts by performing a weighted summation of the instance confidence of each pixel position, and the weights are dynamically allocated according to the effectiveness of different query texts.

[0048] Specifically, this application will maintain a historical accuracy record, recording the recognition accuracy of each query text for buildings in previous training or testing. query text Weight , the calculation formula is:

[0049]

[0050] in, is the total number of query texts in the query text set, For the query text The historical accuracy of For the query text Historical accuracy of

[0051] For pixel locations , is the horizontal coordinate of the pixel, is the vertical coordinate of the pixel, and the confidence of the building instance obtained based on the query text is Perform weighted summation to obtain the fused confidence , the expression is:

[0052]

[0053] After fusion, the application obtains a more accurate and comprehensive building instance set, and then merges the building instances in the building instance set into the initial building mask , , is the set of real numbers, and They are The height and width of the is a pixel-by-pixel probability map where the pixel location Corresponding value , Reflects the image Middle pixel position Possibility of belonging to a building.

[0054] Due to the domain differences between APE’s pre-training data (natural images) and remote sensing images, the system’s confidence scores for buildings are usually conservatively calibrated. To address this domain difference, this application uses a loose binarization threshold. (For example, take 0.25) Binarize the probability map to obtain the initial building segmentation pseudo label, which is expressed as:

[0055]

[0056] in, is the initial building mask Middle pixel position The corresponding value, Pseudo-labeling for initial building segmentation Middle pixel position The corresponding value, is the binarization threshold.

[0057] This binarization threshold A balance is struck between recall and precision, retaining areas with moderate building characteristics (such as partial structures or rooftops with unclear spectral characteristics) while accepting controllable noise. As the initial building segmentation pseudo-label, it will be further refined through uncertainty quantification to filter out false activation areas.

[0058] After obtaining the initial building segmentation pseudo-label Finally, in order to further improve the system's building semantic segmentation accuracy on optical remote sensing image datasets, this application constructs a self-training optimization framework through three core links: pseudo-label screening, model iterative training, and dynamic parameter adjustment.

[0059] Because the initial pseudo-labels are affected by domain differences and model uncertainty, reliable pseudo-labels must be screened out through multi-dimensional evaluation. The pseudo-label screening unit is used to screen out reliable pseudo-labels from the initial building segmentation pseudo-labels based on uncertainty quantification, cross-validation consistency, and geometric constraint filtering.

[0060] Among them, the process of the pseudo-label screening unit based on uncertainty quantification is as follows:

[0061] Calculate the pixel-level entropy value of the initial building mask using the following formula:

[0062]

[0063] in, is the pixel location The corresponding pixel-level entropy value;

[0064] For pixel-level entropy values ​​higher than the entropy threshold For pixel areas with , the pseudo label confidence is lowered or marked as to be corrected;

[0065] The process of pseudo label screening unit based on cross-validation consistency is as follows:

[0066] Use different initialized alignment hint encoders to generate multiple sets of pseudo labels for the same remote sensing image and calculate the pixel-level intersection over union (IoU). If the average pixel-level IoU of multiple sets of pseudo labels in the target area is lower than the IoU threshold, , then the pseudo label of the target area is removed from the training set.

[0067] The process of pseudo-label screening unit based on geometric constraint filtering is as follows:

[0068] Based on the geometric priors of buildings in remote sensing images (such as minimum area, aspect ratio range, etc.), pseudo-labeled areas that do not meet the preset shape characteristics are removed. For example, connected areas with an area less than 10 pixels or an aspect ratio greater than 10:1 will be filtered out.

[0069] The model iterative training unit is used to gradually enhance the generalization ability of the system by adopting a two-stage training method of weak supervision training and fine-tuning optimization.

[0070] In weakly supervised training, the filtered high-quality pseudo labels are combined with a small amount of labeled data. Merge as a pre-training dataset During training, the weighted cross entropy loss function The expression is:

[0071]

[0072] in, For pre-training dataset The number of samples in is the pseudo label quality weight, is the cross entropy function, is the true label (pseudo label can be used when there is no true label).

[0073] Based on pre-training, use labeled data Fine-tune. Introduce a contrastive learning module to input different enhanced versions of the same image (such as rotation and scaling) into the model. By maximizing the similarity of similar pixel features and minimizing the similarity of heterogeneous pixels, the system's ability to distinguish building features is further enhanced. Finally, the total loss function of the remote sensing image building semantic segmentation system is for:

[0074] ;

[0075] in, is the contrastive learning loss function, is the balance coefficient.

[0076] In order to adapt to the scene differences of remote sensing images, the dynamic parameter adjustment unit is used to perform dynamic parameter adjustment by adopting threshold adaptive update and query text weight redistribution.

[0077] The process of the dynamic parameter adjustment unit using threshold adaptive update to adjust the dynamic parameters is as follows:

[0078] Dynamically adjust the binarization threshold based on the system's precision-recall curve on the validation set during training. , when precision increases and recall decreases, reduce ; When precision decreases and recall increases, increase , to balance segmentation accuracy and completeness.

[0079] After each training iteration, count the query text Corresponding historical accuracy , dynamically update its weight in the instance fusion stage. Specifically, the dynamic parameter adjustment unit uses the query text weight redistribution to adjust the dynamic parameters. The update formula is:

[0080]

[0081] in, is an exponential decay factor.

[0082] By adaptively updating thresholds and redistributing query weights, the system gradually focuses on high-performing queries, enhancing its ability to recognize complex building forms. This self-training process effectively leverages the valuable information in the initial pseudo-labels while mitigating the effects of noise through iterative optimization, ultimately significantly improving the accuracy of semantic segmentation of buildings in remote sensing imagery.

[0083] In summary, according to the remote sensing image building semantic segmentation system based on the visual language model of the above embodiment, a diverse query text set is constructed, and the text encoder of the alignment hint encoder is used to generate text features. At the same time, the multi-scale features of the dual-time remote sensing image are extracted through the visual encoder. Then, the text features and visual features are deeply fused using an adaptive cross-modal attention mechanism, which can achieve precise feature alignment. The fused features are subjected to threshold processing, connected region analysis and de-overlapping processing to generate initial building instances. The instance confidences of different query texts are fused by weighted summation to obtain the initial building mask, which is then binarized to obtain the initial building segmentation pseudo-label. Then, through pseudo-label screening, model iterative training and dynamic parameter adjustment, the reliability of the pseudo-label and the adaptability of the system to the remote sensing scene are gradually improved. The initial pseudo-label can be effectively utilized and the influence of noise is reduced, thereby improving the segmentation accuracy of buildings in remote sensing images. This solves the problem that the building labels generated by VLMs pre-trained on natural scene images cannot be directly applied to remote sensing images, and reduces the dependence on manual annotation of remote sensing images.

[0084] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A remote sensing image building semantic segmentation system based on a visual language model, characterized by: It includes an initial building mask generation module and a pseudo-label optimization and model iteration enhancement module; the initial building mask generation module includes a query text set construction unit, a feature extraction unit, a cross-modal attention interaction unit, an initial building instance generation unit, and an instance fusion and mask generation unit; the pseudo-label optimization and model iteration enhancement module includes a pseudo-label screening unit, a model iterative training unit, and a dynamic parameter adjustment unit; The query text set construction unit is used to construct a diverse query text set consisting of multiple prompt words; The feature extraction unit is used to extract text features of the query text set using an alignment hint encoder and to extract multi-scale visual features of remote sensing images; The cross-modal attention interaction unit is used to use an adaptive cross-modal attention mechanism to interact text features and multi-scale visual features to obtain visual features that integrate text information; The initial building instance generation unit is used to perform threshold processing, connected region analysis and de-overlapping processing on the visual features of the fused text information to generate an initial building instance; The instance fusion and mask generation unit is used to fuse the initial building instances obtained based on different query texts to obtain an initial building mask, and obtain an initial building segmentation pseudo label based on the initial building mask; The pseudo-label filtering unit is used to filter out reliable pseudo-labels from the initial building segmentation pseudo-labels based on uncertainty quantification, cross-validation consistency, and geometric constraint filtering; The model iterative training unit is used to gradually enhance the generalization ability of the system using a two-stage training method of weak supervision training and fine-tuning optimization; The dynamic parameter adjustment unit is used to perform dynamic parameter adjustment by adopting threshold adaptive update and query text weight redistribution.

2. The remote sensing image building semantic segmentation system based on the visual language model according to claim 1 is characterized in that: The feature extraction unit is specifically used to input each query text in the query text set into the text encoder in the alignment hint encoder, and the text encoder converts the query text into corresponding text features; The feature extraction unit is further used to process the remote sensing image using the visual encoder in the alignment hint encoder to extract multi-scale visual features.

3. The remote sensing image building semantic segmentation system based on the visual language model according to claim 2 is characterized in that: The cross-modal attention interaction unit is specifically used for: Based on the cosine similarity, the similarity between the text features of each query text in the query text set and the feature maps of each scale in the multi-scale visual features is calculated. Remote sensing images No. Feature maps of different scales ,Will Flattened into a sequence of vectors , then calculate the Text features and The dot product of each element in , we get the similarity score matrix ; Then use the softmax function to normalize the similarity score matrix to obtain the normalized attention weight matrix, which is expressed as: in, express The elements, express The elements, is the sequence length of the similarity score matrix, Represents the normalized attention weight matrix The elements; Then introduce the adaptive adjustment factor Adjust the normalized attention weight matrix, the expression is: in, Represents the final attention weight matrix The elements; Finally, use right Perform weighted summation to obtain the visual features of fused text information , the expression is: in, express The elements.

4. The system for semantic segmentation of buildings in remote sensing images based on a visual language model according to claim 3, characterized in that: The initial building instance generation unit is specifically used to: right Perform threshold processing and set the threshold ,Will Medium to large Elements as part of the building, Less than or equal to The elements are used as background to obtain a binary feature map ; Use the connected component analysis algorithm to analyze the binary feature map Process and find A set of interconnected foreground pixels in , where each connected region represents a potential building instance; The detected connected areas are post-processed. When two or more connected areas overlap, they are screened according to their confidence levels. The connected areas with the highest confidence levels are retained, and the other connected areas are removed, thereby obtaining a set of non-overlapping initial building instances.

5. The remote sensing image building semantic segmentation system based on the visual language model according to claim 4 is characterized in that: The instance fusion and mask generation unit fuses the initial building instances obtained based on different query texts by weighted summing of the instance confidences of each pixel position, and the weights are dynamically allocated according to the validity of different query texts. query text Weight , the calculation formula is: in, is the total number of query texts in the query text set, For the query text The historical accuracy of For the query text Historical accuracy of For pixel locations , is the horizontal coordinate of the pixel, is the vertical coordinate of the pixel, and the confidence of the building instance obtained based on the query text is Perform weighted summation to obtain the fused confidence , the expression is: After fusion, a building instance set is obtained, and then the building instances in the building instance set are merged into the initial building mask , , is the set of real numbers, and They are height and width; Then based on the initial building mask Get the initial building segmentation pseudo label, the expression is: in, is the initial building mask Middle pixel position The corresponding value, Pseudo-labeling for initial building segmentation Middle pixel position The corresponding value, is the binarization threshold.

6. The remote sensing image building semantic segmentation system based on the visual language model according to claim 5 is characterized in that: The process of the pseudo-label screening unit based on uncertainty quantification is as follows: Calculate the pixel-level entropy value of the initial building mask using the following formula: in, is the pixel location The corresponding pixel-level entropy value; For pixel-level entropy values ​​higher than the entropy threshold For pixel areas with , the pseudo label confidence is lowered or marked as to be corrected; The process of pseudo label screening unit based on cross-validation consistency is as follows: Use different initialized alignment hint encoders to generate multiple sets of pseudo labels for the same remote sensing image, calculate the pixel-level intersection-and-union ratio, and if the average pixel-level intersection-and-union ratio of multiple sets of pseudo labels in the target area is lower than the intersection-and-union ratio threshold , then the pseudo label of the target area is removed from the training set. The process of pseudo-label screening unit based on geometric constraint filtering is as follows: Based on the geometric prior of buildings in remote sensing images, pseudo-label areas that do not meet the preset shape characteristics are eliminated.

7. The system for semantic segmentation of buildings in remote sensing images based on a visual language model according to claim 6, characterized in that: The model iterative training unit satisfies the following formula: ; in, is the total loss function of the remote sensing image building semantic segmentation system, is the weighted cross entropy loss function, is the contrastive learning loss function, is the balance coefficient, For pre-training dataset The number of samples in is the pseudo label quality weight, is the cross entropy function, is the true label.

8. The system for semantic segmentation of buildings in remote sensing images based on a visual language model according to claim 7, characterized in that: The process of dynamic parameter adjustment unit using threshold adaptive update to adjust dynamic parameters is as follows: Dynamically adjust the binarization threshold based on the system's precision-recall curve on the validation set during training. , when precision increases and recall decreases, reduce ; When precision decreases and recall increases, increase , to balance segmentation accuracy and completeness; The dynamic parameter adjustment unit uses query text weight redistribution to adjust dynamic parameters. The update formula is: in, is an exponential decay factor.

Images