Point supervision semantic segmentation model construction and segmentation based on sam and cnn feature fusion

Abstract

The application discloses a point supervision semantic segmentation model construction and segmentation method based on SAM and CNN feature fusion, comprising the following steps: step one, acquiring remote sensing image data as a training set, and performing point labeling on target ground objects in the remote sensing image data to obtain a point label set, wherein the point label set is the category and position of the target ground objects; a point supervision semantic segmentation model is constructed; object structure information generated by using a convolutional neural network branch and a SAM driven fusion branch is used to realize collaborative modeling of image semantic information and structure information in combination with a cross-level fusion module CFM; meanwhile, under the condition of point supervision, the model can enhance the integrity control of the target region, improve the boundary positioning accuracy, and reduce the category confusion phenomenon, thereby significantly improving the effect of remote sensing image semantic segmentation. The technical problem that the existing point supervision semantic segmentation method is difficult to fully utilize image structure information and semantic information under the condition of sparse supervision information is solved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing images and relates to a semantic segmentation method for remote sensing images. Specifically, it is a point-supervised semantic segmentation model construction and segmentation method based on the fusion of SAM and CNN features. Background Technology

[0002] Semantic segmentation of remote sensing images is a technique for semantically labeling each pixel in a remote sensing image. It has significant application value in fields such as land cover mapping, urban information extraction, and resource and environmental monitoring. Current technologies typically employ deep learning-based semantic segmentation models for pixel-level classification of remote sensing images. End-to-end training enables direct mapping from images to semantic labels, thereby improving the automation and segmentation accuracy of remote sensing image interpretation.

[0003] However, deep learning-based methods typically rely on large amounts of pixel-level finely labeled data for model training. Acquiring high-quality pixel-level annotations is time-consuming, labor-intensive, and costly, making them difficult to implement in large-scale remote sensing data applications. To reduce annotation costs, existing technologies have proposed point-supervised semantic segmentation methods, which train segmentation models by annotating only a small number of pixels in the image as supervisory information.

[0004] In point-supervised semantic segmentation methods, the supervision signal is sparse because the labeled information is provided only in the form of a small number of pixels, making it difficult to effectively constrain the regions of the entire image. To alleviate this problem, existing techniques typically employ various methods to enhance the model's training signal. For example, one type of method introduces a feature regularization mechanism to impose consistency constraints on samples of the same class in the feature space, thereby enhancing the model's semantic discrimination ability for unlabeled regions. Another type of method uses a region expansion strategy, using point labels as seeds and combining pixel similarity, neighborhood consistency, or superpixel segmentation to gradually expand point-level labels into region-level supervision information, thus expanding the effective supervision range. In addition, there are methods that construct multi-model collaborative learning mechanisms, utilizing the prediction consistency and pseudo-label interaction updates between multiple models or multi-branch networks to collaboratively infer unlabeled regions, thereby improving overall segmentation performance.

[0005] Previous point-supervised learning still has certain limitations in practical applications. On the one hand, feature regularization methods mainly rely on statistical constraints in the feature space, making it difficult to fully utilize the structural information in the image and easily leading to incomplete representation of the target region. On the other hand, region expansion methods are prone to error propagation when the annotation is extremely sparse, further amplifying the noise supervision. In addition, multi-model collaborative learning methods usually rely on the prediction consistency between models for optimization. When there is a bias in the initial prediction, error accumulation is likely to occur, thus affecting the accuracy of the final segmentation result.

[0006] The root cause of these shortcomings lies in the model's insufficient representation learning ability under sparse supervision. On the one hand, existing methods lack an effective multi-scale joint modeling mechanism during feature learning, making it difficult to simultaneously capture fine-grained structural information and regional semantic context of ground features, thus failing to fully perceive the overall structural morphology and spatial continuity of ground features. On the other hand, point-level supervision provides only a very small number of pixel-level semantic constraints, making it difficult to effectively guide the complete structure and spatial extent of the target region, causing the model to rely more on shallow features such as local texture or color during training. In this situation, the model's ability to express the structural information of ground features is limited.

[0007] Therefore, under point-supervised conditions, how to fully extract structural and contextual semantic information from images, improve the model's ability to represent the structural features of ground objects, and thus generate more accurate and complete segmentation results has become an urgent technical problem to be solved. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a point-supervised semantic segmentation model construction and segmentation method based on SAM and CNN feature fusion, in order to solve the technical problem that existing point-supervised semantic segmentation methods cannot fully utilize image structural and semantic information under conditions of sparse supervision information.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0010] A method for constructing a point-supervised semantic segmentation model based on SAM and CNN feature fusion includes the following steps: Step 1: Acquire remote sensing image data as the training set, and label the target features in the remote sensing image data to obtain a point label set. The point label set The category and location of the target features; Step 2: Construct a point-supervised semantic segmentation model; The point-supervised semantic segmentation model includes a convolutional neural network branch and a SAM-driven fusion branch; The convolutional neural network branch includes a five-layer EfficientNet-B3 network. The first four layers of the EfficientNet-B3 network are connected to cross-level fusion modules (CFM), and the fifth layer of the EfficientNet-B3 network is connected to the fourth cross-level fusion module (CFM). The four cross-level fusion modules (CFM) are connected to a local-global decoder. The first two outputs of the local-global decoder are connected to a pixel adaptive processing module (PAR) and a reliable dynamic mixing module. The SAM-driven fusion branch comprises a five-layer EfficientViT-L0 network. The last four layers of the EfficientViT-L0 network are each connected to a bilateral fusion module (BIFFM), and the middle three layers of the BIFFM are each connected to an attention module. Each attention module is sequentially connected to an attention gate (ATG) and an attention residual block (ATR). The attention residual block (ATR) of the first-layer attention module is also connected to three... 3 convolutional layers; the fifth layer of the EfficientViT-L0 network is connected to the third layer attention module; the four bilateral fusion modules (BIFFM) are respectively connected to the last four layers of the EfficientNet-B3 network; The EfficientNet-B3 network is used for feature extraction from remote sensing image data, obtaining multi-scale convolutional features at multiple stages. Five stages of multi-scale convolutional features are selected. , Indicates the feature extraction stage. ; The Cross-Level Fusion (CFM) module is used to process multi-scale convolutional features at five different stages. By merging, cascade fusion characteristics are obtained. ; Local-global decoders are used to process the resulting cascaded fused features. Enhancement was performed to obtain multiple convolutional branch segmentation maps. ; The pixel adaptive processing module PAR is used to segment the obtained convolutional branch map. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement processing is performed to obtain pseudo-tags. ; The EfficientViT network extracts features from remote sensing image data to obtain global semantic features. global semantic features and 5-stage multi-scale convolutional features The graph is fused sequentially through the bilateral fusion module BIFFM, the attention gate ATG, and the attention residual block ATR to obtain the main output prediction map. That is, to obtain the semantic segmentation probability map of the remote sensing image; Step 3: Using the training set obtained in Step 1 as input, and the point label set... As a constraint, the semantic segmentation probability map of the remote sensing image is used as the output, and the loss function is used. The point-supervised semantic segmentation model obtained in step two is trained to obtain a trained point-supervised semantic segmentation model.

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] in: These are the weights of the loss function; H represents the cross-entropy loss function; Dice represents the Dice loss function; Weights for dynamic knowledge crossover loss; J represents the number of training iterations; KL represents the Kullback-Leibler distance loss function; Represents the convolution branch segmentation graph The softening probability map obtained after softening.

[0019] This invention also includes the following technical features: In step two, the cascade fusion characteristics are obtained. The process specifically includes the following steps: Step Q1, for high-level features Perform upsampling to enhance high-level features Spatial resolution and low-level features same; Step Q2 involves processing low-level features. and upsampled high-level features Perform 3 3. Convolutional transformation to obtain preliminary low-level transformation features. and high-level initial transformation characteristics ;

[0020]

[0021] Where v represents the upsampling operation; Step Q3 involves performing preliminary transformations on the lower-level features. and high-level initial transformation characteristics The feature responses are calculated separately and then normalized using the Sigmoid function to obtain the low-level enhancement features. and high-level enhanced features ;

[0022]

[0023] in: S represents the Sigmoid function: Indicates multiplication operation; Step Q4, enhance the low-level features and high-level enhanced features By splicing and merging, cascaded features are obtained. ;

[0024] in: Indicates a splicing operation; Represents the basic convolutional unit, including 3 3. Convolutional layers, normalization layers, and nonlinear activation functions; Step Q5, cascade features obtained in step Q4 The initial transformation features of the lower layer are obtained in step Q2. and high-level initial transformation characteristics Feature fusion is performed to obtain cascaded fused features. ;

[0025] in: This indicates element-wise addition.

[0026] In step two, multiple convolutional branch segmentation diagrams are generated. The process specifically includes the following steps: Step P1, cascading fusion features Perform concatenation and connection to generate a sequence containing global features. For global features Perform feature mapping and generate a global weight map using the Sigmoid function. ;

[0027]

[0028] in: Indicates having 1 The AdaptiveAvgPool2d function outputs 1 unit of space; Step P2, cascade fusion features The input is fed into the local attention module for progressive enhancement, and the enhanced features of each layer are calculated in descending order of quality. :

[0029] in: A( () indicates local attention; This indicates a channel-dimensional splicing operation; Step P3, using global weights Features Enhancement, after enhancement and Perform residual connections and convolutions to obtain multiple convolutional branch segmentation maps. ; .

[0030] In step two, pseudo-labels are obtained. The process specifically includes the following steps: Step M1: The pixel adaptive processing module PAR is used to segment the convolutional branch image. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement is performed to obtain the enhanced probability map No. 1. And the second probability diagram and the corresponding semantic segmentation prediction results and semantic segmentation prediction results ;

[0031]

[0032] argmax represents the category index corresponding to the largest numerical component in a given predicted probability vector.

[0033] Step M2: Obtain probability map number one. And the second probability diagram maximum probability and the probability of The joint entropy is calculated using the following formula to obtain the first pixel-level uncertainty. And the second pixel-level uncertainty ;

[0034] Step M3, the trusted dynamic mixing module, based on the uncertainty at the first pixel level... And the second pixel-level uncertainty Identify pseudo-labels ;

[0035] in: Represents probability diagram number one The maximum probability; Probability Diagram No. 2 The maximum probability.

[0036] A point-supervised semantic segmentation method based on SAM and CNN feature fusion includes the following steps: S1, acquire remote sensing image data as the actual measurement dataset; S2, input the measured dataset into the trained point-supervised semantic segmentation model obtained by the point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion, and obtain the semantic segmentation probability map of the remote sensing image; S3. Perform argmax processing on the semantic segmentation probability map of the remote sensing image obtained in S2 to obtain the semantic segmentation image.

[0037] Compared with the prior art, the beneficial technical effects of this invention are: (I) This invention utilizes object structural information generated by convolutional neural network branches and SAM-driven fusion branches, combined with a cross-level fusion module (CFM), to achieve collaborative modeling of image semantic and structural information. Simultaneously, under point-supervised conditions, it enhances the model's control over the integrity of the target region, improves boundary localization accuracy, and reduces category confusion, thereby significantly improving the semantic segmentation effect of remote sensing images. This method overcomes the problems of target loss and boundary ambiguity caused by sparse supervision information in existing point-supervised techniques, improving both the precision and overall accuracy of the segmentation results. It solves the technical problem that existing point-supervised semantic segmentation methods struggle to fully utilize image structural and semantic information under sparse supervision conditions.

[0038] (II) The convolutional neural network branch integrates multi-scale features based on the cross-level fusion module (CFM) and uses a local-global decoder for context association. This approach can effectively fuse multi-scale semantics and enhance the convolutional feature representation under point supervision, thereby improving the quality of pseudo-label generation.

[0039] (III) The features generated by the convolutional neural network branch and the SAM-driven fusion branch enable the model to effectively perceive the overall structure and spatial distribution of the target region under limited supervision, thereby significantly improving the model accuracy. At the same time, it improves the overall accuracy of semantic segmentation of remote sensing images while maintaining low annotation costs. It also improves the problems of incomplete target regions, inaccurate boundary positioning, and easy category confusion that are common in existing point supervision methods, thus improving the segmentation results in terms of target integrity, boundary fineness, and spatial continuity. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the point-supervised semantic segmentation model in this invention; Figure 2 This is a structural diagram of the Cross-Level Fusion Module (CFM). Figure 3 This is a schematic diagram of the global-local decoder structure; Figure 4 for Figure 1 A schematic diagram of the structure of some modules of the SAM-driven fusion branch; Figure 5 This is a diagram showing the effect of the Cross-Level Fusion Module (CFM) in this invention on the Vaihingen dataset; Figure 6 This is a comparison chart of the performance of this invention and other point supervision methods on the Vaihingen dataset; Figure 7 This is a comparison chart of the performance of the present invention and other point supervision methods on the Zurich Summer dataset.

[0041] The specific content of the present invention will be further explained in detail below with reference to the embodiments. Detailed Implementation

[0042] It should be noted that, unless otherwise specified, all components in this invention are those known in the art.

[0043] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.

[0044] This invention proposes a point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion, comprising the following steps: A method for constructing a point-supervised semantic segmentation model based on SAM and CNN feature fusion includes the following steps: Step 1: Acquire remote sensing image data as the training set, and label the target features in the remote sensing image data to obtain a point label set. dot tag set The category and location of the target features; Step 2: Construct a point-supervised semantic segmentation model; The point-supervised semantic segmentation model includes a convolutional neural network branch and a SAM-driven fusion branch; The convolutional neural network branch consists of a five-layer EfficientNet-B3 network. The first four layers of the EfficientNet-B3 network are connected to the cross-level fusion module (CFM), and the fifth layer of the EfficientNet-B3 network is connected to the fourth cross-level fusion module (CFM). The four cross-level fusion modules (CFM) are connected to the local-global decoder. The first two outputs of the local-global decoder are connected to the pixel adaptive processing module (PAR) and the reliable dynamic mixing module. The SAM-driven fusion branch consists of a five-layer EfficientViT-L0 network. The last four layers of the EfficientViT-L0 network are each connected to a bilateral fusion module (BIFFM), and the middle three layers of the BIFFM are each connected to an attention module. Each attention module is sequentially connected to an attention gate (ATG) and an attention residual block (ATR). The attention residual block (ATR) of the first-layer attention module is also connected to three... 3 convolutional layers; the fifth layer of the EfficientViT-L0 network is connected to the third layer attention module; four bilateral fusion modules (BIFFM) are connected to the last four layers of the EfficientNet-B3 network respectively; The EfficientNet-B3 network is used for feature extraction from remote sensing image data, obtaining multi-scale convolutional features at multiple stages. Five stages of multi-scale convolutional features are selected. , Indicates the feature extraction stage. ; The Cross-Level Fusion (CFM) module is used to process multi-scale convolutional features at five different stages. By merging, cascade fusion characteristics are obtained. ; Local-global decoders are used to process the resulting cascaded fused features. Enhancement was performed to obtain multiple convolutional branch segmentation maps. ; The pixel adaptive processing module PAR is used to segment the obtained convolutional branch map. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement processing is performed to obtain pseudo-tags. ; The EfficientViT network extracts features from remote sensing image data to obtain global semantic features. global semantic features and 5-stage multi-scale convolutional features The graph is fused sequentially through the bilateral fusion module BIFFM, the attention gate ATG, and the attention residual block ATR to obtain the main output prediction map. That is, to obtain the semantic segmentation probability map of the remote sensing image; Step 3: Using the training set obtained in Step 1 as input, and the point label set... As a constraint, the semantic segmentation probability map of the remote sensing image is used as the output, and the loss function is used. The point-supervised semantic segmentation model obtained in step two is trained to obtain a trained point-supervised semantic segmentation model.

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] in: These are the weights of the loss function; H represents the cross-entropy loss function; Dice represents the Dice loss function; Weights for dynamic knowledge crossover loss; J represents the number of training iterations; KL represents the Kullback-Leibler distance loss function; Represents the convolution branch segmentation graph The softening probability map obtained after softening.

[0053] See Figure 1The point-supervised semantic segmentation model in this scheme includes a convolutional neural network branch and a SAM-driven fusion branch. The convolutional neural network branch performs multi-level cross-sensory feature fusion to extract and fuse features from the input image. Under point supervision, it combines contextual information to enhance the model's feature representation ability and generate high-confidence pseudo-labels. The SAM-driven fusion branch is used to introduce structured prior information, fusing the local information perception ability of the convolutional neural network with the ability of SAM to capture target structural information. This enables the point-supervised learning method to simultaneously acquire local detail information, structural relationship information, and long-distance dependency information of the image.

[0054] By utilizing object structural information generated by convolutional neural network branches and SAM-driven fusion branches, and combining it with the cross-level fusion module CFM, collaborative modeling of image semantic and structural information is achieved. Simultaneously, under point supervision, this method enhances the model's control over the integrity of target regions, improves boundary localization accuracy, and reduces class confusion, thereby significantly improving the performance of remote sensing image semantic segmentation. This method overcomes the problems of target loss and boundary blurring caused by sparse supervision information in existing point-supervised techniques, improving both the fineness and overall accuracy of the segmentation results. It also solves the technical problem that existing point-supervised semantic segmentation methods struggle to fully utilize image structural and semantic information under sparse supervision conditions.

[0055] The convolutional neural network branch integrates multi-scale features based on the cross-level fusion module (CFM) and uses a local-global decoder for context association. This approach can effectively fuse multi-scale semantics and enhance convolutional feature representation under point supervision, thereby improving the quality of pseudo-label generation.

[0056] By generating features through convolutional neural network branches and SAM-driven fusion branches, the model can still effectively perceive the overall structure and spatial distribution of the target region under limited supervision, thereby significantly improving the model accuracy. At the same time, it improves the overall accuracy of semantic segmentation of remote sensing images while maintaining low annotation costs. It also improves the problems of incomplete target regions, inaccurate boundary localization, and easy category confusion that are common in existing point-supervised methods, thus improving the segmentation results in terms of target integrity, boundary fineness, and spatial continuity.

[0057] In this example, for a remote sensing image with an input size of 512×512×3, preliminary feature extraction is first performed using a 3×3 convolution (Conv3×3+BN+Swish), and the feature map is downsampled to 256×256×40. Subsequently, the network consists of multiple stages of MBConv modules. The first stage uses MBConv1 (3×3), outputting a feature size of 256×256×24; the second stage uses MBConv6 (3×3) and downsamples to output a feature size of 128×128×32; the third stage uses MBConv6 (5×5), outputting a feature size of 64×64×48; the fourth stage uses MBConv6 (3×3), outputting a feature size of 32×32×96; the fifth stage uses MBConv6 (5×5) stacked multiple times to further extract high-level semantic features, maintaining a size of 32×32×136; the sixth stage continues to use MBConv6 (5×5) and downsamples to 16×16×232. This example uses five convolutional features from the second, third, fourth, fifth, and sixth stages to obtain... When i=0, When the feature dimension is 32 and i=1, When the feature dimension is 48 and i=2, When the feature dimension is 96 and i=3, When the feature dimension is 136 and i=4, The feature dimension is 232.

[0058] See Figure 4 The SAM-driven fusion branch includes a four-layer EfficientViT-L0 network, a bilateral fusion module (BIFFM), an attention residual block (ATR), and an attention gate (ATG). The BIFFM module processes SAM and CNN features in parallel, extracting statistical information from different dimensions using average pooling and max pooling, and adaptively adjusting feature weights by combining depthwise separable convolution (DSConv) and the sigmoid activation function. Through feature concatenation (Cat) and cross-multiplication operations, it achieves deep aggregation of semantic features in both spatial and channel dimensions. The attention residual block (ATR) introduces a residual connection structure, utilizing the 3D receptive fields of different receptive fields... The third convolutional layer combines batch normalization (BN) and ReLU activation functions to further enhance the expressive power of local features, and optimizes the saliency of feature representation through the channel space squeezing excitation (cSe) mechanism. The Attention Gate (ATG) module, to capture contextual information over a wide area, employs multiple sets of depthwise separable convolutions with different dilation rates to expand the receptive field without increasing computational cost, effectively aligning and fusing multi-level features from the encoder and decoder.

[0059] To further enhance the performance of the EfficientNet-B3 network, feedback information from multi-level and multi-scale features learned from the hybrid branch is provided to the remaining branches of the EfficientNet-B3 network. This improves the overall understanding of complex scenes by the convolutional feature cross-sensing branch, encouraging the convolutional neural network branches to generate higher-quality pseudo-labels. The EfficientViT-L0 network is used as the feature extraction backbone for the image. To fully utilize the prior knowledge in SAM, the encoder weights are fixed after loading the pre-trained weights and do not participate in parameter updates.

[0060] This scheme sequentially performs the above fusion operation on the convolutional features and SAM features of the five stages. Before the output layer, the fusion branch gathers the highest-dimensional deep semantic features from the four stages, fully integrating the multi-scale semantic information of the SAM encoder and the CNN branch, and finally inputs it into the 3D output layer. Three convolutional layers perform feature mapping to generate the main output prediction map. .

[0061] in, Set to respectively ; This invention also includes the following technical features: In step two, the cascade fusion characteristics are obtained. The process specifically includes the following steps: Step Q1, for high-level features Perform upsampling to enhance high-level features Spatial resolution and low-level features same; Step Q2 involves processing low-level features. and upsampled high-level features Perform 3 3. Convolutional transformation to obtain preliminary low-level transformation features. and high-level initial transformation characteristics ;

[0062]

[0063] Where v represents the upsampling operation; Step Q3 involves performing preliminary transformations on the lower-level features. and high-level initial transformation characteristics The feature responses are calculated separately and then normalized using the Sigmoid function to obtain the low-level enhancement features. and high-level enhanced features ;

[0064]

[0065] in: S represents the Sigmoid function: Indicates multiplication operation; Step Q4, enhance the low-level features and high-level enhanced features By splicing and merging, cascaded features are obtained. ;

[0066] in: Indicates a splicing operation; Represents the basic convolutional unit, including 3 3. Convolutional layers, normalization layers, and nonlinear activation functions; Step Q5, cascade features obtained in step Q4 The initial transformation features of the lower layer are obtained in step Q2. and high-level initial transformation characteristics Feature fusion is performed to obtain cascaded fused features. ;

[0067] in: This indicates element-wise addition.

[0068] See Figure 2 Under point-supervised conditions, the semantic information provided by point annotations is relatively limited, and convolutional neural network branches lack sufficient structural information constraints during feature learning, easily leading to deficiencies in spatial integrity and contextual consistency of the obtained features. To address this, this invention incorporates a cross-level fusion module (CFM) into the convolutional neural network branches to enhance feature representation capabilities. The CFM module improves contextual information aggregation and semantic consistency by realizing information interaction and fusion between multi-level features, and enhances boundary awareness and spatial consistency by combining low-level structural information, thereby providing a more stable feature foundation for pseudo-label generation. As shown in the figure, the CFM module receives features from adjacent layers. With current layer features As input, feature information is extracted through convolution operations, and weight coefficients are generated using the Sigmoid function to modulate the features element-wise. Subsequently, multi-level feature fusion is achieved through concatenation, addition, and convolution operations, ultimately yielding the enhanced features. .

[0069] In the above technical solution, step Q3 is used to construct feature-level spatial attention, thereby realizing the interaction enhancement between adjacent layer features. In step Q5, feature fusion is performed to obtain cascaded fused features, which are used to realize the joint modeling of high-level semantic information and low-level spatial detail information.

[0070] In step two, multiple convolutional branch segmentation diagrams are generated. The process specifically includes the following steps: Step P1, cascading fusion features Perform concatenation and connection to generate a sequence containing global features. For global features Perform feature mapping and generate a global weight map using the Sigmoid function. ;

[0071]

[0072] in: Indicates having 1 The AdaptiveAvgPool2d function outputs 1 unit of space; Step P2, cascade fusion features The input is fed into the local attention module for progressive enhancement, and the enhanced features of each layer are calculated in descending order of quality. :

[0073] in: A( () indicates local attention; This indicates a channel-dimensional splicing operation; Step P3, using global weights Features Enhancement, after enhancement and Perform residual connections and convolutions to obtain multiple convolutional branch segmentation maps. ; .

[0074] See Figure 3 Even with the introduction of a cross-level fusion module (CFM) for multi-scale feature fusion in the convolutional neural network branches, the model's ability to distinguish similar targets may still be unstable when relying solely on point label supervision. To further explore enhanced features... Complementary information in the convolutional neural network is used to set up a local-global decoder in the convolutional neural network branch. The local-global decoder is used to simultaneously model the global contextual dependencies between each feature layer and the local correlations between adjacent decoding layers, thereby improving the feature representation capability. For example Figure 3 As shown, the local-global decoder includes a two-level feature aggregation structure based on key features, used to achieve joint aggregation and fusion of multi-layer feature information. Figure 3 As shown, the local-global decoder includes a global attention branch and a local attention branch. The global attention branch uses high-level features to generate a global weight map. The features are weighted to enhance global contextual information; the local attention branch models the local correlations between adjacent decoding layers through convolution and normalization operations. Subsequently, multi-scale output is obtained through feature concatenation and step-by-step decoding. to This is to achieve the fusion and enhancement of multi-layer features.

[0075] In step two, pseudo-labels are obtained. The process specifically includes the following steps: Step M1: The pixel adaptive processing module PAR is used to segment the convolutional branch image. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement is performed to obtain the enhanced probability map No. 1. And the second probability diagram and the corresponding semantic segmentation prediction results and semantic segmentation prediction results ;

[0076]

[0077] argmax represents the category index corresponding to the largest numerical component in a given predicted probability vector.

[0078] Step M2: Obtain probability map number one. And the second probability diagram maximum probability and the probability of The joint entropy is calculated using the following formula to obtain the first pixel-level uncertainty. And the second pixel-level uncertainty ;

[0079] Step M3, the trusted dynamic mixing module, based on the uncertainty at the first pixel level... And the second pixel-level uncertainty Identify pseudo-labels ;

[0080] in: Represents the first probability diagram The maximum probability; Probability Diagram No. 2 The maximum probability.

[0081] This invention also provides a point-supervised semantic segmentation method based on the fusion of SAM and CNN features, comprising the following steps: S1, acquire remote sensing image data as the actual measurement dataset; S2, input the measured dataset into the trained point-supervised semantic segmentation model obtained by the point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion, and obtain the semantic segmentation probability map of the remote sensing image; S3. Perform argmax processing on the semantic segmentation probability map of the remote sensing image obtained in S2 to obtain the semantic segmentation image.

[0082] Example: This embodiment is used to verify the effect of the proposed cross-level fusion module CFM on the Vaihingen dataset, such as... Figure 5 As shown in the figure, the feature response of the cross-level fusion module (CFM) at different stages is illustrated. (a) is the feature map extracted during the encoding stage of the EfficientNet-B3 network; (b) is the feature map enhanced by the CFM module; and (c) is the feature map further processed by the local-global decoder. The figure shows that the original encoded features gradually exhibit enhanced semantic response but weakened structural information at higher levels. After processing by the CFM module, the structural and semantic information in each layer of features is effectively fused, resulting in a clearer response and more complete boundary structure in the target region. The introduction of the local-global decoder further strengthens the contextual association between multiple layers of features, making the distinction between the target region and the background region more obvious. These results demonstrate that introducing the CFM module for multi-scale feature interaction and fusion can effectively improve feature representation capabilities and provide a more reliable feature foundation for subsequent pseudo-label generation and target recognition.

[0083] Comparative Example 1: This comparative example is used to compare the performance of the proposed method with other point-supervised methods on the Vaihingen dataset, as shown in Table 1 and... Figure 6The figure shows a comparison of the semantic segmentation performance of our proposed method with several existing methods on the Vaihingen dataset. The columns represent the input image, ground truth, and classification results of different methods, with the last column showing the classification results of our proposed method. As can be seen from the figure, compared to methods such as AGMM, CGNet, DBNet, dcBR, Festa, MambaVSeg, and SparseFormer, our proposed method exhibits better classification consistency and spatial integrity in the recognition of categories such as buildings, low vegetation, trees, and vehicles. Especially in complex scenes and target boundary regions, our proposed method can more accurately preserve the target's structural morphology and effectively reduce misclassification and boundary blurring, thus obtaining classification results that are closer to the ground truth.

[0084] Table 1. Accuracy results of the proposed method on the Vaihingen dataset.

[0085] Comparative Example 2: This comparative example is used to compare the performance of the proposed method with other point-supervised methods on the Zurich-Summer dataset, as shown in Table 2 and... Figure 7 The figure shows a comparison of the semantic segmentation results of the proposed method with those of various existing methods on Zurich Summer data. The columns represent the input image, ground truth, and classification results of different methods, with the last column showing the classification results of the proposed method. As can be seen from the figure, compared with methods such as AGMM, CGNet, DBNet, dcBR, Festa, MambaVSeg, and SparseFormer, the proposed method generates more continuous and clear boundaries in areas with obvious structural features such as roads and buildings; in areas with natural features such as trees, grasslands, and water bodies, the regions are more complete and the category distribution is more reasonable. Furthermore, in areas with overlapping categories and complex structures, the proposed method effectively reduces category confusion, making the spatial structure representation of various features more stable. This demonstrates that the proposed method achieves better classification results on this dataset.

[0086] Table 2. Accuracy results of the proposed method on the Zurich-Summer dataset.

Claims

1. A method for constructing a point-supervised semantic segmentation model based on SAM and CNN feature fusion, characterized in that, Includes the following steps: Step 1: Acquire remote sensing image data as the training set, and label the target features in the remote sensing image data to obtain a point label set. The point label set The category and location of the target features; Step 2: Construct a point-supervised semantic segmentation model; The point-supervised semantic segmentation model includes a convolutional neural network branch and a SAM-driven fusion branch; The convolutional neural network branch includes a five-layer EfficientNet-B3 network. The first four layers of the EfficientNet-B3 network are connected to cross-level fusion modules (CFM), and the fifth layer of the EfficientNet-B3 network is connected to the fourth cross-level fusion module (CFM). The four cross-level fusion modules (CFM) are connected to a local-global decoder. The first two outputs of the local-global decoder are connected to a pixel adaptive processing module (PAR) and a reliable dynamic mixing module. The SAM-driven fusion branch comprises a five-layer EfficientViT-L0 network. The last four layers of the EfficientViT-L0 network are each connected to a bilateral fusion module (BIFFM), and the middle three layers of the BIFFM are each connected to an attention module. Each attention module is sequentially connected to an attention gate (ATG) and an attention residual block (ATR). The attention residual block (ATR) of the first-layer attention module is also connected to three... 3 convolutional layers; the fifth layer of the EfficientViT-L0 network is connected to the third layer attention module; the four bilateral fusion modules (BIFFM) are respectively connected to the last four layers of the EfficientNet-B3 network; The EfficientNet-B3 network is used for feature extraction from remote sensing image data, obtaining multi-scale convolutional features at multiple stages. Five stages of multi-scale convolutional features are selected. , Indicates the feature extraction stage. ; The Cross-Level Fusion (CFM) module is used to process multi-scale convolutional features at five different stages. By merging, cascade fusion characteristics are obtained. ; Local-global decoders are used to process the resulting cascaded fused features. Enhancement was performed to obtain multiple convolutional branch segmentation maps. ; The pixel adaptive processing module PAR is used to segment the obtained convolutional branch map. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement processing is performed to obtain pseudo-tags. ; The EfficientViT network extracts features from remote sensing image data to obtain global semantic features. global semantic features and 5-stage multi-scale convolutional features The graph is fused sequentially through the bilateral fusion module BIFFM, the attention gate ATG, and the attention residual block ATR to obtain the main output prediction map. That is, to obtain the semantic segmentation probability map of the remote sensing image; Step 3: Using the training set obtained in Step 1 as input, and the point label set... As a constraint, the semantic segmentation probability map of the remote sensing image is used as the output, and the loss function is used. The point-supervised semantic segmentation model obtained in step two is trained to obtain a trained point-supervised semantic segmentation model. in: These are the weights of the loss function; H represents the cross-entropy loss function; Dice represents the Dice loss function; Weights for dynamic knowledge crossover loss; J represents the number of training iterations; KL represents the Kullback-Leibler distance loss function; Represents the convolution branch segmentation graph The softening probability map obtained after softening.

2. The point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion as described in claim 1, characterized in that, In step two, the cascade fusion characteristics are obtained. The process specifically includes the following steps: Step Q1, for high-level features Perform upsampling to enhance high-level features Spatial resolution and low-level features same; Step Q2 involves processing low-level features respectively. and upsampled high-level features Perform 3 3. Convolutional transformation to obtain preliminary low-level transformation features. and high-level initial transformation characteristics ; Where v represents the upsampling operation; Step Q3 involves performing preliminary transformations on the lower-level features. and high-level initial transformation characteristics The feature responses are calculated separately and then normalized using the Sigmoid function to obtain the low-level enhancement features. and high-level enhanced features ; in: S represents the Sigmoid function: Indicates multiplication operation; Step Q4, enhance the low-level features and high-level enhanced features By splicing and merging, cascaded features are obtained. ; in: Indicates a splicing operation; Represents the basic convolutional unit, including 3 3. Convolutional layers, normalization layers, and nonlinear activation functions; Step Q5, cascade features obtained in step Q4 The initial transformation features of the lower layer are obtained in step Q2. and high-level initial transformation characteristics Feature fusion is performed to obtain cascaded fused features. ; in: This indicates element-wise addition.

3. The point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion as described in claim 1, characterized in that, In step two, multiple convolutional branch segmentation diagrams are generated. The process specifically includes the following steps: Step P1, cascading fusion features Perform concatenation and connection to generate a sequence containing global features. For global features Perform feature mapping and generate a global weight map using the Sigmoid function. ; in: Indicates having 1 The AdaptiveAvgPool2d function outputs 1 unit of space; Step P2, cascade fusion features The input is fed into the local attention module for progressive enhancement, and the enhanced features of each layer are calculated in descending order of quality. : in: A( ) represents local attention; This indicates a channel-dimensional splicing operation; Step P3, using global weights Features Enhancement, after enhancement and Perform residual connections and convolutions to obtain multiple convolutional branch segmentation maps. ; 。 4. The point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion as described in claim 1, characterized in that, In step two, pseudo-labels are obtained. The process specifically includes the following steps: Step M1: The pixel adaptive processing module PAR is used to segment the convolutional branch image. Partition diagram of convolution branch 1 in the diagram And the second convolution branch segmentation diagram Enhancement is performed to obtain the enhanced probability map #1. And the second probability diagram and the corresponding semantic segmentation prediction results and semantic segmentation prediction results ; argmax represents the category index corresponding to the largest numerical component in a given predicted probability vector. Step M2: Obtain probability map number one. And the second probability diagram maximum probability and the probability of The joint entropy is calculated using the following formula to obtain the first pixel-level uncertainty. And the second pixel-level uncertainty ; Step M3, the trusted dynamic mixing module, based on the uncertainty at the first pixel level... And the second pixel-level uncertainty Identify pseudo-labels ; in: Represents probability diagram number one The maximum probability; Probability Diagram No. 2 The maximum probability.

5. A point-supervised semantic segmentation method based on SAM and CNN feature fusion, characterized in that, Includes the following steps: S1, acquire remote sensing image data as the actual measurement dataset; S2, input the measured dataset into the trained point-supervised semantic segmentation model obtained by the point-supervised semantic segmentation model construction method based on SAM and CNN feature fusion as described in any one of claims 1 to 4, and obtain the semantic segmentation probability map of the remote sensing image; S3. Perform argmax processing on the semantic segmentation probability map of the remote sensing image obtained in S2 to obtain the semantic segmentation image.

Images