Remote sensing image segmentation method based on balanced fusion of RGB and NDVI index

Abstract

The application discloses a remote sensing image segmentation method based on balanced fusion of RGB and NDVI index, and relates to the technical field of remote sensing image interpretation. The method obtains remote sensing image data, calculates an NDVI original value graph and an NDVI gradient graph, and combines the NDVI original value graph and the NDVI gradient graph along a channel dimension to form an NDVI double-channel index input. After patch embedding of RGB optical images and the NDVI double-channel index input, in each stage of multiple encoding stages, the feature expression is enhanced through an index-aware bidirectional hint adaptation module, then cross-modal feature sampling, weighting and attention fusion are performed at key positions selected dynamically through a dynamic sparse cross-index fusion module, and finally, a semantic segmentation result is generated by a lightweight segmentation head. During training, the input of one mode is randomly masked, and correct areas in the bimodal fusion prediction are used to impose self-guided auxiliary supervision on the single-mode branch. The application realizes balanced fusion of RGB spatial texture and NDVI vegetation index information.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image interpretation technology, specifically to a remote sensing image segmentation method based on the balanced fusion of RGB and NDVI indices. Background Technology

[0002] The goal of remote sensing semantic segmentation is to assign a land cover category label to each pixel in a remote sensing image. Traditional techniques typically rely on methods such as spectral thresholding, texture features, morphological features, random forests, or support vector machines for land cover classification. With the development of deep learning, encoder-decoder networks such as U-Net, DeepLab, HRNet, SegFormer, and SwinTransformer have gradually become the mainstream solutions for remote sensing semantic segmentation. These methods can learn high-level semantics such as building edges, road morphology, water texture, and vegetation texture end-to-end. However, when using only RGB optical imagery, the model is susceptible to problems such as seasonal changes, shadow occlusion, sensor color differences, crop phenological changes, and the presence of different objects with the same or different spectral characteristics.

[0003] In existing technologies, one approach is semantic segmentation based on single-modal imagery. This type of approach typically uses RGB optical imagery or multispectral imagery as input, employs convolutional neural networks or Transformer networks for multi-scale feature extraction, and then obtains pixel-level classification results through dilated convolution, pyramid pooling, skip connections, or decoder upsampling. The advantages of this approach are its mature network structure, clear training process, and relatively simple deployment, enabling it to learn spatial morphology, texture boundaries, and contextual semantic information of ground features relatively well. However, in remote sensing scenarios, single-modal input is easily affected by imaging conditions and the spectral similarity of ground features. For example, relying solely on RGB optical imagery, categories such as bare land and rooftops, shadows and water bodies, and low vegetation and farmland roads may appear similar in visible light space, leading to category confusion. Relying solely on a single spectral index can easily result in the loss of fine-grained spatial textures such as building, road, and water body boundaries. Directly inputting all multispectral bands increases the costs of data storage, model computation, and cross-sensor adaptation, and places high demands on the band configuration, radiometric correction, and spatial registration consistency of different satellite sensors.

[0004] Another approach is semantic segmentation of remote sensing or natural scenes based on multimodal fusion. This type of approach typically inputs RGB optical imagery along with other auxiliary modalities into the network, such as RGB optical imagery combined with thermal infrared images, SAR images, near-infrared band images, or vegetation index images. Information complementarity is achieved through dual-stream encoders, attention mechanisms, gated fusion, channel swapping, feature-level addition or cascading, and Transformer cross-modal interaction. Compared to single-modal methods, multimodal methods can leverage the complementarity between different sensors or different feature spaces to improve the model's segmentation robustness in areas with varying illumination, shadow occlusion, low-texture regions, and complex mixed terrain features. In existing multimodal semantic segmentation research, some schemes have adopted different approaches. These include setting separate feature extraction branches for different modalities, allowing each modality to extract relatively independent low-level texture features and high-level semantic features during the encoding stage, and then fusing them at different scales. Other schemes utilize attention weights, channel weights, or spatial weights to adaptively adjust the contribution of different modalities to the segmentation results. Still others introduce Transformers or cross-modal interaction units to establish long-distance dependencies between features from different modalities, enhancing the model's ability to understand the context of complex scenes. Furthermore, some studies have attempted to reduce the computational cost of multimodal models by employing lightweight branches, shared trunks, or a small number of trainable parameters, thereby improving their deployment feasibility in practical remote sensing data processing platforms.

[0005] However, most of the aforementioned multimodal fusion methods are geared towards general auxiliary modal design. The network structure typically treats auxiliary modalities as two-dimensional image features similar to RGB optical imagery, focusing more on how to fuse different image features and less on specifically designing for the physical meaning, numerical range, saturation characteristics, and regional applicability of remote sensing vegetation indices. NDVI, calculated from near-infrared and red bands, reflects vegetation cover, growth status, and chlorophyll-related information, and is of significant value in identifying vegetation-related categories such as forests, cultivated land, grasslands, and wetlands. However, NDVI is not a typical image modality. Its numerical range, noise sources, vegetation threshold significance, saturation characteristics, and correspondence with land cover categories all differ from RGB optical imagery. For example, NDVI may saturate in high-density vegetation areas, exhibit negative or unstable responses in water bodies and shaded areas, and has limited ability to represent detail boundaries in urban building and road areas.

[0006] Therefore, simply using NDVI as the fourth channel and directly stitching it with RGB optical images, or simply adding, cascading, or weighting it with attention in the encoder and decoder, can easily lead to problems such as NDVI noise diffusion, blurred vegetation boundaries, and an imbalance between the contributions of RGB and NDVI. Existing remote sensing RGB and NDVI fusion semantic segmentation schemes also suffer from problems such as simple input stitching failing to fully utilize NDVI exponential information, unbalanced modal representation, the potential introduction of cross-regional noise through global cross-modal fusion, and the tendency for the training process to become overly dependent on a particular modality. Summary of the Invention

[0007] The purpose of this invention is to provide a remote sensing image segmentation method based on the balanced fusion of RGB and NDVI exponentials. This method solves the technical problems in existing remote sensing RGB and NDVI fusion semantic segmentation, such as NDVI noise diffusion, blurred vegetation boundaries, imbalance of RGB and NDVI contributions, and excessive dependence on a certain modality during training, caused by simple stitching or global fusion.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: Remote sensing image segmentation methods based on RGB and NDVI exponential fusion include: Acquire remote sensing image data of the area to be processed. The remote sensing image includes at least RGB optical image and near-infrared and red bands for calculating NDVI. Perform radiometric correction, atmospheric correction and geometric correction on the RGB optical image, near-infrared band and red band. Perform geometric transformation and label-preserving resampling consistent with the image on the semantically labeled map to achieve spatial registration. The NDVI raw numerical map is calculated based on the near-infrared band and the red band. The NDVI gradient map is calculated based on the NDVI raw numerical map. The NDVI raw numerical map and the NDVI gradient map are combined along the channel dimension to form the NDVI dual-channel exponential input. The RGB optical image and the NDVI dual-channel index input are respectively processed through an independent patch embedding layer to obtain RGB feature tags and NDVI feature tags; Within each encoding stage, RGB and NDVI features are processed sequentially through the exponentially-aware bidirectional cueing adaptation module and the dynamic sparse cross-exponential fusion module to obtain the fusion features corresponding to each encoding stage. The fusion features of each stage are then fed into the lightweight segmentation head to generate the final semantic segmentation result. During training, each training sample first undergoes a complete bimodal forward propagation to compute the bimodal fusion prediction and its correct region mask. Then, the RGB input or NDVI dual-channel exponential input is randomly occluded to perform a single-modal / missing modality forward propagation. The occlusion states include three cases: occluded RGB, occluded NDVI, and no occlusion, with a sampling probability of 1 / 3 for each case. The correct regions in the bimodal fusion prediction are used to apply self-guided auxiliary supervision to the RGB single-modal prediction and NDVI single-modal prediction. The network is trained using the main segmentation loss and the correct region self-guided auxiliary loss.

[0009] As a preferred technical solution of the present invention, the original NDVI numerical map is calculated based on the near-infrared band and the red band. The original NDVI numerical map is equal to the difference between the reflectance of the near-infrared band and the reflectance of the red band divided by the sum of the reflectance of the near-infrared band and the red band and the sum of the preset minimum constant. Based on the original NDVI numerical map, the NDVI gradient map is calculated. The Sobel operator or the first-order difference operator is used to calculate the gradient of the original NDVI numerical map in the horizontal and vertical directions, respectively. Then, the square root of the sum of the squares of the horizontal gradient and the squares of the vertical gradient is taken to obtain the NDVI gradient map. The original NDVI numerical map and the NDVI gradient map are combined along the channel dimension to form the NDVI dual-channel exponential input.

[0010] As a preferred embodiment of the present invention, the network input includes an RGB optical image with height H, width W, and 3 channels, and an NDVI dual-channel exponential input with height H, width W, and 2 channels. The two images are respectively processed through independent patch embedding layers to obtain RGB feature tags and NDVI feature tags. The independent patch embedding layers include an RGB patch embedding layer and an NDVI dual-channel exponential patch embedding layer. The network can be set to four encoding stages. Each encoding stage is processed according to the method of exponential-aware bidirectional cue adaptation, dynamic sparse cross-exponential fusion, and stage output. The fused features of each stage are sent to a lightweight segmentation head to generate the final semantic segmentation result.

[0011] As a preferred technical solution of the present invention, the exponentially-aware bidirectional prompt adaptation module treats RGB and NDVI as equal input modalities, converges their information in a unified low-rank space, and then generates dedicated prompts for RGB and NDVI respectively. For RGB features and NDVI features in an encoding stage, the RGB features are first reduced in dimensionality using RGB dimensionality reduction mapping parameters to obtain an RGB low-rank representation, and the NDVI features are reduced in dimensionality using NDVI dimensionality reduction mapping parameters to obtain an NDVI low-rank representation. The RGB low-rank representation and the NDVI low-rank representation are concatenated, and after bimodal convergence mapping and dimensionality increase mapping, a unified representation containing both RGB texture semantics and NDVI exponential semantics is obtained. The unified representation is subjected to a first linear transformation and a second linear transformation, and after the transformation, a scaling factor is added to the product of the unified representation to obtain RGB-specific prompts and NDVI-specific prompts. The first linear transformation has a learnable first weight and a first bias, the second linear transformation has a learnable second weight and a second bias, and the scaling factor is a learnable parameter or a preset fixed hyperparameter.

[0012] As a preferred technical solution of the present invention, the prompted features are fed into a Transformer encoding layer with a modality-specific bottleneck adapter. For each modality, the modality features are added to the corresponding modality prompt to obtain a first intermediate feature. The first intermediate feature is then subjected to layer normalization and multi-head self-attention processing, and then added to the first intermediate feature to obtain a second intermediate feature. The second intermediate feature is then dimensionality-reduced using the bottleneck dimensionality reduction parameters of the modality, and after passing through a nonlinear activation function, it is then dimensionality-upgraded using the bottleneck dimensionality-upgrade parameters of the modality to obtain the modality-specific bottleneck adapter output. The second intermediate feature is then subjected to layer normalization and feedforward network processing, and then added to the product of the second intermediate feature, the adapter scaling coefficient, and the modality-specific bottleneck adapter output to obtain the adapted modality features. The adapter scaling coefficient is a learnable parameter or a preset fixed hyperparameter. The pre-trained Transformer backbone parameters are kept frozen, and only the prompt parameters, bottleneck adapter parameters, and subsequent fusion parameters are trained.

[0013] As a preferred technical solution of the present invention, the dynamic sparse cross-exponential fusion module projects the RGB features and NDVI features output by the exponentially-aware bidirectional cue adaptation module onto the low-rank fusion space through low-rank projection parameters to obtain RGB low-rank features and NDVI low-rank features; constructs regular grid reference points, and predicts the sampling offsets of RGB and NDVI respectively through a lightweight offset prediction network; adds the regular grid reference points and the sampling offsets to obtain RGB dynamic sampling points and NDVI dynamic sampling points, wherein the offset prediction network consists of depthwise separable convolution, layer normalization, activation function and 1×1 convolution; merges the RGB dynamic sampling points and NDVI dynamic sampling points into a cross-modal candidate point set; on the cross-modal candidate point set, samples the RGB low-rank features and NDVI low-rank features respectively through bilinear sampling or differentiable sampling operations to obtain RGB key features and NDVI key features.

[0014] As a preferred technical solution of the present invention, a lightweight fusion network is used to concatenate RGB low-rank features and NDVI low-rank features, and then process them through convolution, normalization and activation functions to generate coarse fusion query features. The coarse fusion query features are sampled on the cross-modal candidate point set to obtain sampled coarse fusion query features. The sampled coarse fusion query features are processed through a modal contribution evaluation function, and the fusion weights of RGB and NDVI at the candidate points are obtained through a softmax function. The fusion weights are multiplied element-wise with the corresponding RGB key features and NDVI key features and summed to obtain the key features after dynamic sparse fusion. The key features after dynamic sparse fusion are projected as keys and values ​​in attention calculation, and multi-head attention calculation is performed using the coarse fusion query features as the query to obtain the fusion features output in the encoding stage. The modal contribution evaluation function consists of two layers of linear mapping and nonlinear activation.

[0015] As a preferred technical solution of the present invention, after multiple encoding stages, the network obtains fusion features at different scales. The fusion features of each stage are upsampled to a unified scale and spliced ​​or summed. Then, the final semantic probability map is output through a lightweight segmentation head. The maximum value of the category dimension of the final semantic probability map is taken to obtain the predicted category of each pixel.

[0016] As a preferred technical solution of the present invention, during the training process, in addition to outputting the bimodal fusion prediction, the network also sets auxiliary segmentation heads on the RGB branch features and NDVI branch features before fusion to obtain RGB single-modal prediction results and NDVI single-modal prediction results; the main segmentation loss is calculated using the cross-entropy loss function on the bimodal fusion prediction results and the ground semantic annotation map; a correct region mask is generated based on whether the predicted category of each pixel in the bimodal fusion prediction result is consistent with the ground semantic annotation map, and this mask indicates the region correctly predicted by the fusion branch; using the cross-entropy loss function, the self-guided auxiliary loss between the RGB single-modal prediction result and the ground annotation map, and the self-guided auxiliary loss between the NDVI single-modal prediction result and the ground annotation map are calculated only at the pixel positions where the correct region mask value is 1, and pixels with a mask value of 0 are not included in the auxiliary loss calculation.

[0017] As a preferred embodiment of the present invention, during training, the RGB input or the NDVI dual-channel exponential input is randomly occluded. The occlusion operation is achieved by multiplying the input with an occlusion coefficient that is randomly set to 0 or 1. During training, either the RGB occlusion coefficient or the NDVI occlusion coefficient is randomly set to 0, or both are set to 1; it is prohibited for both to be 0 simultaneously. The final loss function is composed of a weighted sum of the main segmentation loss and the self-guided auxiliary loss, wherein the weight coefficient of the main segmentation loss is greater than or equal to the weight coefficient of the self-guided auxiliary loss. The optimizer uses Adam, AdamW, or SGD, for example, the AdamW optimizer, with the initial learning rate set to 1×10⁻⁶. -4 Or 6×10 -5 The weight decay is set to 0.01, the batch size is set to 4, 8 or 16 depending on the memory size, the training epochs are set to 100 to 300, and the learning rate adjustment strategy adopts cosine annealing, polynomial decay or staged decay strategy.

[0018] In addition, the present invention also discloses a remote sensing image segmentation device based on RGB and NDVI exponential fusion, including a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the remote sensing image segmentation method based on RGB and NDVI exponential fusion as described above.

[0019] In addition, the present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the remote sensing image segmentation method based on RGB and NDVI exponential equalization fusion as described above.

[0020] In addition, the present invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the remote sensing image segmentation method based on RGB and NDVI exponential equalization fusion as described above.

[0021] Compared with the prior art, the present invention has the following beneficial effects: This invention addresses the problems of imbalanced contributions from RGB visible light features and NDVI vegetation index features, cross-modal noise diffusion, and excessive model dependence on a single modality in existing remote sensing semantic segmentation by constructing a dual-modal input network and an equalization fusion mechanism. The invention constructs NDVI as a dual-channel index input consisting of the original NDVI numerical map and the NDVI gradient map, enabling the network to simultaneously learn vegetation index intensity and index boundary change information. This allows for the simultaneous utilization of NDVI's spectral discrimination and spatial boundary response capabilities in land cover discrimination, enhancing segmentation accuracy at the boundary between vegetation and non-vegetation categories. The index-aware bidirectional cue adaptation module aggregates RGB and NDVI features in a unified low-rank space, generating dedicated cueing for each modality. It then utilizes a modality-specific bottleneck adapter to transfer and adapt the feature representation of the pre-trained Transformer encoding layer, achieving balanced enhancement of both modalities while maintaining a small number of trainable parameters. This effectively avoids the problem of a single modality dominating and weakening the information of the other modality.

[0022] The dynamic sparse cross-exponential fusion module predicts dynamic sampling points for RGB and NDVI respectively, merges them into a cross-modal candidate point set, and performs cross-modal feature sampling, weighting, and attention fusion at a few key locations. This reduces computational redundancy and cross-regional noise diffusion caused by global dense fusion, while preserving the long-distance semantic association ability of key regions, thus improving the model's robustness in segmenting complex mixed-feature areas. The occlusion modality self-guided training strategy randomly occludes RGB input or NDVI dual-channel exponential input during training, simulating real-world application scenarios where a single modality is disturbed or missing. Simultaneously, it uses the correct regions predicted by dual-modal fusion to provide auxiliary supervision to the RGB single-modal branch and the NDVI single-modal branch, enabling the single-modal branch to gradually learn to approach the discriminative ability of the dual-modal fusion result, thereby enhancing the model's generalization ability and stability under conditions such as shadow occlusion, exponential saturation, and cross-regional migration. The multi-scale lightweight segmentation head aggregates multi-scale fusion features output from different encoding stages, further improving the network's ability to segment targets of different sizes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0024] Figure 1This is a diagram showing the overall structure of the remote sensing semantic segmentation network integrated in this invention.

[0025] Figure 2 This is the bidirectional prompting adaptation module for index perception in this invention.

[0026] Figure 3 This is the dynamic sparse cross-index fusion module of the present invention. Detailed Implementation

[0027] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0028] The following is in conjunction with the appendix Figures 1-3 The embodiments of the present invention will be described in detail below.

[0029] Figure 1 It demonstrates the connection relationship between RGB optical image input, NDVI dual-channel exponential input, dual-modal patch embedding, exponential-aware bidirectional prompting adaptation module, dynamic sparse cross-exponential fusion module, multi-scale lightweight segmentation head and final semantic segmentation output.

[0030] Figure 2 This demonstrates how RGB and NDVI features are represented by a unified representation through low-rank bimodal convergence, then RGB-specific and NDVI-specific hints are generated separately, and feature enhancement is achieved through a modality-specific bottleneck adapter. Figure 3 This demonstrates how RGB and NDVI features can be used to predict dynamic sampling points, which are then merged into a cross-modal candidate point set. Key features are sampled on the candidate points, and the fused features are calculated using modal contribution weighting and attention.

[0031] Figure 1 The Index-AwareBidirectionalPromptAdaptationModule is the index-aware bidirectional prompt adaptation module, the DynamicSparseCross-IndexFusionModule is the dynamic sparse cross-index fusion module, and Stage1-4 represent the four encoding stages. Figure 3 The Multi-head in this context refers to the multi-head attention module. , The sampling offsets for RGB and NDVI predicted by the network, respectively. To integrate query features.

[0032] This invention discloses a remote sensing image segmentation method based on balanced fusion of RGB and NDVI exponential features. Addressing the issue that RGB visible light features and NDVI exponential features in remote sensing images are complementary but have significantly different distributions, the method constructs an RGB+NDVI bimodal semantic segmentation network. The network takes RGB optical images and a dual-channel exponential input consisting of the original NDVI numerical map and the NDVI gradient map as input. The basic segmentation process is completed through bimodal patch embedding, a pre-trained Transformer encoding layer, and a lightweight segmentation head. The technical units include an exponential-aware bidirectional cue adaptation module, a dynamic sparse cross-exponential fusion module, and a self-guided training strategy for occlusion modality.

[0033] Acquire remote sensing image data of the area to be processed. The remote sensing images include at least RGB optical images and near-infrared and red bands for calculating NDVI. RGB optical images provide information on the color, texture, shape boundaries, and spatial context of ground features; near-infrared and red bands are used to calculate NDVI to provide spectral index information for vegetation-related ground features. For image data from different sensor sources, perform radiometric correction, atmospheric correction, geometric correction, and spatial registration to ensure that the RGB optical images, near-infrared bands, red bands, and corresponding semantic annotation maps are consistent in spatial location and resolution.

[0034] The raw NDVI values ​​are calculated based on the near-infrared and red light bands. The formula for calculating NDVI is as follows:

[0035] in, This represents the raw numerical plot of NDVI. Indicates near-infrared reflectivity. Indicates the reflectivity in the red light band. A very small constant set to prevent the denominator from being zero, for example, can be set as follows: Typically, the range of NDVI values ​​is... Before actually inputting the data into the network, the NDVI values ​​can be normalized to... Alternatively, it can be standardized to a form with a mean of 0 and a variance of 1 to improve the stability of network training.

[0036] Furthermore, an NDVI gradient map is calculated based on the original NDVI numerical map to represent the magnitude of NDVI variation and boundary response within the spatial neighborhood. The Sobel operator or a first-order difference operator is used to calculate the gradients of NDVI in the horizontal and vertical directions, respectively, yielding:

[0037]

[0038] in, and These represent the gradients of NDVI in the horizontal and vertical directions, respectively. This represents the NDVI gradient map. The NDVI gradient map can highlight vegetation boundaries, farmland ridges, water-land boundaries, urban green space boundaries, and areas of exponential abrupt changes between different vegetation cover types, thus providing auxiliary information for subsequent feature extraction and fusion.

[0039] Then, the original NDVI numerical plot With NDVI gradient plot Combining along the channel dimension forms the NDVI dual-channel exponential input:

[0040] in, This represents the dual-channel NDVI exponential input. Unlike the method of directly splicing a single-channel NDVI as the fourth channel of RGB, the dual-channel NDVI exponential input expresses the numerical intensity information and spatial variation information of NDVI separately, enabling the network to learn vegetation index intensity and exponential boundary changes separately.

[0041] To improve the model's generalization ability, simultaneous data augmentation is performed on the RGB optical image, NDVI dual-channel exponential input, and annotation map during the training phase. Data augmentation methods include geometric enhancements such as random horizontal flipping, random vertical flipping, random rotation, random scaling, and random cropping. For the RGB optical image, further perturbations of brightness, contrast, saturation, and hue can be applied. For the NDVI dual-channel exponential input, slight Gaussian noise, local random occlusion, or exponential value perturbations can be added. All geometric transformations are applied simultaneously to the RGB optical image, NDVI dual-channel exponential input, and semantic annotation map to ensure that pixel-level labels are not misaligned.

[0042] A remote sensing semantic segmentation network based on RGB+NDVI fusion is constructed. The network consists of an RGB input branch, an NDVI input branch, a bimodal patch embedding layer, a multi-stage pre-trained Transformer encoding layer, a lightweight segmentation head, and three core technical units. The bimodal patch embedding layer, the pre-trained Transformer encoding layer, and the lightweight segmentation head are the basic components of the network. An exponentially-aware bidirectional cue adaptation module generates specific cues for both RGB and NDVI modalities and performs low-rank adaptation within a frozen or partially frozen pre-trained visual backbone. A dynamic sparse cross-exponential fusion module performs cross-modal feature sampling, weighting, and attention fusion of RGB and NDVI at a small number of dynamically selected key locations. An occlusion modality self-guided training strategy randomly occludes one modality during training and uses the correct region in the bimodal fusion prediction to inversely constrain the single-modal branch, reducing the model's over-reliance on any single modality.

[0043] The network input includes RGB optical images:

[0044] Dual-channel index input with NDVI:

[0045] in, Represents RGB optical image, This indicates the NDVI dual-channel exponential input. Indicates the image height. This represents the image width. Both are processed through independent patch embedding layers to obtain initial feature labels:

[0046] in, Indicates the RGB patch embedding layer. This indicates the NDVI dual-channel exponential patch embedding layer. and These represent RGB feature labels and NDVI feature labels, respectively. The network can be configured with four encoding stages. Each encoding stage is processed using exponentially-aware bidirectional cue adaptation, dynamic sparse cross-exponential fusion, and stage output. The fused features from each stage are then fed into a lightweight segmentation head to generate the final semantic segmentation result.

[0047] For example, if an input 512x512 RGB optical image is used with a patch size of 16x16, the RGB patch embedding layer will convert it into a sequence of length 1024, with each marker having a dimension of 256 or 768.

[0048] The exponentially-aware bidirectional cue adaptation module addresses the differences between RGB and NDVI in terms of numerical distribution, physical meaning, and feature representation capabilities. Instead of treating NDVI as a simple additional channel to RGB, this module treats both RGB and NDVI as equal input modalities, aggregating their information in a unified low-rank space. It then generates dedicated cues for RGB and NDVI respectively, and transfers the feature representation capabilities of the pre-trained visual backbone to both modalities through a modality-specific bottleneck adapter.

[0049] For RGB features in an encoding stage and NDVI features First, we obtain the low-rank representations of both through dimensionality reduction mapping:

[0050] in, and These represent the dimensionality reduction mapping parameters for RGB and NDVI, respectively. For example, if... Its dimensions are 1024x256. The 256 dimensions can be reduced to 64 dimensions, resulting in a dimension of 1024x64. Subsequently, the two low-rank representations are concatenated and mapped to a unified bimodal representation:

[0051] in, This represents a two-modal convergence mapping. This represents a dimension-up mapping. This represents a unified representation that simultaneously incorporates RGB texture semantics and NDVI index semantics. Because this process is performed in a low-rank space, it reduces the number of additional parameters and computational cost. Then, the unified representation... Projected as RGB and NDVI indicators respectively:

[0052]

[0053] in, and These respectively indicate RGB-specific prompts and NDVI-specific prompts. , , , For learnable parameters, Indicates the scaling factor. These can be learned parameters or preset fixed hyperparameters. Through this design, the cues for both modalities come from a unified bimodal representation, but are each processed through modal-specific projections, thus avoiding the weakening of modal differences caused by using identical cues for RGB and NDVI.

[0054] The prompted features are fed into a Transformer encoding layer with a modality-specific bottleneck adapter. For modality... ,make Indicates the corresponding modal features, If the corresponding modal prompt is indicated, the processing procedure is as follows:

[0055]

[0056]

[0057]

[0058] in, This indicates the bulls' self-attention. Representation layer normalization, Indicates a feedforward network. and Representing modes The bottleneck parameters for dimensionality reduction and dimensionality increase. Represents a non-linear activation function. This indicates the scaling factor of the adapter output. These can be learned parameters or preset fixed hyperparameters. This represents the modal characteristics after adaptation.

[0059] The pre-trained Transformer backbone parameters are kept frozen or partially frozen, and only the cue parameters, bottleneck adapter parameters, and subsequent fusion parameters are trained to reduce training costs and improve generalization ability on small sample remote sensing data.

[0060] In some embodiments, the pre-trained Transformer backbone may be a ViT-B / 16 model pre-trained on the ImageNet-22k dataset. During training, the parameters of its first 12 layers of multi-head attention and feedforward networks can be frozen, with only the parameters of layers 13 to 24 of the multi-head attention and feedforward networks being fine-tuned.

[0061] During training, the parameters of the first 12 layers of the multi-head attention and feedforward network can be frozen, with only the parameters of subsequent layers fine-tuned. The role of the exponentially-aware bidirectional cue adaptation module is not to add a new large-scale backbone, but to enhance the expressive power of RGB and NDVI respectively on the basis of the same pre-trained backbone. The RGB branch can focus on preserving texture, edges, and spatial structure, while the NDVI branch can focus on learning vegetation exponential intensity and exponential change boundaries, thereby achieving bimodal equalization enhancement during the encoding stage.

[0062] The Dynamic Sparse Cross-Exponential Fusion module addresses the issues of high computational cost and the easy introduction of irrelevant regional noise in global cross-modal fusion. Instead of establishing dense cross-modal associations across the entire image, this module predicts a small number of deformable sampling locations for RGB and NDVI in low-rank space, merges the key location sets of the two modalities, and performs cross-modal sampling, modality weight evaluation, and attention fusion at these key locations. The RGB features output by the exponentially-aware bidirectional cue adaptation module are then integrated into the fusion module. and NDVI features Projected into the low-rank fusion space:

[0063] in, and This represents the low-rank projection parameters. Subsequently, a regular grid reference point is constructed. The sampling offsets for RGB and NDVI are predicted separately using a lightweight offset prediction network.

[0064]

[0065] in, The offset prediction network can be represented by depthwise separable convolutions, layer normalization, activation functions, and... Convolutional structure; and These represent dynamic sampling points for RGB and NDVI, respectively. RGB dynamic sampling points tend to focus on structural areas such as texture edges, building outlines, and road extensions, while NDVI dynamic sampling points tend to focus on areas such as abrupt changes in vegetation index, farmland boundaries, forest-grass transitions, and water-land boundaries.

[0066] The sampling points from the two modalities are merged into a cross-modal candidate point set. In this embodiment, a normalized image coordinate format (with values ​​ranging from [-1, +1]) is used. , After merging, duplicate coordinates are removed. For sampled coordinates that exceed the boundary, a boundary padding mode is used (the out-of-bounds coordinates are truncated to the boundary value of [-1, +1]). The total number of candidate points has a preset upper limit.

[0067] In the candidate point set The following samples were taken from RGB low-rank features and NDVI low-rank features respectively:

[0068] in, This indicates a bilinear sampling or differentiable sampling operation. and This represents the key RGB and NDVI features obtained on a unified candidate point set. The advantage of using a unified candidate point set is that even if a certain position is considered important by the RGB branch, the corresponding NDVI features at the same position can be sampled synchronously, and vice versa, thus achieving cross-modal complementarity.

[0069] Furthermore, a lightweight fusion network is used to generate coarse-grained fusion query features:

[0070] in, This represents a coarse fusion function composed of convolution, normalization, and activation functions. In the candidate point set Upsampling ,in, This represents the coarsely fused query features after sampling; and the contribution weights of RGB and NDVI at each candidate point are calculated using the modal contribution evaluation function:

[0071] in, This represents the modal contribution evaluation function, which consists of two layers of linear mapping and nonlinear activation. and These represent the fusion weights of RGB and NDVI at the candidate points, respectively. Then, a weighted sum of the two modal features at the candidate points is performed:

[0072] in, This represents element-wise multiplication. This represents the key features after dynamic sparse fusion. Projection is the key and value in attention computation, and coarsely fused query features. Modeling cross-modal contexts as queries:

[0073]

[0074] in, and Representing the key and value respectively. , and Represents the linear mapping parameters. This indicates multi-head attention computation. This represents the fused features output at this stage. The fused features can be fed into the lightweight segmentation head or the next encoding stage after dimensionality upscaling. Through a dynamic sparse cross-exponential fusion module, the network can establish cross-modal relationships between RGB and NDVI at a small number of key candidate points, avoiding the propagation of irrelevant ground features and NDVI noise to the current pixel by global attention, while preserving long-distance semantic association capabilities. For high-resolution remote sensing imagery, this approach can also reduce computational burden.

[0075] After multiple encoding stages, the network acquires fused features at different scales. , , , The fused features from each stage are upsampled to a uniform scale and then concatenated or summed. A semantic probability map is then output through a lightweight segmentation head.

[0076] in, Indicates the fusion of the segmentation heads. This represents the final semantic probability graph. This represents the number of land cover categories. Taking the maximum value for the category dimension yields the final segmentation result.

[0077] in, Represents pixels The prediction category.

[0078] To ensure the network learns both the segmentation results from the dual-modal fusion and avoids weakening of either the RGB or NDVI branches during training, this invention employs a self-guided training strategy based on occlusion modality. Training supervision consists of the main segmentation loss and a self-guided auxiliary loss for correct regions.

[0079] During training, the network outputs a bimodal fusion prediction. In addition, there are RGB branch features before fusion. and NDVI branch features Auxiliary segmentation heads are set up on each side to obtain single-mode prediction results:

[0080] in, and These represent the RGB auxiliary segmentation head and the NDVI auxiliary segmentation head, respectively. Main segmentation supervision is applied to the fusion prediction results:

[0081] in, Represents cross-entropy loss, This represents the ground truth semantic annotation map. Then, based on the consistency between the fusion prediction result and the ground truth annotation map, a correct region mask is constructed:

[0082] in, Represents pixels Does it belong to the correctly predicted region of the fusion branch? This indicates the indicator function. Using the correct region mask, the self-guided auxiliary loss is calculated only at pixels where the mask value is 1; pixels with a mask value of 0 are excluded from the loss calculation.

[0083]

[0084] in, This represents the self-guided auxiliary loss for RGB single-modal prediction. This represents the self-guided auxiliary loss for NDVI single-modal prediction. This design avoids forcibly applying overly difficult supervision in areas where single-modality is difficult to judge or is occluded, thereby enabling the RGB and NDVI branches to gradually learn to approach the discriminative ability of the dual-modal fusion result.

[0085] During training, each training sample first undergoes a full bimodal forward propagation to compute the bimodal fusion prediction and its correct region mask. Then, a single-modal / missing modal forward propagation is performed with the RGB input or NDVI dual-channel exponential input randomly occluded to simulate scenarios where a single modality is perturbed, missing, or degraded in quality. The occlusion states include RGB occlusion, NDVI occlusion, and no occlusion, each with a sampling probability of 1 / 3. The occlusion method can be represented as:

[0086] in, During training, one of the parameters is randomly set to 0 or both to 1, and both are prohibited from being 0 simultaneously. When indicates that the RGB input is masked, The time interval represents the occlusion of the NDVI input. Through random occlusion, the model is forced to learn to maintain a stable output even when any modality is perturbed. The final loss function is:

[0087] in, and For loss weighting coefficients, Greater than or equal to The primary optimization objective is to ensure the dual-modal fusion output, while the single-modal auxiliary output serves as the characterization enhancement objective. For example, it can be... Set to 1.0, Set it to 0.5.

[0088] During the model training phase, the RGB+NDVI fusion semantic segmentation network was trained end-to-end using the training set. RGB optical images and NDVI dual-channel exponential inputs were simultaneously fed into the network, with the semantically labeled map serving as the supervision signal. The network obtained enhanced features from RGB and NDVI through an exponentially-aware bidirectional cue adaptation module, obtained fused features through a dynamic sparse cross-exponential fusion module, and output the final semantic segmentation result through a lightweight segmentation head.

[0089] Simultaneously, the occlusion modality self-guided training strategy jointly supervises both the bimodal fusion prediction and the single-modal assisted prediction. The optimizer can be Adam, AdamW, or SGD; for example, using the AdamW optimizer with an initial learning rate set to 1×10⁻⁶. -4 Or 6×10 -5The weight decay is set to 0.01, the batch size is set to 4, 8, or 16 depending on the GPU memory size, and the number of training epochs is set to 100 to 300. The learning rate adjustment strategy can employ cosine annealing, polynomial decay, or staged decay strategies. During training, the average intersection-over-union ratio, overall accuracy, and loss value are calculated on the validation set every few epochs to monitor the model's convergence status and select the optimal model parameters. To improve model robustness, RGB optical images and NDVI dual-channel exponential inputs are simultaneously and randomly cropped, flipped, rotated, and scaled during training.

[0090] For RGB optical imagery, random brightness, contrast, and color perturbations can be applied; for NDVI dual-channel exponential input, slight noise perturbations, local occlusion, or random jitter of exponential values ​​can be applied. Combined with an occlusion modality self-guided training strategy, the model can adapt to data variations under different seasons, lighting conditions, sensor types, and land cover conditions.

[0091] In terms of training parameter settings, most parameters of the pre-trained Transformer backbone can be frozen, and only the parameters of the patch embedding layer, the exponentially-aware bidirectional cueing adaptation module, the dynamic sparse cross-exponential fusion module, and the segmentation head can be trained. Alternatively, some high-level coding parameters can be fine-tuned according to the data scale. This efficient parameter training method can reduce the risk of overfitting when there are insufficient remote sensing labeled samples and reduce training costs when deploying in multiple regions and with multiple sensors.

[0092] After model training, the method of this invention was quantitatively and qualitatively evaluated on a test set. Quantitative evaluation metrics included mean Intersection over Union (mIoU), overall accuracy (OA), mean pixel accuracy (mPA), category F1 score, and Kappa coefficient. Specifically, mIoU evaluates the degree of overlap between segmented regions of each category and the ground truth annotations; OA evaluates the overall pixel classification accuracy; and the category F1 score evaluates the balance between precision and recall for a single land cover category. In the specific evaluation, IoU and F1 scores for categories such as buildings, roads, water bodies, forest land, cultivated land, grassland, and bare land were statistically analyzed to assess the segmentation effect of RGB and NDVI fusion on different types of land cover.

[0093] Example 1: In this example, remote sensing image data of the area to be processed is acquired. The remote sensing image includes at least RGB optical images and near-infrared and red bands for calculating NDVI, and the corresponding semantic annotation map is obtained. Radiometric correction, atmospheric correction, and geometric correction are performed on the RGB optical images, near-infrared bands, and red bands. Geometric transformation and label-preserving resampling consistent with the images are performed on the semantic annotation map to achieve spatial registration, ensuring that the RGB optical images, near-infrared bands, red bands, and corresponding semantic annotation maps are consistent in spatial location and resolution. For example, all images are resampled to a uniform resolution of 0.5m. The original NDVI value map is calculated based on the near-infrared and red bands using the following formula:

[0094] The gradients of NDVI in the horizontal and vertical directions are calculated using either the Sobel operator or the first-order difference operator:

[0095]

[0096] Combine the original NDVI numerical plot and the NDVI gradient plot along the channel dimension:

[0097] The NDVI dual-channel exponential input is obtained. In this embodiment, the original NDVI numerical range is preserved before actual input to the network. During the training phase, simultaneous data augmentation is performed on RGB optical images, NDVI dual-channel exponential inputs, and semantic annotation maps. For example, all images are randomly cropped to 512x512 pixels. Data augmentation methods include random horizontal flipping, random vertical flipping, random rotation, random scaling, and random cropping. RGB optical images can be further perturbed in terms of brightness, contrast, saturation, and hue. The NDVI dual-channel exponential inputs can be supplemented with slight Gaussian noise, local random occlusion, or exponential value perturbation.

[0098] RGB optical imaging (Dimensions 512x512x3) and NDVI dual-channel index input (Size 512x512x2) Each is embedded through a separate patch layer:

[0099] Obtain RGB feature tags (Dimension 1024x768) and NDVI feature markers (Dimensionality 1024x768). The network can be configured with four encoding stages, each processed using exponentially-aware bidirectional cue adaptation, dynamic sparse cross-exponential fusion, and stage output. For the RGB and NDVI features of an encoding stage, low-rank representations are obtained through dimensionality reduction mapping, for example, reducing the 768 dimensions to 64 dimensions:

[0100] The two low-rank representations are concatenated and mapped to a unified bimodal representation:

[0101] The unified representation will be projected as RGB and NDVI hints respectively:

[0102]

[0103] The prompted features are fed into a Transformer encoding layer with a modality-specific bottleneck adapter:

[0104]

[0105]

[0106]

[0107] Obtain the adapted RGB features and NDVI features .

[0108] Project the RGB and NDVI features output by the exponentially-aware bidirectional cueing adaptation module into the low-rank fusion space:

[0109] Constructing rule grid reference points The sampling offsets for RGB and NDVI are predicted separately using a lightweight offset prediction network:

[0110]

[0111] The sampling points from the two modes are merged into a cross-modal candidate point set:

[0112] Sampling RGB low-rank features and NDVI low-rank features on the candidate point set:

[0113] Generate coarse-grained query features using a lightweight fusion network:

[0114] Will Sampling on the candidate point set ,in, This represents the coarsely fused query features after sampling; the contribution weights of RGB and NDVI are calculated using the modal contribution evaluation function.

[0115] The two modal features at the candidate point are weighted and summed:

[0116] The key features after dynamic sparse fusion are projected as keys and values ​​in attention computation:

[0117] Cross-modal context modeling using coarse-grained fusion query features as queries:

[0118] The four encoding stages each generate fusion features at different scales. , , , The fused features from each stage are upsampled to a uniform scale and concatenated, then a semantic probability map is output through a lightweight segmentation head.

[0119] The final segmentation result is obtained by taking the maximum value of the category dimension:

[0120] During training, auxiliary segmentation heads are set on the RGB branch features and NDVI branch features before fusion, respectively:

[0121] Apply principal segmentation supervision to the fusion prediction results:

[0122] Construct the correct region mask based on the consistency between the fusion prediction results and the actual labeled map:

[0123] Using the correct region mask, the self-guided auxiliary loss is calculated only at pixels where the mask value is 1; pixels with a mask value of 0 are not included in the loss calculation.

[0124]

[0125] Randomly mask RGB input or NDVI dual-channel exponential input during training:

[0126] in, During training, one of the parameters is randomly assigned to 0 or both to 1; it is forbidden for both to be 0 simultaneously. The final loss function is:

[0127] in, , The optimizer used is the AdamW optimizer, with an initial learning rate set to 6 × 10⁻⁶. -5 The weight decay was set to 0.01, the batch size to 8, and the training epochs to 200. A cosine annealing strategy was used to adjust the learning rate. During training, the mean intersection-over-union (mIoU), overall precision (OA), and loss were calculated on the validation set every 10 epochs to monitor model convergence and select optimal model parameters. For training parameter settings, the first 12 layers of the ViT-B / 16 pre-trained backbone were frozen, and only the patch embedding layer, the exponentially-aware bidirectional cue adaptation module, the dynamic sparse cross-exponential fusion module, and the segmentation head parameters were trained. After model training, quantitative and qualitative evaluations were performed on the test set. Quantitative evaluation metrics included mean intersection-over-union (mIoU), overall precision (OA), mean pixel precision (mPA), class F1 score, and Kappa coefficient. IoU and F1 scores were statistically analyzed for categories such as buildings, roads, water bodies, forest land, cultivated land, grassland, and bare land.

[0128] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing image segmentation method based on RGB and NDVI exponential fusion, characterized in that: include: Acquire remote sensing image data of the area to be processed. The remote sensing image includes at least RGB optical image and near-infrared and red bands for calculating NDVI. Perform radiometric correction, atmospheric correction and geometric correction on the RGB optical image, near-infrared band and red band. Perform geometric transformation and label-preserving resampling consistent with the image on the semantically labeled map to achieve spatial registration. The original NDVI numerical map is calculated based on the near-infrared and red bands. The NDVI gradient map is calculated based on the original NDVI numerical map. The original NDVI numerical map and the NDVI gradient map are combined along the channel dimension to form the NDVI dual-channel exponential input. The RGB optical image and the NDVI dual-channel index input are respectively processed through independent patch embedding layers to obtain RGB feature tags and NDVI feature tags; Within each encoding stage, RGB and NDVI features are processed sequentially through the exponentially-aware bidirectional cueing adaptation module and the dynamic sparse cross-exponential fusion module to obtain the fusion features corresponding to each encoding stage. The fusion features of each stage are then fed into the lightweight segmentation head to generate the final semantic segmentation result. During training, each training sample first undergoes a complete bimodal forward propagation to compute the bimodal fusion prediction and its correct region mask. Then, the RGB input or NDVI dual-channel exponential input is randomly occluded to perform a single-modal / missing modality forward propagation. The occlusion states include three cases: occluded RGB, occluded NDVI, and no occlusion, with a sampling probability of 1 / 3 for each case. The correct regions in the bimodal fusion prediction are used to apply self-guided auxiliary supervision to the RGB single-modal prediction and NDVI single-modal prediction. The network is trained using the main segmentation loss and the correct region self-guided auxiliary loss.

2. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 1, characterized in that: The raw NDVI numerical map is calculated based on the near-infrared and red bands. The raw NDVI numerical map is equal to the difference between the reflectance of the near-infrared band and the reflectance of the red band divided by the sum of the reflectance of the near-infrared band and the red band plus a preset minimum constant. Based on the raw NDVI numerical map, the NDVI gradient map is calculated. The Sobel operator or the first-order difference operator is used to calculate the gradient of the raw NDVI numerical map in the horizontal and vertical directions, respectively. The square root of the sum of the squares of the horizontal gradient and the squares of the vertical gradient is then taken to obtain the NDVI gradient map. The raw NDVI numerical map and the NDVI gradient map are combined along the channel dimension to form the NDVI dual-channel exponential input.

3. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 2, characterized in that: The network input includes an RGB optical image with height H, width W, and 3 channels, and an NDVI dual-channel exponential input with height H, width W, and 2 channels. Both are processed by independent patch embedding layers to obtain RGB feature labels and NDVI feature labels, respectively. The independent patch embedding layers include an RGB patch embedding layer and an NDVI dual-channel exponential patch embedding layer. The network can be set to four encoding stages. Each encoding stage is processed according to the method of exponential-aware bidirectional cue adaptation, dynamic sparse cross-exponential fusion, and stage output. The fused features of each stage are sent to a lightweight segmentation head to generate the final semantic segmentation result.

4. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 3, characterized in that: The exponentially-aware bidirectional prompting adaptation module treats RGB and NDVI as equal input modalities, converges their information in a unified low-rank space, and then generates dedicated prompts for RGB and NDVI respectively. For RGB and NDVI features in an encoding stage, the RGB features are first reduced in dimensionality using RGB dimensionality reduction mapping parameters to obtain an RGB low-rank representation, and the NDVI features are reduced in dimensionality using NDVI dimensionality reduction mapping parameters to obtain an NDVI low-rank representation. The RGB and NDVI low-rank representations are concatenated, and after bimodal convergence mapping and dimensionality-upgrading mapping, a unified representation containing both RGB texture semantics and NDVI exponential semantics is obtained. The unified representation is subjected to a first linear transformation and a second linear transformation, and after the transformation, a scaling factor is added to the product of the unified representation to obtain RGB-specific prompts and NDVI-specific prompts. The first linear transformation has a learnable first weight and a first bias, and the second linear transformation has a learnable second weight and a second bias. The scaling factor is a learnable parameter or a preset fixed hyperparameter.

5. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 4, characterized in that: The prompted features are fed into a Transformer encoding layer with a modality-specific bottleneck adapter. For each modality, the modality features are added to the corresponding modality prompt to obtain a first intermediate feature. The first intermediate feature is then subjected to layer normalization and multi-head self-attention processing, and then added to the first intermediate feature to obtain a second intermediate feature. The second intermediate feature is then dimensionality-reduced using the bottleneck dimensionality reduction parameters of the modality, passed through a non-linear activation function, and then dimensionality-increasing using the bottleneck dimensionality-increasing parameters of the modality to obtain the output of the modality-specific bottleneck adapter. The second intermediate feature is then subjected to layer normalization and feedforward network processing, and then added to the product of the second intermediate feature, the adapter scaling coefficient, and the output of the modality-specific bottleneck adapter to obtain the adapted modality features. The adapter scaling coefficient is a learnable parameter or a preset fixed hyperparameter. The pre-trained Transformer backbone parameters are kept frozen, and only the prompt parameters, bottleneck adapter parameters, and subsequent fusion parameters are trained.

6. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 5, characterized in that: The dynamic sparse cross-exponential fusion module projects the RGB and NDVI features output by the exponential-aware bidirectional prompting adaptation module onto the low-rank fusion space through low-rank projection parameters to obtain RGB low-rank features and NDVI low-rank features. A regular grid reference point is constructed, and the sampling offsets of RGB and NDVI are predicted separately through a lightweight offset prediction network. The regular grid reference point is added to the sampling offset to obtain RGB dynamic sampling points and NDVI dynamic sampling points. The offset prediction network consists of depthwise separable convolution, layer normalization, activation function and 1×1 convolution. The RGB dynamic sampling points and NDVI dynamic sampling points are merged into a cross-modal candidate point set. On the cross-modal candidate point set, RGB low-rank features and NDVI low-rank features are sampled respectively through bilinear sampling or differentiable sampling operations to obtain RGB key features and NDVI key features.

7. The remote sensing image segmentation method based on RGB and NDVI exponential equalization fusion according to claim 6, characterized in that: A lightweight fusion network is used to concatenate RGB low-rank features and NDVI low-rank features, followed by convolution, normalization, and activation function processing to generate coarse fusion query features. These coarse fusion query features are then sampled on the cross-modal candidate point set to obtain sampled coarse fusion query features. The sampled coarse fusion query features are then processed using a modal contribution evaluation function, and the fusion weights of RGB and NDVI at the candidate points are obtained using a softmax function. Finally, the fusion weights are element-wise multiplied and summed with the corresponding RGB key features and NDVI key features to obtain the dynamically sparsely fused key features. The key features after dynamic sparse fusion are projected as keys and values ​​in attention calculation, and multi-head attention calculation is performed using the coarse fusion query features as queries to obtain the fusion features output in the encoding stage; wherein, the modality contribution evaluation function consists of two layers of linear mapping and nonlinear activation.

8. The remote sensing image segmentation method based on RGB and NDVI exponential fusion according to claim 7, characterized in that: After multiple encoding stages, the network obtains fused features at different scales. The fused features from each stage are upsampled to a unified scale and then concatenated or summed. Finally, a lightweight segmentation head is used to output the final semantic probability map. The maximum value of the category dimension of the final semantic probability map is taken to obtain the predicted category of each pixel.

9. The remote sensing image segmentation method based on RGB and NDVI exponential equalization fusion according to claim 8, characterized in that: During training, in addition to outputting bimodal fusion predictions, the network also sets auxiliary segmentation heads on the RGB branch features and NDVI branch features before fusion to obtain RGB single-modal prediction results and NDVI single-modal prediction results; the main segmentation loss is calculated using the cross-entropy loss function on the bimodal fusion prediction results and the real semantic annotation map. Based on whether the predicted category of each pixel in the dual-modal fusion prediction result is consistent with the ground semantic annotation map, a correct region mask is generated. This mask indicates the region correctly predicted by the fusion branch. Using the cross-entropy loss function, the self-guided auxiliary loss between the RGB single-modal prediction result and the ground annotation map, and the self-guided auxiliary loss between the NDVI single-modal prediction result and the ground annotation map are calculated only at the pixel positions where the correct region mask value is 1. Pixels with a mask value of 0 are not included in the auxiliary loss calculation.

10. The remote sensing image segmentation method based on RGB and NDVI exponential equalization fusion according to claim 9, characterized in that: During training, the RGB input or the NDVI dual-channel exponential input is randomly occluded. The occlusion operation is achieved by multiplying the input with an occlusion coefficient that is randomly set to 0 or 1. During training, either the RGB occlusion coefficient or the NDVI occlusion coefficient is randomly set to 0, or both are set to 1. It is prohibited for both to be 0 at the same time. The final loss function is composed of the weighted sum of the main segmentation loss and the self-guided auxiliary loss, where the weight coefficient of the main segmentation loss is greater than or equal to the weight coefficient of the self-guided auxiliary loss.

Images