Multispectral and visible light remote sensing image fusion segmentation method, system and medium

Abstract

The present invention relates to the technical field of remote sensing vegetation identification, and more specifically to a method, system, and medium for fusion and segmentation of multispectral and visible light remote sensing images. The method comprises: constructing a dual-branch model comprising a visible light branch encoder, a multispectral branch encoder, a multiscale feature fusion module, and a shared decoder; inputting visible light data into the visible light branch encoder to obtain a spatial feature map; inputting multispectral data into the multispectral branch encoder to obtain a spectral feature map; the multiscale feature fusion module applies channel-spatial joint attention weights to the input feature map, then performs channel concatenation based on scale, compresses the feature map to the original number of channels using 1×1 convolution, and then applies channel attention weights to obtain a multiscale fused feature sequence; and the shared decoder parses the multiscale fused feature sequence and outputs an invasive plant segmentation mask image for the target detection area, thereby achieving highly robust invasive plant identification in complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the technical field of vegetation remote sensing identification, and in particular to a method, system and medium for fusion and segmentation of multispectral and visible light remote sensing images. Background Art

[0002] Invasive plants, with their robust reproductive and dispersal abilities, have significant allelopathic effects on crops, and serve as hosts for pests, posing a serious threat to both ecosystems and agricultural production. Therefore, large-scale, high-precision automated identification and monitoring of invasive plants is crucial. However, traditional monitoring methods, which rely on manual surveys for invasive plants, inevitably suffer from technical challenges such as low efficiency and limited spatial coverage, making them incapable of addressing the rapid spread of invasive plants.

[0003] With the rapid development of drone and multispectral remote sensing technologies, drone multispectral remote sensing, combining the flexibility of drones with the high resolution of multispectral imaging, has become an important complement to traditional ground-based monitoring and satellite remote sensing. While drone multispectral remote sensing has made significant progress in agriculture and ecology, research on invasive plant monitoring remains insufficient, particularly with the lack of a dedicated remote sensing technology system for identifying the distribution of invasive plants.

[0004] Currently, due to the significant differences in modal properties between multispectral and visible light data, existing general models (such as standard U-Net and DeepLabV3+) use simple band combinations or direct stacking processing methods. This leads to redundant feature information interference and noise amplification, making it difficult to effectively integrate the contributions of the two data sources. As a result, the detection accuracy of invasive plants cannot be met in complex vegetation scenes (such as urban green spaces and weeds). Therefore, existing remote sensing image segmentation methods mainly target a single data source, such as visible light data. However, single visible light data is affected by multiple factors such as the canopy structure of invasive plants and chlorophyll content, making the visible light segmentation results susceptible to background interference. Single multispectral data also generally suffers from the technical problem of insufficient spatial resolution. In addition, multispectral data has many bands, and direct use for high-resolution segmentation results in a huge computational burden, making it difficult to meet the needs of real-time monitoring. Therefore, multispectral data is generally used for biomass inversion rather than fine segmentation. Summary of the Invention

[0005] In response to the above technical problems, the present invention proposes a multispectral and visible light remote sensing image fusion segmentation method, system and medium, aiming to overcome the three core problems of feature conflict, computational inefficiency and poor scene adaptability in the segmentation of invasive plants using multimodal remote sensing data, effectively improve the segmentation accuracy of detection targets, and achieve highly robust invasive plant identification in complex backgrounds.

[0006] In a first aspect, the present application provides a method for fusion and segmentation of multispectral and visible light remote sensing images, comprising the following steps:

[0007] Collect visible light data and multispectral data of the target detection area;

[0008] Constructing a dual-branch model, the dual-branch model includes a visible light branch encoder, a multispectral branch encoder, a multi-scale feature fusion module, and a shared decoder;

[0009] The visible light data is input into the visible light branch encoder of the dual-branch model to extract a multi-scale spatial feature map;

[0010] The multispectral data is input into the multispectral branch encoder of the dual-branch model to extract the multi-scale spectral feature map;

[0011] The multi-scale spatial feature map and spectral feature map are input into the multi-scale feature fusion module. The channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features. The weighted feature map is then channel-jointed according to the scale. The joined feature map is compressed to the original number of channels using 1×1 convolution. The channel attention weight is then applied to the compressed fusion feature map to obtain a multi-scale fusion feature sequence.

[0012] The multi-scale fusion feature sequence is input into the shared decoder, the multi-scale fusion feature sequence is parsed, and the invasive plant segmentation mask image of the target detection area is output.

[0013] In some embodiments, applying a channel-spatial joint attention weight to the input feature map to dynamically adjust the weight distribution of multispectral and visible light features includes:

[0014] Generate channel attention weights based on the input feature map, weight the input feature map channel by channel to highlight the red edge and near-infrared band, and obtain the first weighted feature map;

[0015] A spatial attention weight is generated based on the first weighted feature map, and the first weighted feature map is spatially weighted to highlight the distribution area of ​​invasive plants to obtain a second weighted feature map.

[0016] In some embodiments, channel attention weights are generated based on the input feature map, and the input feature map is weighted channel by channel to highlight the red edge and the near-infrared band, thereby obtaining a first weighted feature map, including:

[0017] Perform global average pooling and maximum pooling on each channel of the input feature map to obtain two 1×1×C channel descriptors, where C represents the number of channels;

[0018] Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio.

[0019] Input two 1×1×C channel descriptors into a shared two-layer fully connected network, and use the Sigmoid function to generate the first channel attention weight;

[0020] The first channel attention weight is weighted with the input feature map channel by channel to highlight the red edge and near-infrared band to obtain the first weighted feature map.

[0021] In some embodiments, generating a spatial attention weight based on the first weighted feature map, performing spatial weighting on the first weighted feature map to highlight the distribution area of ​​invasive plants, and obtaining a second weighted feature map includes:

[0022] Perform global average pooling and maximum pooling on the first weighted feature map along the channel dimension to obtain two H×W×1 spatial descriptors, where H represents height and W represents width;

[0023] Two H×W×1 spatial descriptors are concatenated along the channel dimension;

[0024] Perform a 7×7 convolution operation on the splicing result and generate spatial attention weights after normalization using the Sigmoid function;

[0025] The spatial attention weight is multiplied pixel by pixel with the first weighted feature map to highlight the invasive plant distribution area, and the second weighted feature map is obtained.

[0026] In some embodiments, channel attention weights are applied to the compressed fused feature map to obtain a multi-scale fused feature sequence, including:

[0027] Perform global average pooling on each channel of the compressed fusion feature map to obtain the channel statistical descriptor;

[0028] Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio.

[0029] The channel statistical descriptor is input into a shared two-layer fully connected network, and the Sigmoid function is used to generate the second channel attention weight;

[0030] Weight the second channel attention weight and the compressed fusion feature map channel by channel to obtain the third weighted feature map;

[0031] The third weighted feature maps of all scales constitute a multi-scale fusion feature sequence.

[0032] In some embodiments, multispectral data is input into a multispectral branch encoder of a dual-branch model to extract a multi-scale spectral feature map, including:

[0033] Perform 3×3 convolution operation on each channel of the multispectral data independently to obtain the feature map after channel-by-channel convolution;

[0034] Perform 1×1 convolution operation on the feature map after channel-by-channel convolution to obtain the feature map after point-by-point convolution;

[0035] The feature map after point-by-point convolution is subjected to four-level downsampling operation to extract multi-scale spectral feature maps.

[0036] In some embodiments, each level of the downsampling operation sequentially performs two 3×3 convolution operations, a batch normalization operation, a ReLU activation operation, and a 2×2 maximum pooling operation.

[0037] In some embodiments, the multi-scale fusion feature sequence is input into a shared decoder, the multi-scale fusion feature sequence is parsed, and an invasive plant segmentation mask image of the target detection area is output, including:

[0038] Perform four-level upsampling operations on the multi-scale fusion feature sequence. A 3×3 transposed convolution operation is performed on each level. After each level of upsampling operation, a jump connection is performed with the feature map of the corresponding scale in the encoder, and then feature analysis is performed on the feature map after the jump connection.

[0039] Based on the parsed feature map, a segmentation mask image of invasive plants in the target detection area is generated.

[0040] In a second aspect, the present application provides a multispectral and visible light remote sensing image fusion and segmentation system, comprising:

[0041] Data acquisition module, used to collect visible light data and multispectral data of the target detection area;

[0042] A model construction module is used to construct a dual-branch model, wherein the dual-branch model includes a visible light branch encoder, a multispectral branch encoder, a multi-scale feature fusion module and a shared decoder;

[0043] A visible light branch encoder, configured to input visible light data to extract a multi-scale spatial feature map;

[0044] A multispectral branch encoder, used for inputting multispectral data to extract multi-scale spectral feature maps;

[0045] The multi-scale feature fusion module is used to apply channel-space joint attention weights to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features. The weighted feature map is then channel-jointed according to the scale. The joined feature map is compressed to the original number of channels using 1×1 convolution. The channel attention weights are then applied to the compressed fused feature map to obtain a multi-scale fused feature sequence.

[0046] The shared decoder is used to parse the multi-scale fusion feature sequence and output the invasive plant segmentation mask image of the target detection area.

[0047] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned multispectral and visible light remote sensing image fusion and segmentation method.

[0048] The beneficial technical effects of the present invention include at least:

[0049] 1. A multi-spectral and visible light remote sensing image fusion segmentation method, system and medium are used. The visible light segmentation results of the existing remote sensing image segmentation method are easily affected by background interference, and the single multi-spectral data used generally has the technical problem of insufficient spatial resolution. In addition, due to the significant difference in modal characteristics between multi-spectral data and visible light data, the existing general model is difficult to effectively fuse the contributions of the two types of data source information. For this reason, the present application designs a dual-branch model including independent dual-branch heterogeneous encoders, multi-scale feature fusion modules and shared decoders to avoid cross-modal interference and improve feature purity. First, high-resolution details are supplemented by the visible light branch encoder to improve the plant boundary segmentation ability. Secondly, in the multi-scale feature fusion module, the channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features to resolve the feature conflict between modalities. Then, the weighted feature map is processed according to The channels are spliced ​​at different scales to retain spectral differences and spatial consistency, forming a fusion feature map containing cross-modal information. Then, 1×1 convolution is used to compress and align the channels of the spliced ​​feature map, and the compressed fusion feature map is subjected to secondary channel screening to solve the cross-scale feature redundancy, thereby solving the problem of feature redundancy and weakened discriminability in the existing multi-scale fusion method, forming a complete technical chain of "modal alignment → scale fusion → channel optimization", which can significantly improve the accuracy and efficiency of the dual-branch model in segmenting invasive plants in complex scenes. Finally, by designing the dual-branch encoder to share the same decoder, the multi-modal features (visible light spatial details + multi-spectral spectral information) are uniformly decoded after fusion to enhance feature consistency. The shared decoder design not only effectively reuses the fusion features, but also reduces the parameter redundancy of the dual decoder, improves the inference speed of the dual-branch model, and is suitable for batch processing of large-scale remote sensing images.

[0050] 2. By combining sensor characteristics (such as band resolution) to design a targeted multispectral branch encoder, the existing standard convolution is decomposed into channel-by-channel convolution (spectral feature extraction) and point-by-point convolution (cross-channel fusion). This significantly reduces the computational complexity of the multispectral branch while ensuring recognition accuracy. The dual-branch encoder maintains the same downsampling level to ensure cross-modal feature scale alignment, providing a homogeneous feature foundation for subsequent operations.

[0051] 3. Traditional segmentation models in existing technologies rely on the local receptive field of convolution, making it difficult to globally perceive target distribution patterns (such as the clustering characteristics of invasive plants). They are also sensitive to background interference. Complex backgrounds (such as dead grass and soil texture) are similar to the target spectrum, which can easily lead to missegmentation. At the same time, areas with dense plants are easily ignored by the model due to feature overlap, resulting in a high missed detection rate. To this end, this application performs double-pooling fusion on the input feature map in the multi-scale fusion module, combines global statistics and local significance, comprehensively characterizes the importance of channels, avoids the information bias of single pooling, and then reduces the number of parameters by sharing the fully connected network (only 2C2 / r parameters are required), improves computational efficiency, and adaptively adjusts the contribution of multi-spectral bands through attention weights to highlight the red edge and near-infrared bands to obtain the first weighted feature map, which provides an efficient and explainable spectral selection mechanism for core segmentation tasks (such as invasive plant identification). Then, based on the first weighted feature map after the first channel attention weighting, global average pooling is used to characterize global distribution characteristics, identify large continuous areas (such as patches of invasive plants), and global maximum pooling is used to capture local significant peaks and locate sparse but highly responsive targets (such as scattered seedlings). , solve the problem of incomplete representation of spatial information by single pooling, and then break through the local limitations of traditional small convolution kernels, use 7×7 convolution to cover a larger spatial range, capture the continuous distribution pattern of dense plant areas (such as canopy cluster structure), enhance the spatial continuity perception of dense targets, and adaptively adjust the spatial attention weight through data-driven, dynamically adapt to input features, and avoid over-segmentation or under-segmentation caused by fixed thresholds; finally, perform secondary channel screening on the compressed fusion feature map to solve the cross-scale feature redundancy, thereby solving the feature redundancy and weakening of discriminability in existing multi-scale fusion methods, forming a complete technical chain of "modal alignment → scale fusion → channel optimization", which can significantly improve the segmentation accuracy and efficiency of the dual-branch model for invasive plants in complex scenarios, and provide reliable technical support for the precise monitoring of invasive plants;

[0052] 4. Traditional decoders usually rely on simple interpolation upsampling, which easily leads to blurred segmentation boundaries (such as the loss of jagged edges of leaves). In addition, the features of different levels of the encoder are not effectively fused, and the deep features lack low-level spatial details, resulting in poor small target detection performance. To this end, this application uses a learnable transposed convolution to replace bilinear interpolation. The details are restored and high-frequency information is retained through adaptive inverse convolution kernel parameters. At the same time, the encoder's low-level high-resolution features (such as leaf texture) are skipped with the decoder's high-level semantic features (such as plant distribution) to achieve cross-layer feature fusion with multi-scale feature complementarity, compensating for the spatial information loss that may occur during the encoding and fusion stages. This avoids the boundary blurring problem caused by the dense distribution of invasive plants and amplifies the local response of scattered invasive plant seedlings, enhancing the small target detection capability. In addition, a dual-branch encoder is designed to share the same decoder, so that multimodal features (visible light spatial details + multispectral spectral information) are uniformly decoded after fusion, enhancing feature consistency. This shared decoder design not only effectively reuses the fused features, but also reduces the parameter redundancy of the dual decoders, improves the inference speed of the dual-branch model, and is suitable for batch processing of large-scale remote sensing images.

[0053] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described below with reference to the accompanying drawings:

[0055] Figure 1 This is a flow chart of a multispectral and visible light remote sensing image fusion and segmentation method according to an embodiment of the present invention.

[0056] Figure 2 The figure is a schematic diagram of the structure of the multispectral and visible light remote sensing image fusion and segmentation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is an explanation and description of the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. However, the following embodiments are only preferred embodiments of the present invention and are not exhaustive. Based on the embodiments in the implementation manner, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0058] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0059] Example 1:

[0060] Please see the attached Figure 1 , Figure 1 The figure shows a flow chart of a multispectral and visible light remote sensing image fusion and segmentation method provided by one embodiment of the present specification.

[0061] like Figure 1 As shown, the multispectral and visible light remote sensing image fusion and segmentation method may at least include the following steps:

[0062] Step 101: Collect visible light data and multispectral data of the target detection area.

[0063] It can be understood that this embodiment ensures that the multi-spectral image of the target detection area is spatially aligned with the visible light image through resampling.

[0064] Furthermore, this embodiment can also perform preprocessing such as distortion correction, radiation correction, and image stitching on the collected visible light data and multispectral data to generate a complete map of the target detection area. The stitched complete map is then cut into 512×512 pixel image blocks and manually annotated using annotation tools. Since the distribution of invasive plants in images usually presents an irregular polygonal distribution, relying solely on manual outlining is inefficient. Therefore, this embodiment introduces the Segment Anything Model (SAM) for auxiliary annotation to construct a dataset containing multiple annotated images.

[0065] Step 102: construct a dual-branch model, which includes a visible light branch encoder, a multispectral branch encoder, a multi-scale feature fusion module, and a shared decoder.

[0066] Among them, the implementation method of the training and optimization operations of the dual-branch model in this embodiment is similar to the training and optimization operations of the image detection model in the prior art, which will not be elaborated in this embodiment. For example, the labeled image dataset is used as the training set, and a weighted combination of Dice loss (to solve category imbalance) and cross entropy loss is used as the loss function. The encoder loads the pre-trained ResNet34 weights to accelerate convergence, and uses the Adam optimizer (initial learning rate 1e-4, weight decay 1e-5) for optimization. If the verification loss does not decrease for 10 consecutive rounds, the training is terminated.

[0067] It is understandable that traditional dual-branch networks (such as Two-Stream CNN) are usually used for video or RGB-depth data. However, this embodiment designs an "independent encoder + multi-scale fusion strategy" to address the band differences between multispectral and visible light. Specifically, in the encoding stage, the dual-branch model sets two independent U-Net encoder branches, which process visible light data and multispectral data separately through independent encoding paths, and extract their spatial details and spectral features respectively, ensuring the retention of each modality-specific information, and solving the recognition limitations of existing technologies in complex backgrounds.

[0068] Step 103: Input the visible light data into the visible light branch encoder of the dual-branch model to extract a multi-scale spatial feature map.

[0069] Specifically, in this embodiment, 3-channel RGB image data is input, and the visible light branch encoder adopts a four-level downsampling structure. Each downsampling structure contains two 3×3 convolutional layers, batch normalization and ReLU activation function, and cooperates with 2×2 maximum pooling to achieve step-by-step compression of the feature map size. The number of output channels is 64, 128, 256, and 512 respectively.

[0070] Step 104 : input the multispectral data into the multispectral dual-branch encoder of the branch model to extract and obtain a multi-scale spectral feature map.

[0071] Specifically, in this embodiment, multispectral data is input into the multispectral branch encoder of the dual-branch model to extract multi-scale spectral feature maps, including:

[0072] Step 201: Perform a 3×3 convolution operation on each channel of the multispectral data independently to obtain a feature map after channel-by-channel convolution.

[0073] In this embodiment, the spectral features in each band (such as the chlorophyll response in the red edge band) are independently extracted through channel-by-channel convolution.

[0074] In step 202, a 1×1 convolution operation is performed on the feature map after channel-by-channel convolution to obtain a feature map after point-by-point convolution.

[0075] Next, this embodiment implements cross-band fusion (such as complementarity between near infrared and red edge) through point-by-point convolution to enhance discrimination.

[0076] In step 203, a four-level downsampling operation is performed on the feature map after point-by-point convolution to extract a multi-scale spectral feature map.

[0077] Among them, in this embodiment, the four-level downsampling structure of the multi-spectral branch encoder is consistent with the four-level downsampling structure adopted by the visible light branch encoder, and this embodiment will not be repeated here.

[0078] It can be understood that this embodiment combines sensor characteristics (such as band resolution) to carry out targeted design of the multispectral branch encoder, decomposing the existing standard convolution into channel-by-channel convolution (extracting spectral features) and point-by-point convolution (cross-channel fusion), while ensuring recognition accuracy. The computational complexity of the multispectral branch is significantly reduced, and the computational amount of the multispectral branch is reduced to 1 / 3 of the traditional convolution, which is suitable for edge computing devices (such as drones) to meet the real-time processing needs of drones.

[0079] It can be understood that this embodiment maintains the same downsampling level (4 levels) for the dual-branch encoder to ensure cross-modal feature scale alignment and provide a same-scale feature basis for subsequent operations.

[0080] In step 105, the multi-scale spatial feature map and the spectral feature map are input into the multi-scale feature fusion module, and the channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of the multi-spectral and visible light features. The weighted feature map is then channel-joined according to the scale, and the joined feature map is compressed to the original number of channels using 1×1 convolution. The channel attention weight is then applied to the compressed fusion feature map to obtain a multi-scale fusion feature sequence.

[0081] It can be understood that this embodiment independently performs a multi-scale feature fusion operation on the input feature map of each scale.

[0082] Specifically, in this embodiment, a channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of multispectral and visible light features, including:

[0083] Step 301: Generate channel attention weights based on the input feature map, perform channel-by-channel weighting on the input feature map to highlight the red edge and the near-infrared band, and obtain a first weighted feature map.

[0084] It can be understood that this embodiment uses multi-spectral bands (such as near-infrared) to penetrate the vegetation canopy to identify invasive plants under the forest, and uses the high reflectivity characteristics of the red edge band (730nm) to distinguish invasive plants from related species, avoiding interference from similar spectra, and greatly improving the adaptability of target detection in complex scenes.

[0085] Furthermore, in this embodiment, channel attention weights are generated based on the input feature map, and the input feature map is weighted channel by channel to highlight the red edge and the near-infrared band, thereby obtaining a first weighted feature map, including:

[0086] Perform global average pooling and maximum pooling on each channel of the input feature map to obtain two 1×1×C channel descriptors, where C represents the number of channels;

[0087] Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio.

[0088] Input two 1×1×C channel descriptors into a shared two-layer fully connected network, and use the Sigmoid function to generate the first channel attention weight;

[0089] The first channel attention weight is weighted with the input feature map channel by channel to highlight the red edge and near-infrared band to obtain the first weighted feature map.

[0090] For example, the specific implementation of this embodiment is as follows:

[0091] First, input the feature map F∈R^H×W×C, where H×W is the spatial dimension and C is the number of channels (corresponding to multi-spectral or visible light bands). Global average pooling and maximum pooling operations are performed on each channel c∈{1,2,...,C}. The global average pooling operation corresponds to the descriptor Zavg∈R^1×1×C, which reflects the global mean of each channel. The global maximum pooling operation corresponds to the descriptor Zmax∈R^1×1×C, which captures the significant feature peaks of each channel.

[0092] Secondly, a shared fully connected network is constructed. The input dimension of the first fully connected layer (channel compression) is 1×1×C, and the output dimension is 1×1×C / r, where r is the preset compression ratio. For example, r is set to 16 to balance computational complexity and performance. The activation function is ReLU to introduce nonlinearity and enhance expressiveness. The input dimension of the second fully connected layer (channel restoration) is 1×1×C / r, and the output dimension is 1×1×C.

[0093] Next, process the two descriptors separately, input Zavg and Zmax into the shared fully connected network in sequence, output the shared fully connected result Aavg of the descriptor Zavg and the shared fully connected result Amax of the descriptor Zmax respectively, add Aavg and Amax element by element, and generate the first channel attention weight through the Sigmoid function, and compress the first channel attention weight to the [0,1] interval;

[0094] Finally, the attention weight of the first channel is multiplied by the input feature map F channel by channel. If the weights of the red edge and near-infrared channels are higher and the weights of other channels (such as green light) are lower, the red edge and near-infrared bands can be highlighted and redundant bands can be suppressed to obtain the first weighted feature map F′.

[0095] It can be understood that this embodiment uses dual pooling fusion to combine global statistics and local significance to comprehensively characterize channel importance, avoid information bias of single pooling, and then reduce the number of parameters by sharing the fully connected network (only 2C2 / r parameter), improves computational efficiency, and adaptively adjusts the contribution of multispectral bands through the first-channel attention weight, providing an efficient and explainable spectral selection mechanism for core segmentation tasks (such as invasive plant identification).

[0096] Step 302: Generate spatial attention weights based on the first weighted feature map, perform spatial weighting on the first weighted feature map to highlight the distribution area of ​​invasive plants, and obtain a second weighted feature map.

[0097] Furthermore, in this embodiment, a spatial attention weight is generated based on the first weighted feature map, and the first weighted feature map is spatially weighted to highlight the distribution area of ​​invasive plants, thereby obtaining a second weighted feature map, including:

[0098] Perform global average pooling and maximum pooling on the first weighted feature map along the channel dimension to obtain two H×W×1 spatial descriptors, where H represents height and W represents width;

[0099] Two H×W×1 spatial descriptors are concatenated along the channel dimension;

[0100] Perform a 7×7 convolution operation on the splicing result and generate spatial attention weights after normalization using the Sigmoid function;

[0101] The spatial attention weight is multiplied pixel by pixel with the first weighted feature map to highlight the invasive plant distribution area, and the second weighted feature map is obtained.

[0102] For example, the specific implementation of this embodiment is as follows:

[0103] First, the first weighted feature map F′∈R^H×W×C is input, in which key bands (such as red edge and near infrared) have been enhanced through channel attention. Global average pooling and maximum pooling operations are performed on each spatial position along the number of channels (C). The global average pooling operation corresponds to the descriptor Savg∈R^H×W×1, which reflects the global average response intensity of each spatial position. The global maximum pooling operation corresponds to the descriptor Smax∈R^H×W×1, which captures the significant feature peaks of each spatial position.

[0104] Secondly, Savg and Smax are concatenated along the channel dimension to form the concatenation result Sconcat∈R^H×W×2, which combines global distribution and local significance to fully represent the importance of spatial positions.

[0105] Next, a 7×7 convolution operation is performed on the concatenated result Sconcat to output a single-channel spatial attention map. This map captures local contextual relationships (such as the continuous distribution pattern of densely populated areas) through a large receptive field. The single-channel spatial attention map is normalized using the Sigmoid function to generate spatial attention weights, which are then compressed to the [0, 1] interval.

[0106] Finally, the spatial attention weight is multiplied by the first weighted feature map F′ position by position. If the weight of the spatial location with dense plant distribution (such as the center of the invasive plant canopy) is close to 1, and the weight of the background area such as soil and shadow is close to 0, the distribution area of ​​invasive plants can be highlighted, and the second weighted feature map is obtained.

[0107] It is understandable that traditional segmentation models in the prior art rely on the local receptive field of convolution, making it difficult to globally perceive target distribution patterns (such as the clustering characteristics of invasive plants). They are also sensitive to background interference. Complex backgrounds (such as dead grass and soil texture) that are similar to the target spectrum can easily lead to missegmentation. At the same time, dense plant areas are easily ignored by the model due to feature overlap, resulting in a high missed detection rate. To this end, based on the first weighted feature map after the first channel attention weighting, this embodiment first uses global average pooling to characterize global distribution features to identify large continuous areas (such as patches of invasive plants). It also uses global maximum pooling to capture local significant peaks and locate sparse but highly responsive targets (such as scattered seedlings), addressing the problem of single pooling's incomplete representation of spatial information. Subsequently, it breaks through the local limitations of traditional small convolution kernels and uses 7×7 convolution to cover a larger spatial range, capturing the continuous distribution patterns of dense plant areas (such as canopy cluster structure), enhancing the spatial continuity perception of dense targets. The spatial attention weights are then adaptively adjusted through data-driven, dynamically adapting to input features to avoid over-segmentation or under-segmentation caused by fixed thresholds.

[0108] It can be understood that in this embodiment, the weighted feature map is spliced ​​according to the scale, and the 1×1 convolution is used to compress the spliced ​​feature map to the original number of channels. The implementation method is:

[0109] First, the feature maps of the same scale (such as 64×64, 128×128) in the second weighted feature map are spliced ​​according to the channel dimension to retain spectral differences and spatial consistency, forming a fused feature map containing cross-modal information. Then, a 1×1 convolution is used to compress the spliced ​​feature map to the original number of channels (such as 64→64) to eliminate redundant information. This not only reduces the computational complexity but also achieves the unification of feature dimensions and the preliminary integration of cross-modal features. The fused feature sequence still maintains a multi-scale structure (such as 64→64, 128→128, 256→256, 512→512) for subsequent unified decoding.

[0110] Specifically, in this embodiment, channel attention weights are applied to the compressed fusion feature map to obtain a multi-scale fusion feature sequence, including:

[0111] Perform global average pooling on each channel of the compressed fusion feature map to obtain the channel statistical descriptor;

[0112] Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio.

[0113] The channel statistical descriptor is input into a shared two-layer fully connected network, and the Sigmoid function is used to generate the second channel attention weight;

[0114] Weight the second channel attention weight and the compressed fusion feature map channel by channel to obtain the third weighted feature map;

[0115] The third weighted feature maps of all scales constitute a multi-scale fusion feature sequence.

[0116] Existing studies mostly use the attention mechanism for single-modal feature enhancement. Even the existing multi-scale fusion technology only integrates features through simple splicing or addition, without considering the differences in the importance of feature channels at different scales, resulting in redundant information interfering with model judgment. In addition, there may be channel response conflicts between low-level detail features (such as leaf edges) and high-level semantic features (such as plant distribution). Direct fusion is prone to introduce noise. At the same time, key bands (such as red edges) are easily diluted by non-key channels (such as green light) after existing multi-scale fusion, affecting segmentation accuracy. To this end, this embodiment overcomes the gradient propagation problem of the attention mechanism in multi-modal interaction by designing multi-scale feature fusion for the spectral characteristics and morphological characteristics of invasive plants. Specifically, by applying channel- The spatial joint attention weight is used to dynamically adjust the weight distribution of multispectral and visible light features to resolve feature conflicts between modalities. The weighted feature maps are then channel-spliced ​​according to scale to retain spectral differences and spatial consistency, forming a fused feature map containing cross-modal information. 1×1 convolution is then used to perform channel compression and alignment on the spliced ​​feature maps, and the compressed fused feature maps are subjected to secondary channel screening to resolve cross-scale feature redundancy, thereby solving the problem of feature redundancy and weakened discriminability in existing multi-scale fusion methods, forming a complete technical chain of "modal alignment → scale fusion → channel optimization", which can significantly improve the segmentation accuracy and efficiency of the dual-branch model for invasive plants in complex scenarios, and provide reliable technical support for the precise monitoring of invasive plants.

[0117] Step 106: Input the multi-scale fusion feature sequence into a shared decoder, parse the multi-scale fusion feature sequence, and output an invasive plant segmentation mask image of the target detection area.

[0118] It can be understood that this embodiment reuses dual-branch features through a shared decoder, reduces redundant calculations, and improves inference speed.

[0119] Specifically, in this embodiment, the multi-scale fusion feature sequence is input into the shared decoder, the multi-scale fusion feature sequence is parsed, and the invasive plant segmentation mask image of the target detection area is output, including:

[0120] Perform four-level upsampling operations on the multi-scale fusion feature sequence. A 3×3 transposed convolution operation is performed on each level. After each level of upsampling operation, a jump connection is performed with the feature map of the corresponding scale in the encoder, and then feature analysis is performed on the feature map after the jump connection.

[0121] Based on the parsed feature map, a segmentation mask image of invasive plants in the target detection area is generated.

[0122] For example, the specific implementation of this embodiment is as follows:

[0123] 1. Upsampling and skip connections

[0124] (1) First level upsampling (32×32→64×64)

[0125] Transposed convolution: Use a 3×3 transposed convolution kernel, stride 2, padding 1, and an output size of 64×64×256.

[0126] Parameters: Input channels 512 → Output channels 256, parameter quantity = 3 × 3 × 256 × 512 = 1,179,648.

[0127] Skip connection: Extract the feature map from the third level of the encoder (corresponding to a resolution of 64×64×256) and concatenate it with the upsampled feature map (64×64×256) along the channel dimension to obtain 64×64×512.

[0128] (2) Second level upsampling (64×64→128×128)

[0129] Transposed convolution: 3×3 transposed convolution (input 256 → output 128), output 128×128×128.

[0130] Skip connection: concatenated with the encoder level 2 features (128×128×128) to obtain 128×128×256.

[0131] (3) Level 3 upsampling (128×128→256×256)

[0132] Transposed convolution: 3×3 transposed convolution (input 128 → output 64), output 256×256×64.

[0133] Skip connection: concatenated with the encoder level 1 features (256×256×64) to obtain 256×256×128.

[0134] (4) Level 4 upsampling (256×256→512×512)

[0135] Transposed convolution: 3×3 transposed convolution (input 64 → output 32), output 512×512×32.

[0136] 2. Feature analysis, each level of analysis operation is:

[0137] (1) Input: Feature map after skip connection (e.g. 64×64×512).

[0138] (2) Operation sequence:

[0139] Convolution 1: 3×3 convolution (input 512→512), stride 1, padding 1→BN→ReLU.

[0140] Convolution 2: 3×3 convolution (input 512→512), stride 1, padding 1→BN→ReLU.

[0141] Thus, a high-resolution segmentation map is gradually reconstructed.

[0142] (3) Output: parsed feature map (e.g., 64×64×512).

[0143] 3. Output layer

[0144] (1) Input: Final upsampled feature map (512×512×32).

[0145] (2) 1×1 convolution:

[0146] Convolution kernel: 1×1×32×2, parameter count = 1×1×32×2 = 64, used to map high-dimensional features into binary masks (invasive plants vs. background) to achieve pixel-level semantic segmentation results.

[0147] (3) Softmax activation: Normalize each pixel and generate a probability map.

[0148] (4) The output result is a binary mask with a size of 512×512×2, where each pixel value represents the probability of belonging to an invasive plant or background.

[0149] It is understandable that traditional decoders typically rely on simple interpolation upsampling, which can easily lead to blurred segmentation boundaries (such as the loss of jagged leaf edges). Furthermore, features from different encoder levels are not effectively fused, and deep features lack low-level spatial details, resulting in poor small target detection performance. Therefore, this embodiment uses learnable transposed convolution instead of bilinear interpolation. The parameters of the inverse convolution kernel are adaptively adjusted to restore details and preserve high-frequency information. At the same time, low-level high-resolution features from the encoder (such as leaf texture) are skipped with high-level semantic features from the decoder (such as plant distribution). This achieves cross-layer feature fusion with complementary multi-scale features, compensating for spatial information loss that may occur during the encoding and fusion stages. This avoids the blurred boundaries caused by densely distributed invasive plants and amplifies the local responses of scattered invasive plant seedlings, enhancing small target detection capabilities. Furthermore, a dual-branch encoder is designed to share the same decoder, allowing multimodal features (visible light spatial details + multispectral spectral information) to be uniformly decoded after fusion, enhancing feature consistency. This shared decoder design not only effectively reuses fused features, but also reduces parameter redundancy in the dual decoders, improving the inference speed of the dual-branch model and making it suitable for batch processing of large-scale remote sensing images.

[0150] The following experiments are conducted to verify the beneficial effects of the embodiments of this specification:

[0151] This experiment used the PyTorch deep learning framework for model development and training. Servers equipped with NVIDIA A6000 GPUs were used to perform large-scale image segmentation tasks, supporting efficient computing and accelerated optimization for complex network structures. A systematic evaluation of the performance of three mainstream semantic segmentation models, U-Net, DeepLabV3+, and PSPNet, on a remote sensing image dataset of spring fleabane was conducted. The dual-branch model proposed in this example is denoted as ADBU-Net (Attention-enhanced DualBranch U-Net). To ensure fairness in the experiments, all models used a pre-trained ResNet-50 as the backbone network. Four mainstream evaluation metrics were used for performance comparison: IoU (Intersection over Union, a measure of segmentation accuracy), Precision (a measure of segmentation accuracy), Recall (a measure of the proportion of successful detections by the model), and F1-score (a measure of classification reliability). The experimental results are summarized in Table 1.

[0152] Table 1 Model prediction results based on visible light and multispectral

[0153]

[0154] As can be seen from Table 1, the ADBU-Net proposed in this example achieves the best results in all indicators. Its IoU reaches 80.31% and F1-score is 96.07%, indicating that it has strong robustness and accuracy in identifying the spring fleabane area. The Precision is 96.76%, indicating that its false detection rate is very low and it can effectively exclude non-target areas. The Recall is 95.76%, indicating that its overall coverage capability in the target area is strong.

[0155] Example 2:

[0156] Please see the attached Figure 2 , Figure 2 This is a schematic diagram of the structure of a multispectral and visible light remote sensing image fusion and segmentation system provided in one embodiment of this specification.

[0157] like Figure 2 As shown, the multispectral and visible light remote sensing image fusion and segmentation system may at least include:

[0158] Data acquisition module 1, used to collect visible light data and multispectral data of the target detection area;

[0159] Model construction module 2, used to construct a dual-branch model, the dual-branch model includes a visible light branch encoder 3, a multispectral branch encoder 4, a multi-scale feature fusion module 5 and a shared decoder 6;

[0160] A visible light branch encoder 3, configured to input visible light data to extract a multi-scale spatial feature map;

[0161] A multispectral branch encoder 4 is used to input multispectral data to extract a multi-scale spectral feature map;

[0162] Multi-scale feature fusion module 5 is used to apply channel-space joint attention weights to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features, then perform channel splicing on the weighted feature map according to the scale, use 1×1 convolution to compress the spliced ​​feature map to the original number of channels, and then apply channel attention weights to the compressed fused feature map to obtain a multi-scale fused feature sequence;

[0163] The shared decoder 6 is used to parse the multi-scale fusion feature sequence and output an invasive plant segmentation mask image in the target detection area.

[0164] It is understandable that the technical concept of the multispectral and visible light remote sensing image fusion and segmentation system provided in this embodiment is similar to the technical concept of the aforementioned multispectral and visible light remote sensing image fusion and segmentation method, and this embodiment will not be repeated here.

[0165] Example 3:

[0166] Another embodiment of the present disclosure provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of the aforementioned method embodiments. If the components of the aforementioned electronic device are implemented as software functional units and used as independent downstream task predictions or tasks, they can be stored in the computer-readable storage medium.

[0167] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0168] The above description is merely a description of the preferred embodiments disclosed in this application and the technical principles employed. Those skilled in the art should understand that the scope of protection provided by this disclosure is not limited to technical solutions formed by a specific combination of the aforementioned technical features, but also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents without departing from the scope of the disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A multispectral and visible light remote sensing image fusion and segmentation method, characterized in that: The following steps are involved: Collect visible light data and multispectral data of the target detection area; Constructing a dual-branch model, the dual-branch model includes a visible light branch encoder, a multispectral branch encoder, a multi-scale feature fusion module, and a shared decoder; The visible light data is input into the visible light branch encoder of the dual-branch model to extract a multi-scale spatial feature map; Perform 3×3 convolution operation on each channel of the multispectral data independently to obtain the feature map after channel-by-channel convolution; Perform 1×1 convolution operation on the feature map after channel-by-channel convolution to obtain the feature map after point-by-point convolution; Perform four-level downsampling on the feature map after point-by-point convolution to extract multi-scale spectral feature maps; The multi-scale spatial feature map and spectral feature map are input into the multi-scale feature fusion module. The channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features. The weighted feature map is then channel-jointed according to the scale. The joined feature map is compressed to the original number of channels using 1×1 convolution. The channel attention weight is then applied to the compressed fusion feature map to obtain a multi-scale fusion feature sequence. The multi-scale fusion feature sequence is input into the shared decoder, the multi-scale fusion feature sequence is parsed, and the invasive plant segmentation mask image of the target detection area is output; Among them, the channel-space joint attention weight is applied to the input feature map to dynamically adjust the weight distribution of multispectral and visible light features, including: Perform global average pooling and maximum pooling on each channel of the input feature map to obtain two 1×1×C channel descriptors, where C represents the number of channels; Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio. Input two 1×1×C channel descriptors into a shared two-layer fully connected network, and use the Sigmoid function to generate the first channel attention weight; The first channel attention weight is weighted with the input feature map channel by channel to highlight the red edge and near-infrared band, and the first weighted feature map is obtained; Perform global average pooling and maximum pooling on the first weighted feature map along the channel dimension to obtain two H×W×1 spatial descriptors, where H represents height and W represents width; Two H×W×1 spatial descriptors are concatenated along the channel dimension; Perform a 7×7 convolution operation on the splicing result and generate spatial attention weights after normalization using the Sigmoid function; The spatial attention weight is multiplied pixel by pixel with the first weighted feature map to highlight the invasive plant distribution area, and the second weighted feature map is obtained.

2. The multispectral and visible light remote sensing image fusion and segmentation method according to claim 1, characterized in that: Apply channel attention weights to the compressed fusion feature map to obtain a multi-scale fusion feature sequence, including: Perform global average pooling on each channel of the compressed fusion feature map to obtain the channel statistical descriptor; Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio. The channel statistical descriptor is input into a shared two-layer fully connected network, and the Sigmoid function is used to generate the second channel attention weight; Weight the second channel attention weight and the compressed fusion feature map channel by channel to obtain the third weighted feature map; The third weighted feature maps of all scales constitute a multi-scale fusion feature sequence.

3. The multispectral and visible light remote sensing image fusion and segmentation method according to claim 1, wherein: Each level of downsampling operation sequentially performs two 3×3 convolution operations, a batch normalization operation, a ReLU activation operation, and a 2×2 maximum pooling operation.

4. The multispectral and visible light remote sensing image fusion and segmentation method according to claim 1, wherein: The multi-scale fusion feature sequence is input into the shared decoder, the multi-scale fusion feature sequence is parsed, and the invasive plant segmentation mask image of the target detection area is output, including: Perform four-level upsampling operations on the multi-scale fusion feature sequence. A 3×3 transposed convolution operation is performed on each level. After each level of upsampling operation, a jump connection is performed with the feature map of the corresponding scale in the encoder, and then feature analysis is performed on the feature map after the jump connection. Based on the parsed feature map, a segmentation mask image of invasive plants in the target detection area is generated.

5. Multispectral and visible light remote sensing image fusion and segmentation system, characterized by: include: Data acquisition module, used to collect visible light data and multispectral data of the target detection area; A model construction module is used to construct a dual-branch model, wherein the dual-branch model includes a visible light branch encoder, a multispectral branch encoder, a multi-scale feature fusion module and a shared decoder; A visible light branch encoder, configured to input visible light data to extract a multi-scale spatial feature map; A multispectral branch encoder, used for inputting multispectral data to extract multi-scale spectral feature maps; The multi-scale feature fusion module is used to apply channel-space joint attention weights to the input feature map to dynamically adjust the weight distribution of multi-spectral and visible light features. The weighted feature map is then channel-jointed according to the scale. The joined feature map is compressed to the original number of channels using 1×1 convolution. The channel attention weights are then applied to the compressed fused feature map to obtain a multi-scale fused feature sequence. A shared decoder is used to parse the multi-scale fusion feature sequence and output an invasive plant segmentation mask image of the target detection area; The multispectral branch encoder is used to perform the following steps: Perform 3×3 convolution operation on each channel of the multispectral data independently to obtain the feature map after channel-by-channel convolution; Perform 1×1 convolution operation on the feature map after channel-by-channel convolution to obtain the feature map after point-by-point convolution; Perform four-level downsampling on the feature map after point-by-point convolution to extract multi-scale spectral feature maps; Among them, the multi-scale feature fusion module is used to perform the following steps: Perform global average pooling and maximum pooling on each channel of the input feature map to obtain two 1×1×C channel descriptors, where C represents the number of channels; Build a shared two-layer fully connected network. The number of channels of the first layer is compressed to C / r, and the number of channels of the second layer is C, where r is the preset compression ratio. Input two 1×1×C channel descriptors into a shared two-layer fully connected network, and use the Sigmoid function to generate the first channel attention weight; The first channel attention weight is weighted with the input feature map channel by channel to highlight the red edge and near-infrared band, and the first weighted feature map is obtained; Perform global average pooling and maximum pooling on the first weighted feature map along the channel dimension to obtain two H×W×1 spatial descriptors, where H represents height and W represents width; Two H×W×1 spatial descriptors are concatenated along the channel dimension; Perform a 7×7 convolution operation on the splicing result and generate spatial attention weights after normalization using the Sigmoid function; The spatial attention weight is multiplied pixel by pixel with the first weighted feature map to highlight the invasive plant distribution area, and the second weighted feature map is obtained.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multispectral and visible light remote sensing image fusion and segmentation method according to any one of claims 1 to 4 is implemented.

Images