A multi-modal unmanned aerial vehicle image segmentation method

Abstract

This invention discloses a multimodal UAV image segmentation method, comprising the following steps: Step 1: Constructing a UAV image semantic segmentation base network; Step 2: Capturing high-resolution local features at shallow layers using a Convolutional Neural Network (CNN) module, extracting deep features from the image through a multi-head self-attention mechanism in a transformer module, and fusing the local features with deep global features through skip connections; Step 3: Constructing a parallel convolution module to optimize the fused features from Step 2; Step 4: Combining the feature extraction results from Steps 2 and 3, fusing the feature information through a cross-attention mechanism to improve segmentation performance in complex scenes; Step 5: Constructing a dual attention module to optimize the processing of local and global information in the image. This invention features efficient fusion of global and local features, accurate segmentation of multi-scale targets, and significantly improved robustness in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of semantic segmentation technology for UAV images, and specifically relates to a multimodal UAV image segmentation method. Background Technology

[0002] Target distribution in UAV images often exhibits local density, with each target rotating in any direction from a top-down perspective. Furthermore, significant scale differences exist between different instances, and the images are susceptible to variations in lighting, shadows, nighttime conditions, and weather conditions (such as clouds and fog). These complex factors pose challenges to semantic segmentation of UAV images, hindering the application of general segmentation algorithms in UAV imagery. To more accurately represent the feature information of UAV images, some researchers have made specific improvements to convolutional neural networks (CNNs) tailored to the characteristics of UAV images. While some success has been achieved, the limitations of CNNs in feature extraction hinder their further development. For example, their insufficient global modeling capabilities limit the segmentation of large-scale targets; simultaneously, the weak correlation of image context information leads to low segmentation accuracy or even failure to complete segmentation.

[0003] Currently, most UAV image semantic segmentation networks based on pure convolutional structures have certain limitations due to a lack of global information; while detection networks based on transformer structures, although possessing strong long-range dependency modeling capabilities, are relatively weak in local feature extraction. Therefore, both of these architectures have some shortcomings when dealing with multi-scale scenarios in UAV image semantic segmentation. Summary of the Invention

[0004] To overcome the shortcomings of the existing technologies, the present invention aims to provide a multimodal UAV image segmentation method. This method combines a novel multimodal semantic segmentation network that integrates convolution and transformer series-parallel connections. It features efficient fusion of global and local features, accurate segmentation of multi-scale targets, significantly improved robustness in complex scenarios (such as low light, occlusion, and severe weather conditions), and high model parameter efficiency and low computational complexity.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multimodal UAV image segmentation method includes the following steps;

[0007] Step 1: Introduce a hybrid encoder on the basis of the UNet model, and fuse convolution and transformer to form a feature extraction framework to build a base network for semantic segmentation of UAV images;

[0008] Step 2: Step 2 is a further refinement and implementation based on the framework proposed in Step 1. It details how to fuse convolution and transformers using a U-shaped structure to construct a specific segmentation network, RGB-TransUNet.

[0009] The convolutional neural network (CNN) module captures high-resolution local features in the shallow layers of the network in the first few layers, and then fuses these local features with global features extracted from deep features of the image through a multi-head self-attention mechanism in the transformer module via skip connections.

[0010] Step 3: The features were further optimized from local and global perspectives by using a parallel convolution module and a multi-head self-attention mechanism. The parallel convolution module extracted detailed local information, and the multi-head self-attention mechanism captured long-distance global dependencies. The missing local information of the transformer in Step 2 was supplemented, and the fused features in Step 2 were optimized.

[0011] Step 4: Based on the local and global features (including features of RGB and T images) extracted in Step 2 and Step 3, the feature information is fused into RGB-T multimodal modes through the cross attention mechanism (CA) module to improve the segmentation effect in complex scenes;

[0012] Step 5: Optimize the feature representation of the basic feature extraction framework built in Steps 1 and 2 by combining convolution and transformers, constructing a dual attention module (PCAM-Block) to integrate location and channel feature information, thereby optimizing the local and global information processing of the image. PCAM-Block is closely related to the previous steps; it is a module that further optimizes feature fusion within the entire network structure.

[0013] The UAV image semantic segmentation network is used to extract global and local features from UAV images, process multi-scale targets, improve segmentation accuracy in complex environments, and achieve information fusion in multimodal images, namely RGB images and T images.

[0014] The UAV image semantic segmentation network includes a convolutional module, a Transformer block, a cross-attention module, and a PCAM module. The convolutional module is at the front end of the network and is used for preliminary local feature extraction of the input image (including RGB and T images). It is fused with the output of the deep transformer in the skip connection. The Transformer block fuses the convolution and transformer and achieves accurate modeling of targets at different scales through multi-level feature transfer and fusion. The cross-attention module integrates complementary information from RGB and T images to ensure the collaborative processing of multimodal images. The PCAM module enhances spatial and channel features to ensure that targets of different scales and categories can be accurately segmented.

[0015] Input to the drone image semantic segmentation network:

[0016] RGB images: ordinary visible light images from drones, containing color and detail information of the scene, typically used to capture visual features in the image (such as buildings, roads, vehicles, etc.);

[0017] T-image (thermal imaging image): An infrared image acquired by a drone using thermal imaging equipment, typically used for semantic segmentation in low-light, complex background, and other environments. It provides thermal radiation information of objects, complementing the RGB image. These two images (RGB and T) are used as multimodal inputs to the network for feature extraction and processing.

[0018] Output of the UAV image semantic segmentation network:

[0019] Segmentation Mask: An image with the same dimensions as the input image, where each pixel corresponds to a category label in the image.

[0020] For example:

[0021] Each pixel can be classified into different categories such as buildings, roads, vehicles, and trees. The segmentation network assigns the category to each pixel based on the input RGB and T images. The output mask can be a two-dimensional array, with values ​​representing the category to which the pixel belongs.

[0022] In step 1, the UAV image semantic segmentation base network built based on the UNet model extracts local and global features from the input image through the combination of convolution and transformer. These features are fused through U-shaped structure and skip connections to ensure the preservation of image features at different scales (especially small targets and large-scale backgrounds).

[0023] Local feature extraction is used to accurately capture small targets and details in images, while global feature extraction ensures the segmentation accuracy of large-scale targets and backgrounds.

[0024] In step 2, the feature extraction network is divided into four stages. The first two stages use convolution to extract local features, and the last two stages use transformers for global modeling. The two are fused through skip connections. In the first two stages, in order to make full use of the efficiency of convolution in capturing shallow local features and the advantage of transformers in global modeling of deep global features, ResNetV2 convolutional blocks are used.

[0025] F1=”ResNetV”“2”1(X) (1)

[0026] F2=”ResNetV”“2”2(F1) (2)

[0027] Next, in the third stage, namely the Patch Embedding stage, convolutional operations are used to flatten and position the input, and dropout is applied to finally generate a one-dimensional vector containing positional information.

[0028] F3="Flatten"("Conv"(F2)) (3)

[0029] F3=”PositionEmbedding”(F3) (4)

[0030] F3 = "Dropout" (F3) (5)

[0031] In the final stage, a TransformerBlock containing a parallel convolutional module (CNT) is used to increase the transformer's ability to perceive local information. After repeating 12 times, a one-dimensional vector of the same size as the original is output.

[0032] T0 = ​​F3 (6)

[0033] T i ="TransformerBlock"(T i-1 )”for”i=1, 2,…,12 (7)

[0034] TransformerBlock(T)="LN"(T+"CNT"("MHSA"("LN"(T)))) (8)

[0035] LN stands for Layer Normalization, and MHSA is an abbreviation for Multi-Head Self-Attention, which is used to capture the dependencies between different positions in the input sequence. MHSA can process information from different representation spaces simultaneously through parallel multi-head self-attention mechanisms, thereby improving the model's representational power.

[0036] The features obtained in the four stages are as follows: "Embeddings" = [F1, F2, F3, T] 12 ];

[0037] Embeddings refer to the embedded features after network feature extraction, including the combination of shallow convolutional features and deep transformer features. By fusing features from different stages, accurate modeling of targets at different scales in the image is ensured.

[0038]

[0039] F1 represents the first layer of features obtained through convolution, with a spatial resolution one-quarter of the input image resolution and a feature dimension of C1. Here, H and W are the height and width of the input image, respectively, and C1 is the number of feature channels in this layer;

[0040] F2 represents the second layer of features obtained through convolution, with a spatial resolution one-sixteenth of the input image resolution and a feature dimension of C2. Here, H and W are the height and width of the input image, respectively, and C2 is the number of feature channels in this layer;

[0041] F3 represents the third layer of features obtained through deep convolution and transformer operations. Its spatial resolution is also one-sixteenth of the input image. After flattening, the embedding dimension is C3. Here, HW represents the total number of pixels in the input image, and C3 is the number of channels in the deep features;

[0042] T 12 This represents the features processed by the transformer module (including the multi-head self-attention mechanism and the parallel convolution module), and its embedding dimension is the same as that of F3: This layer of features is used to enhance global feature modeling, especially for modeling long-distance dependencies of multi-scale targets.

[0043] In step 3, let f be the i-th transformer block. i The input features are:

[0044]

[0045] For parallel convolutional modules and multi-head self-attention mechanisms, f i Since they share the same feature input, the parallel convolutional module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f. i Then, the features are processed differently in different branches.

[0046] f i It has two branches, one of which is a multi-head self-attention mechanism used in f i One branch models long-range global information; the other branch is a parallel convolution module used to extract f. i The multi-head self-attention mechanism is applied to one-dimensional sequences, while the parallel convolution module is applied to two-dimensional images.

[0047] In the multi-head self-attention mechanism, the input label f ​​is first projected using a projection matrix. iProjecting onto the query (Q), key (K), and value (V), i.e., Q, K, V = f i W Q f i W K f i W V ,in Let Q, K, and V represent the projection matrices, respectively. The self-attention operation is calculated as follows:

[0048]

[0049] K T : Represents the transpose operation of the key vector, used to perform a dot product with the query vector Q, calculating the relevance between each query vector and all key vectors, with dimension R. D×N ;

[0050] This represents the Scaled Dot-Product Attention mechanism, for QK T To scale, use Used as a scaling factor to balance the dot product results It is the scaling factor of the dot product result, where D is the dimension of the query and key vectors;

[0051] Softmax is a normalization function that transforms the dot product of the input into a probability distribution. Through the softmax function, the correlation values ​​between the query vector and each key vector are converted into weighted coefficients, which are used to weight the vector V. This ensures that for each query vector, the weighted sum of all key vectors is 1.

[0052] The module iterates h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of self-attention heads. Features from the h attention heads are concatenated along the channel dimension to form the output of the MHSA. After passing through the MHSA, f is obtained. i Global information, represented as: f i g =MHSA(f i );

[0053] In the parallel convolution module, firstly, the input features of the transformer module are flattened into a one-dimensional sequence and then converted back into the corresponding two-dimensional image format (seq2img). Then, spatial local information is extracted through convolution operations, and the convolution result is converted back into a one-dimensional sequence (img2seq) to obtain f. i Local information f i l , represented as:

[0054] f il =CNT i (f i =img2seq(Conv i (seq2img(f i (12)

[0055] CNT i (·) represents a parallel convolution module, which consists of an imq2seq(·), three consecutive convolutional layers, and a seq2img(·) operation. This operation integrates the label features with local context information.

[0056] By directly adding the global features generated by the multi-head self-attention module (MHSA) and the local features extracted by the parallel convolutional module (CNT), the fusion of global and local information is achieved, as shown in the following formula:

[0057] f i gl =f i g +f i l (13)

[0058] f i gl : The fusion feature in the i-th transformer block, f i g f represents the global features extracted through the transformer's self-attention mechanism (MHSA). i l This represents the local features extracted by the parallel convolutional module (CNT).

[0059] In step 4, the cross-attention mechanism is not only a direct fusion of information from the RGB and T images, but also an effective combination of information from the two modalities based on the local and global features extracted in steps 2 and 3; specifically:

[0060] Fusion of local features: Local features extracted by convolutional networks and parallel convolutional modules, namely the detailed features (such as edges, textures, etc.) in RGB and T images, can be combined through a cross-attention mechanism, so that the detailed information of the RGB image and the supplementary information captured in the T image in low light or complex background can work together.

[0061] Global feature fusion: Global features extracted from transformer blocks (such as large-scale background and long-distance dependencies) are complemented between modalities through a cross-attention mechanism. RGB images provide global information under visible light conditions, while T images enhance global background understanding in low-light environments.

[0062] Cross-attention mechanism: The cross-attention mechanism identifies relevant parts in the RGB and T images by associating queries in the RGB image with keys and values ​​in the T image. Queries in the T image are also associated with keys and values ​​in the RGB image, achieving bidirectional information fusion.

[0063] The calculation process for cross-attention is as follows:

[0064]

[0065] Query: A representation from an RGB image;

[0066] Key and Value: Feature representations derived from another modality T image. Through cross attention, the query features of the RGB image are multiplied by the key and value features of the T image to find the parts of the T image that are related to the RGB image, and vice versa.

[0067] Q RGB : Represents the query vector extracted from the RGB image, K T : Represents the key vector extracted from image T, where d is the dimension of the key vector, and V T : Represents the value vector in the T-image.

[0068] f RGB-T =CA(f i (15)

[0069] Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA:

[0070] f i =f i gl +f RGB-T (16)

[0071] The fused features are processed by a feedforward neural network (FFN) and then input into the next transformer block.

[0072] f i+1 =FFN(f i )+f i (17)

[0073] f i+1 This represents the input features of the (i+1)th layer transformer block, or the output features after processing by the current layer (the i-th layer). This is the result of fusing global and local features and processing by a feedforward neural network (FFN), and also includes information from residual connections.

[0074] FFN(f i The 'f' represents the feature processed by a Feed Forward Network (FFN). The FFN is an important component of the transformer module, typically consisting of two fully connected layers and a nonlinear activation function. The FFN is used to process the fused feature f. i Further feature extraction and dimensionality transformation are performed to enhance the model's expressive power. FFN's role is to perform non-linear transformations on local and global features, further improving the model's modeling ability.

[0075] In steps 1 and 2, the network constructs a hybrid feature extraction framework based on convolution and transformers. The goal of these two steps is to extract local features through convolution and global features through transformers. These features are then passed to subsequent steps for processing. In step 3, the local features extracted by the parallel convolution module are fused with the features extracted by the PCAM-Block through skip connections, enhancing segmentation accuracy under complex backgrounds and low-light conditions. Positional attention and channel attention mechanisms ensure optimal representation of the fused features. In step 5, the dual attention module (PCAM-Block) further optimizes these features extracted through convolution and transformers. The feature fusion module receives the local and global features extracted in step 2 and further enhances the expressive power of the features through the positional attention module (PAM) and channel attention module (CAM).

[0076] In step 5, a dual attention module (PCAM-Block) is constructed for feature extraction. The dual attention module serves as a feature extractor and includes a positional attention module (PAM) and a channel attention module (CAM), which are responsible for extracting positional and channel features, respectively.

[0077] The Position Attention (PAM) module can capture the spatial dependency between any two points in the feature map and update specific features by weighted summation of all position features, where the weights are calculated based on the feature similarity between two positions.

[0078] Input local features A∈R C×H×W Where C is the number of channels, H is the height, and W is the width, three new feature maps are obtained through convolution: B, C, D ∈ R. C×H×W Reshape B and C into R C×N Where N = H × W represents the number of pixels, matrix multiplication is performed. The elements in matrix S are normalized to attention weights using the softmax function:

[0079]

[0080] Reshape matrix D into R C×N Perform matrix multiplication Reshape the results into R C×H×W Multiply the result by the parameter, and then sum the elements with feature A to obtain the final output:

[0081]

[0082] The weight α is initially set to 0 and is learned gradually through training.

[0083] In the aforementioned CAM (channel attention), the original feature A∈R is directly applied... C×H×W Remodeling into R C×N Then perform matrix multiplication between A and its transpose. Next, a softmax layer is applied to obtain the channel attention map X∈R. C×C :

[0084]

[0085] Perform matrix multiplication between the transpose of X and A. Reshape the result Y into R C×H×W Multiply the result Y by the scale parameter β, and then perform an element-wise summation operation with the original feature A to obtain the final output E∈R. C×H×W :

[0086] E=β·Y+A (21)

[0087] Similar to α, β is also learned through a training process.

[0088] The beneficial effects of this invention are:

[0089] 1. By fusing convolutional neural networks (CNNs) and transformers in a series and parallel manner, the limitations of traditional CNNs in global modeling capabilities are overcome, while the shortcomings of transformers in local feature extraction are addressed. This hybrid structure can better handle multi-scale targets and complex scenes in UAV images, effectively improving the global consistency of segmentation and the preservation of local details.

[0090] The results improved the model's ability to segment targets at multiple scales, complex backgrounds, and from multiple angles, especially the recognition accuracy of small and large targets.

[0091] 2. By using the skip connection mechanism of the U-Net structure, convolutional features and transformer global modeling features are combined at different network layers, which can effectively fuse shallow detail information with deep semantic information, making it particularly suitable for small target detection in UAV images and scenes with complex backgrounds.

[0092] The results enhanced the network's ability to handle small targets and complex backgrounds, enabling the model to more accurately distinguish between foreground and background in UAV imagery and improving overall segmentation performance.

[0093] 3. The parallel convolution module complements the transformer's shortcomings in local feature extraction, especially since convolution can effectively capture spatial neighborhood relationships and local details. By converting one-dimensional sequences back to two-dimensional image format and then extracting local information through convolution operations, this module improves the extraction effect of local features and effectively integrates them with global features.

[0094] The results significantly improved the model's segmentation accuracy in edge regions and local details, especially enhancing its segmentation performance for objects with complex boundaries or occlusions.

[0095] 4. By employing a cross-attention mechanism, features from RGB and T (thermal imaging) images are fused, fully utilizing the complementary information of the two modalities. RGB images capture details under visible light, while T images provide thermal radiation features. The cross-attention mechanism dynamically adjusts the weights of the two modalities, achieving accurate segmentation under low light and complex weather conditions.

[0096] The results significantly improved target segmentation accuracy in complex environments (such as low light, haze, occlusion, etc.) and enhanced the robustness of the model under different lighting and weather conditions.

[0097] 5. By combining the Position Attention Module (PAM) and the Channel Attention Module (CAM), features are enhanced from both spatial and channel dimensions. PAM captures the spatial dependencies between different locations in the image, while CAM highlights important channel features by strengthening the correlation between channels. The two complement each other, giving the network a stronger feature representation capability at different scales.

[0098] The results include: enhanced fusion of global and local features, improved segmentation performance for complex scenes and multi-class targets, further optimized representation of positional and channel features in images, and improved overall segmentation accuracy.

[0099] 6. By combining the efficient feature extraction structures of convolution and transformers, the model not only possesses strong segmentation performance but also optimizes computational efficiency. Through the use of dual attention mechanisms (PAM and CAM) and parallel convolutional modules (CNT), the model maintains high segmentation accuracy while reducing computational burden.

[0100] The results are as follows: While ensuring segmentation accuracy, the computational efficiency and inference speed of the model are improved, meeting the real-time requirements of UAV image segmentation in practical applications. Attached Figure Description

[0101] Figure 1 This is a schematic diagram of the overall system flow of the present invention.

[0102] Figure 2 This is a schematic diagram of the base network for semantic segmentation of drone images.

[0103] Figure 3 This is a schematic diagram of a parallel convolution module.

[0104] Figure 4 This is a schematic diagram of a dual attention module. Detailed Implementation

[0105] The present invention will now be described in further detail with reference to the accompanying drawings.

[0106] like Figure 1 As shown, a multimodal UAV image segmentation method comprises the following steps:

[0107] Step 1: Construct a novel UAV image semantic segmenter that combines convolution and transformers;

[0108] First, a hybrid encoder is introduced on the basis of the UNet model, which combines convolution and transformer to overcome the limitations of traditional convolutional neural networks in modeling long-distance dependencies and processing large-size images.

[0109] The first part of the hybrid encoder consists of convolutional blocks, which excel at extracting shallow features and local information, thereby reducing the computational burden on the transformer. The second part consists of Transformer blocks and a parallel convolutional module. The Transformer blocks are responsible for constructing long-range dependencies based on local information to achieve global modeling. Within each Transformer block, a parallel convolutional module is integrated to compensate for any shortcomings of the Transformer in extracting local information.

[0110] A novel RGB-T multimodal fusion attention mechanism is designed to fuse T-image information with RGB images through cross-attention, enhancing segmentation performance in complex backgrounds, low-light conditions, and occlusion environments. By integrating convolutional and transformer components into the RGB-T multimodal model, a comprehensive and powerful feature extractor is implemented, addressing the poor segmentation performance of Transformer networks on UAV image datasets. Furthermore, a PCAM module is designed to extract and enhance information features from channel and spatial views, effectively fusing features at different scales and strengthening attention to small-scale targets through contextual information. This approach effectively addresses the challenges of diverse image target scales and high segmentation difficulty, thereby improving the network's segmentation performance.

[0111] Step 1 is the foundation of the entire segmentation method, providing a powerful feature extraction framework by combining convolution and transformer. A UAV image semantic segmenter integrating convolution and transformer is constructed to extract global and local features from UAV images, handle multi-scale targets, improve segmentation accuracy in complex environments, and achieve information fusion in multimodal images, namely RGB and T images.

[0112] in:

[0113] 1. Convolutional Networks:

[0114] The convolutional network module is primarily responsible for extracting local features. It excels at capturing shallow features and spatial details in images, especially in drone images, where convolution can effectively extract local information from small-scale targets such as roads, buildings, and vehicles. The convolutional module is located at the front end of the entire network and in the initial stage of feature extraction. It also works in parallel with the subsequent transformer module to share the burden of processing shallow local information.

[0115] 2. Transformer Module:

[0116] The Transformer block, integrating Multi-Head Self-Attention (MHSA) and Parallel Convolutional Networks (CNT), is a core component of the segmentation network. MHSA extracts global features and, through collaboration with the parallel convolutional network, captures both global and local information simultaneously. The Transformer block fuses local convolutional features with global attention features to form a comprehensive understanding of the target. Each Transformer block refines features progressively through hierarchical progression, improving segmentation accuracy.

[0117] 3. Cross-attention module:

[0118] The cross-attention module is used for multimodal fusion of RGB and T images. In this module, query features from the RGB image are associated with key and value features from the T image, and vice versa. This cross-attention mechanism ensures that complementary information from both modalities is fully utilized, enabling accurate target segmentation even in low-light conditions and complex backgrounds. It plays a crucial role in cross-modal information exchange and integration.

[0119] 4. PCAM module (Dual Attention Module):

[0120] The PCAM module (Position Attention Module and Channel Attention Module) further improves segmentation accuracy by enhancing features from both spatial location and channel dimensions. The Position Attention Module (PAM) captures the spatial dependencies between any two locations in an image, thereby extracting spatial features with global context. The Channel Attention Module (CAM) focuses on extracting features between different channels, enhancing the semantic information representation of the image by assigning weights to each channel. The PCAM module integrates global contextual information and refined channel features, making the network's object segmentation more accurate.

[0121] These modules work together to form a highly coordinated UAV image semantic segmentation network, capable of handling the challenges of multimodal, complex backgrounds, and multi-scale targets, ultimately achieving high-precision and robust UAV image segmentation results.

[0122] The segmenter in step 1 captures local information from the UAV image using a convolutional neural network (CNN), while simultaneously utilizing a transformer to construct long-range dependencies to enhance global modeling capabilities, thereby improving the base network for semantic segmentation of UAV images (such as...). Figure 2 (as shown) accuracy and robustness.

[0123] Step 2: Design a U-shaped convolutional and transformer hybrid feature extraction network (RGB-TransUNet). The four-stage feature extraction network in Step 2 is a specific implementation of the idea of ​​combining convolution and transformer proposed in Step 1.

[0124] The structure in step 2 directly addresses the segmentation requirements of UAV images, especially the challenges of multi-scale and complex scenes in UAV images.

[0125] The network is divided into four phases.

[0126] To fully leverage the advantages of convolution in capturing shallow features and the powerful capabilities of transformers in global modeling of deep features, efficient ResNetV2 convolutional blocks are introduced in the first two stages.

[0127] F1=”ResNetV”“2”1(X) (1)

[0128] F2=”ResNetV”“2”2(F1) (2)

[0129] Next, we move to the Patch Embedding stage, where convolution, flattening, positional encoding, and dropout operations are used to generate a one-dimensional vector containing positional information.

[0130] F3="Flatten"("Conv"(F2)) (3)

[0131] F3=”PositionEmbedding”(F3) (4)

[0132] F3 = "Dropout" (F3) (5)

[0133] In the final stage, a transformer block with stronger long-range modeling capabilities is employed. Unlike ViT's multi-head self-attention mechanism, this transformer block includes a parallel convolutional module (CNT) to compensate for the transformer's shortcomings in extracting local information. This process is repeated 12 times, outputting a one-dimensional vector with the same dimensions as the original.

[0134] T0 = ​​F3 (6)

[0135] T i ="TransformerBlock"(T i-1 )”for”i=1, 2,…,12 (7)

[0136] TransformerBlock(T)="LN"(T+"CNT"("MHSA"("LN"(T)))) (8)

[0137] LN stands for Layer Normalization, a normalization technique in deep learning, typically used in the Transformer module of neural networks to accelerate model training and improve stability. Unlike batch normalization, layer normalization normalizes all neurons within a single sample.

[0138] MHSA stands for Multi-Head Self-Attention, a core component of the Transformer model used to capture dependencies between different positions in an input sequence. Through parallel multi-head self-attention, MHSA can process information from different representation spaces simultaneously, thereby improving the model's representational power.

[0139] After these four stages, the resulting labeled embeddings are: "Embeddings" = [F1, F2, F3, T]. 12 ];

[0140] Embeddings refer to the embedded features after network feature extraction, including the combination of shallow convolutional features and deep transformer features. By fusing features from different stages, accurate modeling of targets at different scales in the image is ensured.

[0141]

[0142] In step 2, the RGB-TransUNet network design combines the powerful feature extraction capabilities of the UNet network with the global modeling capabilities of the transformer. The U-shaped structure of UNet enables it to capture high-resolution local features in shallow layers and fuse these features with deeper global features through skip connections. To optimize the segmentation of UAV images, especially for processing multi-scale targets in top-down views, RGB-TransUNet introduces a transformer module into the U-shaped architecture, further enhancing its global modeling capabilities.

[0143] Step 3: To supplement the missing local information in the transformer block, a parallel convolution module (such as...) was constructed. Figure 3 (As shown).

[0144] The input characteristic of the i-th transformer block is denoted as:

[0145] For parallel convolutional modules and multi-head self-attention mechanisms, f i They use the same feature input, but process that feature differently. The parallel convolutional module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f. i Then, the features are processed differently in different branches.

[0146] f i It has two branches, one of which is a multi-head self-attention mechanism used in f i Modeling long-distance global information in the middle;

[0147] f i The process involves two branches, one of which is a multi-head self-attention mechanism used to address f. i Modeling long-distance global information.

[0148] Specifically, the input features are first mapped onto the query (Q), key (K), and value (V) using a projection matrix, i.e., Q, K, V = f i W Q f i W K f i W V ,in, Let Q, K, and V be the projection matrices, respectively. The formula for calculating self-attention is:

[0149]

[0150] The attention module iterates h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of attention heads. Features from the h attention heads are concatenated along the channel dimension to form the output of the MHSA. After MHSA processing, f is obtained. i Global information, represented as: f i g =MHSA(f i ).

[0151] Another branch is the parallel convolution module, used to capture f i Local features in the image. Since self-attention mechanism is applied to one-dimensional sequences, while convolution operation is applied to two-dimensional images, the sequence first needs to be converted to the corresponding image format. Next, spatial local information is extracted through convolution operation, and then the convolution result is converted back to a one-dimensional sequence to obtain f. i Local information is represented as:

[0152] f i l =CNT i (f i =img2seq(Conv i (seq2img(f i (11)

[0153] CNT i (·) represents a parallel convolution module, which consists of an "img"2seq(·) operation, three consecutive convolutional layers, and a seq2img(·) operation, which combines the label features with local context information.

[0154] To address the shortcomings of transformer blocks in local feature extraction, a parallel convolution module (CNT) was introduced. The purpose of this module is to perform convolution operations in parallel within each transformer block to effectively capture local information and supplement the transformer's long-range dependency modeling capabilities.

[0155] Step 4: To enhance segmentation performance in complex backgrounds, low-light conditions, and occlusion environments, the RGB and T images in the RGB-T multimodal fusion task each provide complementary information. The cross-attention mechanism facilitates efficient information fusion between these two modalities, thereby improving task accuracy. The calculation process for cross-attention is as follows: The query features of the RGB image are associated with the key and value features of the T image through the cross-attention mechanism to find the relevant parts between the RGB and T images. Cross-attention calculation formula:

[0156]

[0157] Here, Q represents the query from the RGB image, K and V represent the key and value from the T image, respectively, and d is a scaling factor used to avoid excessively large values ​​caused by the inner product of high-dimensional feature vectors. Through the above formula, cross-attention can dynamically adjust the information fusion between the RGB and T images, improving the segmentation accuracy of targets in complex environments.

[0158] f RGB-T =CA(f i (13)

[0159] Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA:

[0160] f i =f i g +f i l +f RGB-T (14)

[0161] After the fused features are processed by the FNN, they are input into the next transformer block:

[0162] f i+1 =FFN(f i )+f i (15)

[0163] The advantage of RGB-T multimodal fusion lies in capturing complementary information between modalities: cross-attention allows RGB images to extract features from T images that are difficult to capture in low-light or complex environments. By combining the advantages of these two modalities, the model can make more accurate predictions. Dynamic focus on relevant regions: The cross-attention mechanism can dynamically adjust according to task needs, giving priority to relevant regions in the image, thereby helping the model make better decisions in multimodal tasks. Enhanced robustness: In some situations (e.g., insufficient light in RGB images or weak signals in T images), cross-attention can achieve complementary information between different modalities, making the model more robust in multimodal environments.

[0164] Step 5: Optimize the feature fusion module of the UNet network. This addresses the shortcomings of traditional UNet and Transformer in utilizing intrinsic image location and channel features, as well as challenges in parameter efficiency and computational complexity. An efficient feature fusion module, called the Dual Attention Module (PCAM-Block), is used (e.g., ...). Figure 4 As shown in the diagram, this module serves as a feature extractor. It consists of two core parts: a Position Attention Module (PAM) and a Channel Attention Module (CAM), used to integrate and extract positional information and channel features from the image.

[0165] The Position Attention (PAM) module captures the spatial dependencies between any two points in the feature map. It updates specific features by performing a weighted summation of all positional features, where the weights are calculated based on the feature similarity between the two positions. Therefore, PAM effectively extracts spatially significant features.

[0166] Input local features A∈R C×H×W Where C is the number of channels, H is the height, and W is the width, three new feature maps are obtained through convolution: B, C, D ∈ R. C×H×W Reshape B and C into R C×N Where N = H × W represents the number of pixels, matrix multiplication is performed. The elements in matrix S are normalized to attention weights using the softmax function:

[0167]

[0168] Reshape matrix D into R C×N Perform matrix multiplication Reshape the results into R C×H×W Multiply the result by the parameter, and then sum the elements with feature A to obtain the final output:

[0169]

[0170] The weight α is initially set to 0 and learned gradually through training. PAM performs excellently in spatial feature extraction. Since E is a weighted sum of all location features and the original features, it possesses global contextual features and achieves context aggregation through a spatial attention map. This ensures that global contextual information is preserved while effectively extracting location features.

[0171] CAM (Channel Attention) excels at extracting channel features. Unlike PAM, it extracts the original features A∈R. C×H×W Remodeling into R C×N Then perform matrix multiplication between A and its transpose. Next, a softmax layer is applied to obtain the channel attention map X∈R. C×C :

[0172]

[0173] Perform matrix multiplication between the transpose of X and A. Reshape the result Y into R C×H×W Multiply the result Y by the scale parameter β, and then perform an element-wise summation operation with the original feature A to obtain the final output E∈R. C×H×W :

[0174] E=β·Y+A (19)

[0175] Similar to α, β is also learned through the training process. In the Channel Attention Module (CAM), the process of extracting channel features is similar to PAM; the final feature of each channel is a weighted sum of all channel features and the original features. This mechanism gives CAM powerful channel feature extraction capabilities.

[0176] in:

[0177] 1. PAM (Position Attention Module):

[0178] Main function: PAM is responsible for capturing the dependencies between any two spatial locations in an image. It updates the features of each location by calculating the similarity between locations in the image's feature map. This is very effective for capturing complex spatial relationships and target distributions in images, especially in drone images, where the distribution of objects may be random from a top-down perspective.

[0179] Focus: PAM mainly processes image features from a spatial dimension, focusing on the relationship between each spatial location (pixel or region) to help the network understand the structural information in the image.

[0180] Application scenarios: PAM can effectively handle complex spatial distributions in images, especially capturing dependencies between distant pixels and enhancing global contextual understanding. For example, in images taken by drones, different targets (such as buildings) may be far apart, but their correlation can be enhanced through PAM.

[0181] 2. CAM (Channel Attention Module):

[0182] Main function: CAM is responsible for capturing the relationships between different channels in the feature map along the channel dimension. Channel attention assigns higher weights to important channels by calculating the dependencies between different channels, thereby enhancing important features in the image. Each channel typically represents a different type of feature (such as edges, texture, color, etc.), and CAM improves the network's feature representation ability by learning the correlations between different channels.

[0183] Focus: CAM mainly processes image features from the channel dimension, focusing on the information contained in each channel, which makes the understanding of target features more comprehensive and precise.

[0184] Application Scenarios: CAM can enhance the feature representation capabilities of different channels, especially those channels that play a crucial role in semantic segmentation tasks. For UAV image segmentation, CAM can highlight features of different categories such as buildings, roads, and vegetation, making segmentation more accurate.

[0185] The combination of PAM and CAM throughout the network enables it to simultaneously capture local details and global information in images, and enhances the understanding of features from different channels. This synergy improves the network's segmentation performance, especially in multimodal tasks (RGB-T image fusion), where features from different modalities are more comprehensively represented through PAM and CAM. The combination of the two allows the network to effectively process image features in both spatial and channel dimensions, improving the segmentation accuracy and robustness for complex targets.

[0186] To objectively evaluate the performance of this invention on a specified dataset, precision-recall (PRC) and average precision (AP) are used as primary and authoritative evaluation metrics to quantitatively analyze various UAV image segmentation methods. The calculation formulas are as follows:

[0187]

[0188] Where TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.

[0189]

[0190] Where GT represents the truth box and PR represents the predicted box.

[0191]

[0192] Where P and R represent Precision and Recall respectively, and N cls This represents the total number of categories in the dataset. The network of this invention is trained directly using the aforementioned dataset and evaluated using the metrics mentioned above.

[0193] For the appendix Figure 1 To further explain, the network architecture of the entire multimodal UAV image segmentation method is divided into multiple modules, and the specific functions of each part are as follows:

[0194] 1. RGB-Transformer encoder module:

[0195] The left-hand module illustrates the workflow of the RGB-Transformer encoder. This module contains 12 Transformer layers that model the global contextual information of the input sequence through multi-head self-attention (MHSA) and cross-attention (CA) mechanisms.

[0196] Input sequence: Features from RGB images are first transformed into a one-dimensional sequence by image-to-sequence (Img2Seq) transformation, and then enter the RGB-Transformer.

[0197] MSA (Multi-head Self-Attention): Responsible for capturing global dependencies between different positions in a sequence, which helps to extract spatial features over long distances.

[0198] Cross-Attention (CA): Focuses on feature relationships along the channel dimension, enhancing the network's feature representation capabilities by assigning more weights to important channels.

[0199] 2. CNN Feature Extraction Module:

[0200] In the middle part of the diagram, the CNN module is used to extract the fundamental features of the RGB and T images. This module captures local features in the image, such as edges, textures, and local structures, through multiple layers of convolutional operations.

[0201] PCAM-Block (Dual Attention Mechanism Module): Located after the CNN feature maps, it is used to establish dependencies between spatial locations and channels. It combines the functions of PAM (Place Attention Module) and CAM (Channel Attention Module), further optimizing feature representation capabilities and enhancing the network's understanding of complex image structures.

[0202] 3. Multi-scale feature fusion and downsampling:

[0203] The figure illustrates the further processing of features after 12 Transformer layers. Through upsampling and downsampling operations, feature maps at different resolutions are progressively processed and fused to ensure that target features are captured at multiple scales. This design aims to handle targets of different sizes and shapes, and this multi-scale processing is particularly important for small target detection, especially in UAV imagery.

[0204] Conv3x3, ReLU: Convolutional layers and the ReLU activation function are used to perform further nonlinear transformations and spatial detail optimizations on the features.

[0205] 4. PAM & CAM module:

[0206] PAM (Position Attention Module): Captures the dependencies between different spatial locations in the feature map, ensuring that the model can find related regions in the global space, thereby better understanding the target location in the image.

[0207] CAM (Channel Attention Module): It highlights important features by weighting different channels in the channel dimension, ensuring that the most discriminative features are used for segmentation tasks.

[0208] PCAM-Block combines the advantages of PAM and CAM, improving feature extraction capabilities in both spatial and channel dimensions.

[0209] 5. Upsampling and output segmentation results:

[0210] The right side shows how the original resolution is gradually restored through upsampled feature maps, and finally, the segmentation head generates the final segmentation result. Multiple upsampling steps ensure that image details are recovered at a relatively high resolution.

[0211] Ultimately, the segmentation map output by the network can accurately segment the target objects in the image.

[0212] Appendix Figure 1 The demonstrated network architecture combines the global modeling capabilities of Convolutional Neural Networks (CNNs) and Transformers, employing multiple attention mechanisms (MSA, CA, PAM, CAM) to extract and fuse image features at multiple levels. RGB and T images are fused through a cross-attention mechanism, ultimately generating high-precision semantic segmentation results, which are particularly suitable for processing complex scenes and multi-scale targets in UAV images.

[0213] In this invention, multimodal processing refers to the use of multiple different types of sensors or data sources to acquire and process image information. In this invention, the multimodal processing employed includes RGB images and T (thermal imaging) images. RGB images capture color information within the visible spectrum and are typically used to record the appearance and details of objects; while T images can capture the infrared thermal radiation information of objects, making them particularly suitable for object detection and recognition in low-light or complex environments.

[0214] By fusing information from these two modalities, RGB-T multimodal image segmentation can supplement the temperature feature information provided by thermal imaging while preserving the target's appearance information, thereby enhancing the target segmentation effect under harsh conditions and improving the segmentation accuracy and robustness of UAV images.

[0215] The present invention will now be described in more detail with reference to experimental examples. The specific steps are as follows:

[0216] 1. Dataset Selection

[0217] This invention uses the following two datasets in the task of semantic segmentation of UAV images:

[0218] The Semantic Drone dataset focuses on semantic understanding of urban scenes to improve the safety of autonomous drone flight and landing procedures. The images depict over 20 houses from a bird's-eye view, taken at heights ranging from 5 to 30 meters above the ground. High-resolution cameras were used to acquire images of 6000x4000px (24Mpx) in size. This semantic segmentation dataset includes 20 categories. The training set contains 400 publicly available images, and the test set contains 200 private images.

[0219] The homemade drone image RGB-T dataset comprises images captured by drones at altitudes ranging from 5 to 100 meters. This dataset provides 10,000 8000x6000px RGB images of buildings and their corresponding T images, including ground truth masks for two categories.

[0220] 2. Selection of evaluation indicators

[0221] To objectively evaluate the performance of this invention on a given dataset, precision-recall (PRC) and average precision (AP) are used as standard and authoritative evaluation metrics to quantitatively evaluate different UAV image semantic segmentation methods. Precision, Recall, IoU, AP, and mAP are calculated as follows:

[0222]

[0223] 3. Construction of a U-shaped architecture-based feature extraction network combining convolutional and transformer architectures, and pre-training instructions.

[0224] The Aeroscapes dataset is a novel, large-scale benchmark dataset containing 3269 high-resolution images taken by drones, primarily covering urban and rural scenes. This invention utilizes all images from the Aeroscapes dataset for self-supervised pre-training. During pre-training, the batch size is set to 128, the AdamW optimizer is used, and the training is evenly distributed across two 4090 GPUs. Unless otherwise specified, the model is trained for 50 epochs using MAE's default settings.

[0225] 4. Based on the pre-trained model, the model is fine-tuned in the downstream segmentation task.

[0226] 5. In the data preprocessing of the downstream segmentation task, this invention performs unified processing on the input image. First, the image is resized to 256x256 through image scaling and edge padding, and data augmentation is performed by horizontal flipping with a probability of 0.5. Then, the image is normalized, mapping pixel values ​​to between 0 and 1. Finally, the pixel distribution of the image is converted from RGB to RRRGGGBBB through color space conversion.

[0227] Following the de facto standard, all models were trained using the AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.1. A momentum parameter of 0.9 was used to assign weights to the last updated gradient. To fully train the models, a 6x training plan was used, comprising 72 epochs with a batch size of 1. The learning rate was adjusted using a multi-step scheduler, decreasing to 1 / 10 of its original value at epochs 48 and 66. A tiered learning rate decay strategy was also employed, with a decay rate set to 0.7.

[0228] 6. Comparison of Results

[0229] To illustrate the superiority of this invention, it is compared with five other advanced methods, including: "Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation [C] / / Medical image computing and computer-assisted intervention–MICCAI 2015:18th international conference, Munich, Germany, October 5-9, 2015, procedures, part III 18. Springer International Publishing, 2015:234-241.", "He K, Gkioxari G, Dollár P, et al. Mask r-cnn [C] / / Proceedings of the IEEE international conference on computer vision. 2017:2961-2969.", "Liu M, Fu B, Xie S, et al. Comparison of multi-source satellite images for classifying marshvegetation using DeepLabV3 Plus deep learning". algorithm[J].EcologicalIndicators, 2021, 125:107562.", "Chen J, Lu Y, Yu Q, et al.Transunet: Transformersmake strong encoders for medical image segmentation[J].arXiv preprint arXiv:2102.04306, 2021.", "Liu Z, Lin Y, Cao Y, et al.Swin transformer:Hierarchicalvision transformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.".

[0230] To ensure fair comparison, all experimental results were generated under the same settings and tested on the split test sets of both datasets using mAP@50. The RGB-TransUNet network achieved an average accuracy of 82.68% on the Semantic Drone dataset, which is 1.16% higher than Deeplabv3+; and an average accuracy of 84.57% on the self-made drone image dataset, which is also 1.25% higher than Transunet, demonstrating the superiority of the RGB-TransUNet proposed in this invention.

[0231] The multimodal segmentation network proposed in this invention combines the local feature extraction capability of convolutional neural networks (CNNs) with the global modeling capability of transformers, which can better capture the spatial relationship of targets in images, thereby improving the accuracy of segmentation.

[0232] The network proposed in this invention can capture detailed information of different targets under different lighting conditions through RGB-T multimodal fusion, and solve the segmentation problem of multi-scale targets by utilizing the long-distance dependency modeling capability of transformer blocks.

[0233] This invention introduces thermal imaging (T-images), enabling the system to accurately segment targets even in low light or adverse weather conditions. By fusing RGB and T-image information through a cross-attention mechanism, the segmentation performance in complex environments is effectively improved.

[0234] This invention optimizes feature extraction efficiency by introducing a dual attention module (PCAM-Block) and balances accuracy and speed in the combination of convolution and transformer to meet the real-time requirements of UAV image segmentation.

[0235] Through these designs, the multimodal UAV image segmentation method of the present invention can adapt to the special challenges in UAV images, improve segmentation accuracy and robustness, especially under complex and variable environmental conditions.

Claims

1. A multimodal unmanned aerial vehicle (UAV) image segmentation method, characterized in that, Includes the following steps; Step 1: Introduce a hybrid encoder on the basis of the UNet model, and fuse convolution and transformer to form a feature extraction framework to build a base network for semantic segmentation of UAV images; Step 2: Capture high-resolution local features in shallow layers using the Convolutional Neural Network (CNN) module, extract global features from deep image features using the multi-head self-attention mechanism in the transformer module, and fuse the local features with the global features through skip connections. Step 3: By using a parallel convolution module and a multi-head self-attention mechanism, the features are optimized from local and global perspectives, respectively. The parallel convolution module extracts detailed local information, while the multi-head self-attention mechanism captures long-distance global dependencies. Step 4: Based on the local and global features extracted in Step 2 and Step 3, the feature information is fused into RGB-T multimodal modes through the cross-attention mechanism module to improve the segmentation effect; Step 5: Optimize the feature representation of the basic feature extraction framework built by combining convolution and transformer in Step 1 and Step 2, and construct a dual attention module to integrate position and channel feature information, thereby optimizing the local and global information processing of the image; The UAV image semantic segmentation base network in step 1 is used to extract global and local features in UAV images, process multi-scale targets, improve segmentation accuracy in complex environments, and achieve information fusion in multimodal images, namely RGB images and T images. The UAV image semantic segmentation base network includes a convolutional module, a Transformer block, a cross-attention module, and a PCAM module. The convolutional module is at the front end of the network and is used for preliminary local feature extraction of the input image. It is fused with the output of the deep transformer in the skip connection. The Transformer block fuses the convolution and transformer and achieves accurate modeling of targets at different scales through multi-level feature transfer and fusion. The cross-attention module integrates complementary information from RGB and T images to ensure the collaborative processing of multimodal images. The PCAM module enhances spatial and channel features to ensure that targets of different scales and categories can be accurately segmented. Input to the drone image semantic segmentation network: RGB image: A regular visible light image from a drone, containing color and detail information of the scene, used to capture visual features in the image; T-image: Infrared images acquired by drones using thermal imaging equipment, used for target detection in low-light and complex background environments; These two types of images are used as multimodal inputs to the network for feature extraction and processing. Output of the UAV image semantic segmentation network: Segmentation Mask: An image with the same dimensions as the input image, where each pixel corresponds to a category label in the image; In step 2, local and global features in RGB and T images are extracted by combining convolution and transformer. These features are fused through a U-shaped structure and skip connections to ensure the preservation of image features at different scales. Local feature extraction is used to accurately capture small targets and details in images, while global feature extraction ensures the segmentation accuracy of large-scale targets and backgrounds. The feature extraction network is divided into four stages. The first two stages use convolution to extract local features, and the last two stages use transformers to perform global modeling. The two are fused through skip connections. In the first two stages, ResNetV2 convolutional blocks were used; F1 = ResNetV21(X) (1) F2 = ResNetV22(F1) (2) Next, in the third stage, namely the Patch Embedding stage, convolutional operations are used to flatten and position the input, and dropout is applied to finally generate a one-dimensional vector containing positional information. F3 = Flatten(Conv(F2)) (3) F3=PositionEmbedding(F3) (4) F3 = Dropout(F3) (5) In the final stage, a transformer block containing a parallel convolution module is used to increase the transformer's ability to perceive local information. After repeating 12 times, a one-dimensional vector of the same size as the original is output. T0 = ​​F3 (6) T i =TransformerBlock(T i-1 )for i=1,2,...,12 (7) TransformerBlock(T)=LN(T+CNT(MHSA(LN(T)))) (8) LN refers to Layer Normalization. MHSA is used to capture the dependencies between different positions in the input sequence. MHSA processes information from different representation spaces simultaneously through a parallel multi-head self-attention mechanism. In step 4, the fusion of local features: local features extracted by convolutional networks and parallel convolutional modules, namely the features at the detail level in RGB and T images, can be combined through a cross-attention mechanism, so that the detail information of the RGB image and the supplementary information captured in the T image in low light or complex background can work together. Global feature fusion: Global features extracted by transformer blocks are complemented between modalities through a cross-attention mechanism. RGB images provide global information under visible light conditions, while T images enhance global background understanding in low-light environments. The cross-attention mechanism identifies relevant parts in the RGB and T images by associating the query of the RGB image with the key and value of the T image. The query of the T image is also associated with the key and value of the RGB image, thus achieving bidirectional information fusion. The calculation process for cross-attention is as follows: Query: A representation from an RGB image; Key and Value: Feature representations derived from another modality T image. Through cross attention, the query features of the RGB image are multiplied by the key and value features of the T image to find the parts of the T image that are related to the RGB image, and vice versa. Q RGB : Represents the query vector extracted from the RGB image, K T : Represents the key vector extracted from image T, where d is the dimension of the key vector, and V T : Represents the value vector in the T-image; Cross attention can dynamically adjust the information fusion between RGB and T images, improving the segmentation accuracy of targets in complex environments; f RGB-T =CA(f i ) (13) Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA: After the fused features are processed by the FNN, they are input into the next transformer block: f i+1 =FFN(f i ) + f i (15) In step 5, the dual attention module (PCAM-Block) serves as a feature extractor, comprising a positional attention module (PAM) and a channel attention module (CAM), which are responsible for extracting positional and channel features, respectively. The Position Attention (PAM) module can capture the spatial dependency between any two points in the feature map and update specific features by weighted summation of all position features, where the weights are calculated based on the feature similarity between two positions. Input local features A∈R C×H×W Where C is the number of channels, H is the height, and W is the width, three new feature maps are obtained through convolution: B, C, D ∈ R. C×H×W Reshape B and C into R C×N Where N = H × W represents the number of pixels, matrix multiplication is performed. The elements in matrix S are normalized to attention weights using the softmax function: Reshape matrix D into R C×N Perform matrix multiplication Reshape the results into R C×H×W Multiply the result by the parameter, and then sum the elements with feature A to obtain the final output: The weight α is initially set to 0 and is learned gradually through training. In the aforementioned CAM (channel attention), the original feature A∈R is directly applied... C×H×W Remodeling into R C×N Then perform matrix multiplication between A and its transpose. Next, a softmax layer is applied to obtain the channel attention map X∈R. C×C : Perform matrix multiplication between the transpose of X and A. Reshape the result Y into R C×H×W Multiply the result Y by the scale parameter β, and then perform an element-wise summation operation with the original feature A to obtain the final output E∈R. C×H×W : E=β·Y+A (19) β is learned through the training process.

2. The multimodal UAV image segmentation method according to claim 1, characterized in that, The features obtained in the four stages are as follows: Embeddings = [F1, F2, F3, T] 12 ]; Embeddings refer to the embedded features after network feature extraction, including the combination of shallow convolutional features and deep transformer features. By fusing features from different stages, accurate modeling of targets at different scales in the image is ensured.

3. The multimodal UAV image segmentation method according to claim 2, characterized in that, F1 represents the first layer of features obtained through convolution, with a spatial resolution one-quarter of the input image resolution and a feature dimension of C1. Here, H and W are the height and width of the input image, respectively, and C1 is the number of feature channels in this layer; F2 represents the second layer of features obtained through convolution, with a spatial resolution of one-sixteenth of the input image resolution and a feature dimension of C2. H and W are the height and width of the input image, respectively, and C2 is the number of feature channels in this layer. F3 represents the third layer of features obtained through deep convolution and transformer operations. Its spatial resolution is also one-sixteenth of the input image. After flattening, the embedding dimension is C3. HW represents the total number of pixels in the input image, and C3 is the number of channels for the deep features; T 12 This indicates the features processed by the transformer module, whose embedding dimension is the same as that of F3: This layer of features is used to enhance global feature modeling, especially for modeling long-distance dependencies of multi-scale targets.

4. The multimodal UAV image segmentation method according to claim 3, characterized in that, In step 3, let f be the i-th transformer block. i The input features are: For parallel convolutional modules and multi-head self-attention mechanisms, f i Since they share the same feature input, the parallel convolutional module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f. i Then, the features are processed differently in different branches; f i It has two branches, one of which is a multi-head self-attention mechanism used in f i Modeling long-distance global information in the middle; Another branch is the parallel convolution module, used to extract f. i The multi-head self-attention mechanism is applied to one-dimensional sequences, while the parallel convolution module is applied to two-dimensional images.

5. The multimodal UAV image segmentation method according to claim 4, characterized in that, In the multi-head self-attention mechanism, the input label f ​​is first projected using a projection matrix. i Projecting onto the query (Q), key (K), and value (V), i.e., Q, K, V = f i W Q f i W K f i W V ,in Let Q, K, and V represent the projection matrices, respectively. The self-attention operation is calculated as follows: K T : Represents the transpose operation of the key vector, used to perform a dot product with the query vector Q, calculating the relevance between each query vector and all key vectors, with dimension R. D×N ; This represents the Scaled Dot-Product Attention mechanism, and the formula applies to QK. T To scale, use Used as a scaling factor to balance the dot product results It is the scaling factor of the dot product result, where D is the dimension of the query and key vectors; Using the softmax function, the relevance values ​​between the query vector and each key vector are converted into weighted coefficients, which are used to weight the vector V. The weighted sum of all key vectors is 1. The module iterates h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of self-attention heads. Features from the h attention heads are concatenated along the channel dimension to form the output of the MHSA. After passing through the MHSA, f is obtained. i Global information is represented as:

6. The multimodal UAV image segmentation method according to claim 4, characterized in that, In the parallel convolution module, firstly, the input features of the transformer module are flattened into a one-dimensional sequence and then converted back into the corresponding two-dimensional image format (seq2img). Then, spatial local information is extracted through convolution operations, and the convolution result is converted back into a one-dimensional sequence (img2seq) to obtain f. i Local information Represented as: CNT i (·) indicates a parallel convolution module, which consists of an img2seq(·), three consecutive convolutional layers, and a seq2img(·) operation. This operation integrates the label features with local context information. By directly adding the global features generated by the multi-head self-attention module (MHSA) and the local features extracted by the parallel convolutional module (CNT), the fusion of global and local information is achieved, as shown in the formula. The fusion feature in the i-th transformer block This represents the global features extracted through the transformer's self-attention mechanism (MHSA). This represents the local features extracted by the parallel convolutional module (CNT).

Images