Multi-mode unmanned aerial vehicle image segmentation method

Abstract

The invention discloses a multi-mode unmanned aerial vehicle image segmentation method. The method comprises the steps of 1, constructing an unmanned aerial vehicle image semantic segmentation base network; 2, capturing high-resolution local features in a shallow layer through a convolutional neural network (CNN) module, extracting from deep features of the image through a multi-head self-attention mechanism in a transformer module, and fusing the local features with deep global features through jump connection; 3, constructing a parallel convolution module, and optimizing the features fused in the step 2; 4, combining feature extraction results in the step 2 and the step 3, fusing feature information through a cross attention mechanism, and improving a segmentation effect in a complex scene; and 5, constructing a double attention module, and optimizing local and global information processing of the image. The method has the characteristics that the global and local features are efficiently fused, the multi-scale target is accurately segmented, and the robustness in a complex scene is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic segmentation of unmanned aerial vehicle images, and in particular relates to a multimodal unmanned aerial vehicle image segmentation method. Background Art

[0002] The distribution of objects in drone images is often locally dense, and each object has arbitrary rotations from a bird's-eye view. In addition, there are significant scale differences between different instances, and images are susceptible to changes in lighting, shadows, night, and weather conditions (such as fog). These complex factors pose challenges to the semantic segmentation of drone images, resulting in obstacles for general segmentation algorithms in drone image applications. In order to more accurately characterize the feature information of drone images, some researchers have made specific improvements based on convolutional neural networks (CNNs) for the characteristics of drone images. Although some success has been achieved, the limitations of convolutional neural networks in feature extraction have hindered their further development. For example, their insufficient global modeling capabilities limit the segmentation of large-scale objects; at the same time, the weak correlation of image context information leads to low segmentation accuracy or even inability to complete segmentation.

[0003] At present, most drone image semantic segmentation networks based on pure convolutional structures have certain limitations due to the lack of global information; while the detection network based on transformer structure has a strong ability to model long-distance dependencies, but is relatively weak in local feature extraction. Therefore, these two architectures have some shortcomings when dealing with multi-scale scenarios of drone image semantic segmentation. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multimodal drone image segmentation method, which, through a novel multimodal semantic segmentation network combining convolution and transformer series-parallel connection, has the characteristics of efficient fusion of global and local features, accurate segmentation of multi-scale targets, significantly improved robustness in complex scenes (such as low light, occlusions and severe weather conditions), high model parameter efficiency and low computational complexity.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A multimodal UAV image segmentation method comprises the following steps;

[0007] Step 1: Based on the UNet model, a hybrid encoder is introduced to fuse convolution and transformer to form a feature extraction framework and build a base network for semantic segmentation of drone images;

[0008] Step 2: Step 2 is a further refinement and implementation based on the framework proposed in Step 1. It introduces in detail how to fuse convolution and transformer through the U-shaped structure to build a specific segmentation network RGB-TransUNet;

[0009] Through the convolutional neural network (CNN) module, high-resolution local features are captured at shallow levels in the first few layers of the network, and the local features are fused with global features extracted from the deep features of the image through the multi-head self-attention mechanism in the transformer module through skip connections;

[0010] Step 3: Through the parallel convolution module and the multi-head self-attention mechanism, the features are further optimized from the local and global perspectives respectively. The parallel convolution module extracts detailed local information, and the multi-head self-attention mechanism captures long-distance global dependencies; supplement the local information missing from the transformer in step 2, and optimize the features fused in step 2;

[0011] Step 4: Based on the local and global features extracted in steps 2 and 3 (including the features of RGB and T images), the feature information is fused into RGB-T multimodal form through a cross-attention mechanism (CA) module to improve the segmentation effect in complex scenes.

[0012] Step 5: Optimize the feature expression of the basic feature extraction framework built by combining convolution and transformer in steps 1 and 2, and build a dual attention module (PCAM-Block) to integrate position and channel feature information, thereby optimizing the local and global information processing of the image. PCAM-Block is closely related to the previous steps. It is a module that further optimizes feature fusion on the entire network structure.

[0013] The UAV image semantic segmentation network is used to extract global and local features in UAV images, process multi-scale targets, improve segmentation accuracy in complex environments, and realize information fusion in multimodal images, namely RGB images and T images.

[0014] The UAV image semantic segmentation network includes a convolution module, a Transformer block, a cross attention module and a PCAM module. The convolution module is at the front end of the network and is used for preliminary local feature extraction of the input image (including RGB and T images) and is fused with the deep transformer output in the jump connection. The Transformer block fuses convolution and transformer, and realizes accurate modeling of targets of different scales through multi-level feature transmission and fusion. The cross attention module integrates the complementary information of RGB images and T images to ensure the coordinated processing of multimodal images. The PCAM module ensures that targets of different scales and categories can be accurately segmented by enhancing spatial and channel features.

[0015] Input of the drone image semantic segmentation network:

[0016] RGB images: Ordinary visible light images from drones, containing color and detail information of the scene, usually used to capture visual features in the image (such as buildings, roads, vehicles, etc.);

[0017] T image (thermal image): Infrared image obtained by drone using thermal imaging equipment, usually used for semantic segmentation in low light, complex background and other environments. It provides thermal radiation information of objects and complements RGB images. These two images (RGB and T) are used as multi-modal inputs and enter the network for feature extraction and processing;

[0018] Output of the drone image semantic segmentation network:

[0019] Segmentation Mask: It is an image of the same size as the input image, where each pixel corresponds to a class label in the image.

[0020] For example:

[0021] Each pixel may be classified into different categories such as buildings, roads, vehicles, trees, etc. The segmentation network gives the category to which each pixel belongs based on the input RGB and T images. The output mask can be a two-dimensional array, and the value indicates the category to which the pixel belongs.

[0022] The UAV image semantic segmentation base network constructed based on the UNet model in step 1 extracts local and global features in the input image through a combination of convolution and transformer. These features are fused through a U-shaped structure and jump connections to ensure the retention of image features of different scales (especially small targets and large-scale backgrounds);

[0023] The extraction of local features is used to accurately capture small targets and details in the image, and the extraction of global features ensures the segmentation accuracy of large-scale targets and backgrounds;

[0024] In step 2, the feature extraction network is divided into four stages. The first two stages use convolution to extract local features, and the last two stages use transformers for global modeling, and the two are integrated through jump connections. In the first two stages, in order to make full use of the efficiency of convolution in capturing shallow local features and the advantage of transformers in global modeling in deep global features, ResNetV2 convolution blocks are used.

[0025] F1=”ResNetV”“2”1(X) (1)

[0026] F2=”ResNetV”“2”2(F1) (2)

[0027] Next, in the third stage, the Patch Embedding stage, the input is flattened (Flatten), positioned (PositionEmbedding), and dropout is applied using convolution operations, ultimately generating a one-dimensional vector containing position information;

[0028] F3="Flatten"("Conv"(F2)) (3)

[0029] F3=”PositionEmbedding”(F3) (4)

[0030] F3=”Dropout” (F3) (5)

[0031] In the last stage, a transformer block (TransformerBlock) containing a parallel convolution module (CNT) is used to increase the transformer's ability to perceive local information. After repeating 12 times, a one-dimensional vector with the same size as the original is output;

[0032] T0=F3 (6)

[0033] T i =”TransformerBlock”(T i-1 )”for”i=1, 2,…,12 (7)

[0034] TransformerBlock(T)="LN"(T+"CNT"("MHSA"("LN"(T)))) (8)

[0035] LN refers to Layer Normalization, and MHSA is the abbreviation of Multi-Head Self-Attention, which is used to capture the dependencies between different positions in the input sequence. MHSA can simultaneously process information from different representation spaces through a parallel multi-head self-attention mechanism, thereby improving the representation ability of the model.

[0036] The features obtained in the four stages are: "Embeddings" = [F1, F2, F3, T 12 ];

[0037] Embeddings represents the embedded features after network feature extraction, including the combination of shallow convolution features and deep transformer features. By fusing features from different stages, it ensures accurate modeling of objects of different scales in the image.

[0038]

[0039] F1 represents the first layer of features obtained by convolution operation, whose spatial resolution is one quarter of the input image resolution, and the feature dimension is C1: Here, H and W are the height and width of the input image, respectively, and C1 is the number of feature channels of this layer;

[0040] F2 represents the second layer of features obtained through convolution operation, whose spatial resolution is one sixteenth of the input image resolution, and the feature dimension is C2: Here, H and W are the height and width of the input image, respectively, and C2 is the number of feature channels of this layer;

[0041] F3 represents the third layer of features obtained through deep convolution and transformer operations. Its spatial resolution is also one sixteenth of the input image. After flattening, the embedding dimension obtained is C3: Here, HW represents the total number of pixels in the input image, and C3 is the number of channels of the deep features;

[0042] T 12 It represents the features processed by the transformer module (including the multi-head self-attention mechanism and the parallel convolution module), and its embedding dimension is the same as F3: This layer of features is used to strengthen global feature modeling, especially the long-range dependency modeling of multi-scale objects.

[0043] In step 3, the i-th transformer block f i The input features are:

[0044]

[0045] For parallel convolution modules and multi-head self-attention mechanisms, f i is the same feature input, the parallel convolution module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f i , and then process the features differently in different branches.

[0046] f i It has two branches, one of which is a multi-head self-attention mechanism used to i The other branch is a parallel convolution module for extracting f i In order to integrate the local information in the image, the multi-head self-attention mechanism acts on the one-dimensional sequence, while the parallel convolution module acts on the two-dimensional image.

[0047] In the multi-head self-attention mechanism, the input mark f is first transformed into iProjected onto query (Q), key (K) and value (V), i.e. Q, K, V = f i W Q , f i W K , f i W V ,in Denote the projection matrices of Q, K, and V respectively, and the self-attention operation is calculated as follows:

[0048]

[0049] K T : represents the transpose operation of the key vector, which is used to perform dot product operation with the query vector Q to calculate the correlation between each query vector and all key vectors, with a dimension of R D×N ;

[0050] represents the scaled dot-product attention mechanism, for QK T To zoom, use As a scaling factor to balance the dot product result, is the scaling factor of the dot product result, where D is the dimension of the query and key vectors;

[0051] Softmax is a normalization function that converts the dot product result of the input into a probability distribution. Through the softmax function, the correlation value between the query vector and each key vector is converted into a weighting coefficient for the weighted value vector V, which ensures that for each query vector, the weighted sum of all key vectors is 1;

[0052] The attention module is iterated h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of self-attention heads. The features from h attention heads are connected along the channel dimension to form the output of MHSA. After passing through MHSA, f is obtained. i The global information of i g =MHSA(f i );

[0053] In the parallel convolution module, first, the input features of the transformer module are flattened into a one-dimensional sequence and then reconverted into the corresponding two-dimensional image format (seq2img). Then, the spatial local information is extracted through the convolution operation, and the convolution result is converted back into a one-dimensional sequence (img2seq) to obtain f i The local information f i l , expressed as:

[0054] f il =CNT i (f i )=img2seq(Conv i (seq2img(f i ))) (12)

[0055] Among them, CNT i (·) denotes a parallel convolutional module, which consists of an imq2seq(·), three consecutive convolutional layers, and a seq2img(·) operation, which integrates the label features with the local context information;

[0056] By directly adding the global features generated by the multi-head self-attention module (MHSA) and the local features extracted by the parallel convolution module (CNT), the fusion of global information and local information is achieved. The formula is as follows:

[0057] f i gl =f i g +f i l (13)

[0058] f i gl : The fused features in the i-th transformer block, f i g represents the global features extracted by the transformer self-attention mechanism (MHSA), f i l Represents the local features extracted by the parallel convolution module (CNT).

[0059] In step 4, the cross attention mechanism is not only a direct information fusion of RGB and T images, but also an effective combination of the information of the two modalities based on the local and global features extracted in steps 2 and 3; specifically:

[0060] Fusion of local features: The local features extracted by the convolutional network and the parallel convolutional module, that is, the features at the detail level in the RGB and T images (such as edges, textures, etc.), can be combined through the cross-attention mechanism, so that the detail information of the RGB image and the complementary information captured in the T image in low light or complex background can work together;

[0061] Fusion of global features: Global features extracted by transformer blocks (such as large-scale background, long-distance dependency, etc.) are complemented by cross-attention mechanisms. RGB images provide global information under visible light conditions, while T images enhance global background understanding in low-light environments.

[0062] Cross-attention mechanism: The cross-attention mechanism identifies the relevant parts of the RGB image and the T image by associating the query of the RGB image with the key and value of the T image. The query of the T image is also associated with the key and value of the RGB image to achieve bidirectional information fusion;

[0063] The calculation process of cross attention is as follows:

[0064]

[0065] Query: representation from RGB image;

[0066] Key and Value: Feature representation from another modality T image. Through cross attention, the query features of the RGB image are dot-producted with the key and value features of the T image to find the parts of the T image that are related to the RGB image, and vice versa.

[0067] Where Q RGB : represents the query vector (Query) extracted from the RGB image, K T : represents the key vector (Key) extracted from the T image, d is the dimension of the key vector, V T : Represents the value vector in the T image.

[0068] f RGB-T =CA(f i ) (15)

[0069] Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA:

[0070] f i =f i gl +f RGB-T (16)

[0071] The fused features are processed by a feed-forward neural network (FFN) and then input into the next transformer block.

[0072] f i+1 =FFN(f i )+f i (17)

[0073] f i+1 Represents the input features of the i+1th layer transformer block, or the output features after the features of the current layer (ith layer) are processed. This is the result of the fusion of global features and local features and processing through a feedforward neural network (FFN), and also contains the information of the residual connection.

[0074] FFN(f i ) represents the features processed by the Feed Forward Network (FFN). FFN is an important component of the transformer module, usually consisting of two fully connected layers and a nonlinear activation function. FFN is used to process the fused features f i Further feature extraction and dimensionality transformation are performed to enhance the model's expressiveness. The role of FFN is to perform nonlinear transformations on local and global features to further enhance the model's modeling capabilities.

[0075] In step 1 and step 2, the network constructs a hybrid feature extraction framework based on convolution and transformer. The goal of these two steps is to extract local features through convolution and global features through transformer. These features are passed to subsequent steps for processing. The local features extracted by the parallel convolution module in step 3 and the features of the RGB-T multimodal fusion in step 4 are fused with the features extracted by PCAM-Block through jump connections, which enhances the segmentation accuracy under complex backgrounds and low light conditions. The position attention and channel attention mechanisms ensure the optimal expression of the fused features. In step 5, the role of the dual attention module (PCAM-Block) is to further optimize these features that have been extracted by convolution and transformer. The feature fusion module receives the local and global features extracted in step 2, and further improves the expressiveness of the features through the position attention module (PAM) and the channel attention module (CAM).

[0076] In step 5, a dual attention module (PCAM-Block) is constructed for feature extraction, wherein the dual attention module serves as a feature extractor and includes a position attention module (PAM) and a channel attention module (CAM), which are responsible for extracting position and channel features, respectively.

[0077] The position attention module (PAM) can capture the spatial dependency between any two points in the feature map and update a specific feature by taking a weighted sum of all position features, where the weight is calculated based on the feature similarity between the two positions;

[0078] Input local feature A∈R C×H×W , where C is the number of channels, H is the height, and W is the width. Three new feature maps are obtained through convolution operations: B, C, D∈R C×H×W , reshape B and C into R C×N , where N = H × W represents the number of pixels, performing matrix multiplication Use the softmax function to normalize the elements in the matrix S into attention weights:

[0079]

[0080] Reshape the matrix D into R C×N , performs matrix multiplication and reshape the result into R C×H×W , multiply the result by the parameter, and perform element-wise summation with feature A to get the final output:

[0081]

[0082] The weight α is initially set to 0 and is gradually learned through training.

[0083] In the CAM (channel attention), the original feature A∈R C×H×W Reshape to R C×N Then do a matrix multiplication between A and its transpose Next, apply the softmax layer to obtain the channel attention map X∈R C×C :

[0084]

[0085] Perform matrix multiplication between the transpose of X and A Reshape the result Y into R C×H×W , multiply the result Y by the scale parameter β, and perform element-by-element sum operation with the original feature A to obtain the final output E∈R C×H×W :

[0086] E=β·Y+A (21)

[0087] Similar to α, β is also learned through the training process.

[0088] Beneficial effects of the present invention:

[0089] 1. Through the series-parallel fusion of convolutional networks (CNN) and transformers, the limitations of traditional convolutional networks in global modeling capabilities are overcome, while the shortcomings of transformers in local feature extraction are solved. This hybrid structure can better handle multi-scale targets and complex scenes in drone images, effectively improving the global consistency of segmentation and the retention of local details.

[0090] Effect: Improved the model's ability to segment targets of multiple scales, complex backgrounds, and multiple angles, especially the recognition accuracy of small and large-scale targets.

[0091] 2. Through the jump connection mechanism of the U-Net structure, the convolutional features and transformer global modeling features are combined at different network levels, which can effectively fuse shallow detail information with deep semantic information, which is especially suitable for small target detection in drone images and scenes with complex backgrounds.

[0092] Effect: The network's ability to handle small targets and complex backgrounds is enhanced, so that in drone images, the model can more accurately distinguish between foreground and background, improving the overall segmentation performance.

[0093] 3. The parallel convolution module complements the transformer's shortcomings in extracting local features, especially convolution can capture the neighborhood relationship and local details in space well. By reconverting the one-dimensional sequence into a two-dimensional image format and then extracting local information through convolution operations, this module improves the extraction of local features and effectively integrates them with global features.

[0094] Effect: The segmentation accuracy of the model in edge areas and local details is significantly improved, especially the segmentation performance of objects with complex boundaries or occlusions is enhanced.

[0095] 4. Through the cross-attention mechanism, the features of RGB images and T (thermal imaging) images are fused to make full use of the complementary information of the two modalities. RGB images capture details under visible light, and T images provide thermal radiation features. The cross-attention mechanism can dynamically adjust the weights of the two modal information, achieving accurate segmentation under weak light and complex weather conditions.

[0096] The effect is: It significantly improves the target segmentation accuracy in complex environments (such as low light, haze, occlusion, etc.) and improves the robustness of the model under different lighting and climate conditions.

[0097] 5. Through the combination of the Position Attention Module (PAM) and the Channel Attention Module (CAM), the features are strengthened from the two dimensions of space and channel. PAM captures the spatial dependencies between different positions in the image, while CAM highlights important channel features by strengthening the correlation between channels. The two complement each other, making the network have stronger feature expression capabilities at different scales.

[0098] Effects: Enhanced the fusion capability of global and local features, improved the segmentation effect of complex scenes and multi-category targets, further optimized the position and channel feature expression in the image, and improved the overall segmentation accuracy.

[0099] 6. Combining the efficient feature extraction structure of convolution and transformer, the model not only has strong segmentation performance, but also optimizes computational efficiency. By using dual attention mechanisms (PAM and CAM) and parallel convolution modules (CNT), the model reduces the computational burden while still maintaining high segmentation accuracy.

[0100] The effect is: while ensuring the segmentation accuracy, the computational efficiency and reasoning speed of the model are improved, meeting the real-time requirements of drone image segmentation in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 It is a schematic diagram of the overall system flow of the present invention.

[0102] Figure 2 Schematic diagram of the base network for semantic segmentation of drone images.

[0103] Figure 3 Schematic diagram of the parallel convolution module.

[0104] Figure 4 Schematic diagram of the dual attention module. DETAILED DESCRIPTION

[0105] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0106] like Figure 1 As shown in FIG. 1 , a multimodal drone image segmentation method is shown in FIG. 1 , and the steps are as follows:

[0107] Step 1: Build a novel semantic segmenter for drone images that combines convolution with transformers;

[0108] Firstly, a hybrid encoder is introduced based on the UNet model to fuse convolution and transformer to overcome the limitations of traditional convolutional neural networks in modeling long-distance dependencies and processing large-size images.

[0109] The first part of the hybrid encoder consists of convolution blocks, which are good at extracting shallow features and local information, thereby reducing the computational burden of the transformer. The second part consists of Transformer blocks and convolution parallel modules. The Transformer block is responsible for building long-range dependencies based on local information to achieve global modeling; in each Transformer block, a parallel convolution module is integrated to make up for the Transformer's possible shortcomings in extracting local information.

[0110] A RGB-T multimodal fusion attention mechanism is designed to fuse T image information with RGB image through a cross-attention mechanism to enhance the segmentation effect in complex backgrounds, low light and occluded environments. By integrating convolution and transformer components into RGB-T multimodality, a comprehensive and powerful feature extractor is implemented, solving the problem of poor segmentation performance of the Transformer network on drone image datasets. In addition, a PCAM module is designed to extract and enhance information features from channel and spatial views, effectively fuse features of different scales, and strengthen attention to small-scale targets through contextual information. In this way, the problems of image target scale diversity and high segmentation difficulty are effectively addressed, thereby improving the segmentation performance of the network.

[0111] Step 1 is the basis of the entire segmentation method, which provides a powerful feature extraction framework by combining convolution and transformer. A semantic segmenter for drone images that integrates convolution and transformer is constructed to extract global and local features in drone images, handle multi-scale targets, improve segmentation accuracy in complex environments, and achieve information fusion in multi-modal images, namely RGB images and T images;

[0112] in:

[0113] 1. Convolutional Network:

[0114] The convolutional network module is mainly responsible for extracting local features and is good at capturing shallow features and spatial detail information in images. Especially in drone images, convolution can effectively extract local information of small-scale targets such as roads, buildings, and vehicles. The convolutional module is located at the front end of the entire network and the initial stage of feature extraction. It works in parallel with the subsequent transformer module to share the transformer's processing burden on shallow local information.

[0115] 2. Transformer module:

[0116] The Transformer block integrates the multi-head self-attention mechanism (MHSA) and the parallel convolution module (CNT), and is the core component of the segmentation network. The multi-head self-attention mechanism is responsible for extracting global features, and through the synergy with the parallel convolution module, it can capture global and local information at the same time. The role of the Transformer block is to fuse local convolution features with global attention features to form a comprehensive understanding of the target. Each Transformer block is passed through the layers to gradually refine the features and improve the accuracy of segmentation.

[0117] 3. Cross-Attention Module:

[0118] The cross-attention module is used for multimodal fusion of RGB images and T images. In this module, the query features of the RGB image are associated with the key and value features of the T image, and vice versa. The cross-attention mechanism ensures that the complementary information of the two modalities is fully utilized, so that the target can be accurately segmented in low light, complex background and other conditions. It plays a key role in cross-modal information communication and integration.

[0119] 4. PCAM module (dual attention module):

[0120] The PCAM module (position attention module and channel attention module) further improves segmentation accuracy by strengthening features from the spatial position and channel dimensions. The position attention module (PAM) can capture the spatial dependency between any two positions in the image, thereby extracting spatial features with global context. The channel attention module (CAM) focuses on extracting features between different channels and enhances the semantic information expression of the image by assigning weights to each channel. The PCAM module integrates global context information and fine channel features, making the network more accurate in segmenting the target.

[0121] These modules work together to form a highly coordinated drone image semantic segmentation network that can cope with the challenges of multimodality, complex backgrounds, and multi-scale targets, and ultimately achieve high-precision and robust drone image segmentation effects.

[0122] The segmenter in step 1 captures the local information of the drone image through a convolutional neural network (CNN), and uses a transformer to build long-range dependencies to enhance the global modeling ability, thereby improving the semantic segmentation base network of drone images (such as Figure 2 The accuracy and robustness of the proposed method are shown in Figure 2.

[0123] Step 2: Design a convolution and transformer hybrid feature extraction network based on U-shaped structure (RGB-TransUNet). The four-stage feature extraction network in step 2 is a specific implementation of the idea of ​​combining convolution and transformer proposed in step 1.

[0124] The structure in step 2 directly solves the segmentation requirements of drone images, especially the challenges of multi-scale and complex scenes in drone images;

[0125] The network is divided into four stages.

[0126] In order to fully exploit the advantages of convolution in capturing shallow features and the powerful ability of transformer in global modeling of deep features, efficient ResNetV2 convolution blocks are introduced in the first two stages.

[0127] F1=”ResNetV”“2”1(X) (1)

[0128] F2=”ResNetV”“2”2(F1) (2)

[0129] Then enter the Patch Embedding stage, through convolution, flattening, position encoding and dropout operations, and finally generate a one-dimensional vector containing position information.

[0130] F3="Flatten"("Conv"(F2)) (3)

[0131] F3=”PositionEmbedding”(F3) (4)

[0132] F3=”Dropout” (F3) (5)

[0133] In the final stage, a transformer block with stronger long-range modeling capabilities is used. Unlike ViT's multi-head self-attention mechanism, this transformer block contains a parallel convolution module (CNT) to supplement the transformer's deficiencies in local information extraction. This process is repeated 12 times, outputting a one-dimensional vector consistent with the original size.

[0134] T0=F3 (6)

[0135] T i =”TransformerBlock”(T i-1 )”for”i=1, 2,…,12 (7)

[0136] TransformerBlock(T)="LN"(T+"CNT"("MHSA"("LN"(T)))) (8)

[0137] LN refers to Layer Normalization, which is a normalization technique in deep learning and is usually used in the Transformer module in neural networks to accelerate model training and improve stability. Unlike Batch Normalization, Layer Normalization normalizes all neurons in a single sample.

[0138] MHSA is the abbreviation of Multi-Head Self-Attention, which is one of the core components in the Transformer model and is used to capture the dependencies between different positions in the input sequence. MHSA can simultaneously process information from different representation spaces through a parallel multi-head self-attention mechanism, thereby improving the representation ability of the model.

[0139] After these four stages, the token embeddings are: "Embeddings" = [F1, F2, F3, T 12 ];

[0140] Embeddings represents the embedded features after network feature extraction, including the combination of shallow convolution features and deep transformer features. By fusing features from different stages, it ensures accurate modeling of objects of different scales in the image.

[0141]

[0142] In step 2, the design of the RGB-TransUNet network combines the powerful feature extraction capability of the UNet network and the global modeling capability of the transformer. The U-shaped structure of UNet enables it to capture high-resolution local features in the shallow layer and fuse these features with the global features in the deep layer through jump connections. In order to optimize the segmentation of drone images, especially to process multi-scale targets in the bird's-eye view, RGB-TransUNet introduces the transformer module in the U-shaped architecture to further enhance the global modeling capability;

[0143] Step 3: To supplement the missing local information in the transformer block, a parallel convolution module (such as Figure 3 shown).

[0144] The input characteristics of the i-th transformer block are recorded as:

[0145] For parallel convolution modules and multi-head self-attention mechanisms, f i are the same feature input, but they process this feature differently. The parallel convolutional module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f i , and then process the features differently in different branches.

[0146] f i It has two branches, one of which is a multi-head self-attention mechanism used to i Modeling long-range global information;

[0147] f i It is processed through two branches, one of which is a multi-head self-attention mechanism used to i Modeling long-range global information.

[0148] Specifically, the input features are first mapped to query (Q), key (K), and value (V) through the projection matrix, that is, Q, K, V = f i W Q , f i W K , f i W V ,in, Represent the projection matrices of Q, K, and V respectively. The calculation formula for self-attention operation is:

[0149]

[0150] The attention module is iterated h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of self-attention heads. The features from h attention heads are concatenated in the channel dimension to form the output of MHSA. After MHSA processing, we get f i The global information of i g =MHSA(f i ).

[0151] The other branch is a parallel convolution module to capture f i Since the self-attention mechanism is applied to one-dimensional sequences, and the convolution operation is applied to two-dimensional images, the sequence needs to be converted into the corresponding image format first. Next, the spatial local information is extracted through the convolution operation, and the convolution result is converted back to a one-dimensional sequence to obtain f i The local information of is expressed as:

[0152] f i l =CNT i (f i )=img2seq(Conv i (seq2img(f i ))) (11)

[0153] Among them, CNT i (·) denotes a parallel convolutional module, which consists of an “img”2seq(·) operation, three consecutive convolutional layers, and a seq2img(·) operation, which combines the label features with the local context information.

[0154] In order to supplement the deficiency of transformer blocks in local feature extraction, a parallel convolution module (CNT) is introduced. The purpose of this module is to perform convolution operations in parallel in each transformer block to effectively capture local information and supplement the transformer's long-range dependency modeling capabilities;

[0155] Step 4: To enhance the segmentation effect in complex backgrounds, low light and occlusion environments, in the RGB-T multimodal fusion task, the RGB image and the T image each provide complementary information. The cross-attention mechanism helps to achieve efficient information fusion between the two modalities, thereby improving the accuracy of the task. The calculation process of the cross-attention is as follows: Through the cross-attention mechanism, the query features of the RGB image are associated with the key and value features of the T image to find the relevant parts between the RGB image and the T image. Cross-attention calculation formula:

[0156]

[0157] Among them, Q represents the query from the RGB image, K and V represent the key and value from the T image respectively, and d is the scaling factor used to avoid the problem of excessively large values ​​due to the inner product of high-dimensional feature vectors. Through the above formula, cross attention can dynamically adjust the information fusion between RGB and T images, improving the segmentation accuracy of targets in complex environments.

[0158] f RGB-T =CA(f i ) (13)

[0159] Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA:

[0160] f i =f i g +f i l +f RGB-T (14)

[0161] After the fused features pass through the FNN, they are input into the next transformer block:

[0162] f i+1 =FFN(f i )+f i (15)

[0163] The advantage of RGB-T multimodal fusion is that it captures complementary information between modalities: cross attention enables RGB images to obtain features from T images that are difficult to capture in low light or complex environments. By combining the advantages of these two modalities, the model can make more accurate predictions. Dynamically focus on relevant areas: The cross attention mechanism can dynamically adjust according to task needs, focusing on relevant areas in the image, thereby helping the model make better decisions in multimodal tasks. Enhanced robustness: In some cases (such as insufficient light in RGB images or weak signals in T images), cross attention can achieve information complementarity between different modalities, making the model more robust in multimodal environments.

[0164] Step 5: Optimize the feature fusion module of the UNet network. This is to address the shortcomings of traditional UNet and Transformer in utilizing the intrinsic position and channel features of the image, as well as the challenges in parameter efficiency and computational complexity. Through an efficient feature fusion module, called the dual attention module (PCAM-Block) (e.g. Figure 4 As shown in Figure 2, the module is used as a feature extractor. The module consists of two core parts: the Position Attention Module (PAM) and the Channel Attention Module (CAM), which are used to integrate and extract the position information and channel features of the image.

[0165] Among them, the Position Attention Module (PAM) can capture the spatial dependency between any two points in the feature map. It updates a specific feature by taking a weighted sum of all position features, where the weight is calculated based on the feature similarity between the two positions. Therefore, PAM can effectively extract spatial features with important meanings.

[0166] Input local feature A∈R C×H×W , where C is the number of channels, H is the height, and W is the width. Three new feature maps are obtained through convolution operations: B, C, D∈R C×H×W . Reshape B and C into R C×N , where N = H × W represents the number of pixels, performing matrix multiplication Use the softmax function to normalize the elements in the matrix S into attention weights:

[0167]

[0168] Reshape the matrix D into R C×N , performs matrix multiplication and reshape the result into R C×H×W , multiply the result by the parameter, and perform element-wise summation with feature A to get the final output:

[0169]

[0170] The weight α is initially set to 0 and is gradually learned through training. PAM performs well in spatial feature extraction. Since E is the weighted sum of all position features and the original features, it has global context features and achieves context aggregation through the spatial attention map. This ensures that while effectively extracting position features, global context information is retained.

[0171] CAM (Channel Attention) is good at extracting channel features. Unlike PAM, the original feature A∈R C×H×W Reshape to R C×N Then do a matrix multiplication between A and its transpose Next, apply the softmax layer to obtain the channel attention map X∈R C×C :

[0172]

[0173] Perform matrix multiplication between the transpose of X and A Reshape the result Y into R C×H×W , multiply the result Y by the scale parameter β, and perform element-by-element sum operation with the original feature A to obtain the final output E∈R C×H×W :

[0174] E=β·Y+A (19)

[0175] Similar to α, β is also learned through the training process. In the channel attention module (CAM), the process of extracting channel features is similar to PAM, and the final feature of each channel is the weighted sum of all channels and the original features. This mechanism gives CAM a powerful channel feature extraction capability.

[0176] in:

[0177] 1. PAM (Position Attention Module):

[0178] Main function: PAM is responsible for capturing the dependency between any two spatial positions in the image. It weights and updates the features of each position by calculating the similarity between the positions in the image feature map. This is very effective for capturing complex spatial relationships and target distribution in images, especially in drone images, where the distribution of objects may be random from a bird's-eye view.

[0179] Focus: PAM mainly processes image features from the spatial dimension, focusing on the relationship between each spatial position (pixel or area), helping the network understand the structural information in the image.

[0180] Application scenarios: PAM can effectively deal with complex spatial distribution in images, especially capturing dependencies between distant pixels and enhancing global context understanding. For example, in images taken by drones, different targets (such as buildings) may be far apart, but their correlation can be enhanced through PAM.

[0181] 2.CAM (Channel Attention Module):

[0182] Main function: CAM is responsible for capturing the relationship between different channels in the feature map in the channel dimension. Channel attention calculates the dependencies between different channels and gives higher weights to important channels, so that important features in the image are strengthened. Each channel usually represents different types of features (such as edges, textures, colors, etc.). CAM improves the network's feature expression ability by learning the correlation between different channels.

[0183] Focus: CAM mainly processes image features from the channel dimension, focusing on the information contained in each channel, making the understanding of target features more comprehensive and detailed.

[0184] Application scenarios: CAM can enhance the feature expression capabilities of different channels, especially those channels that play a key role in semantic segmentation tasks. For drone image segmentation, CAM can highlight the features of different categories such as buildings, roads, and vegetation, making segmentation more accurate.

[0185] The combination of PAM and CAM in the entire network enables the network to capture local details and global information in the image at the same time, and enhance the understanding of different channel features. This synergy can improve the segmentation performance of the network, especially in multimodal tasks (RGB-T image fusion), where the features of different modalities are more comprehensively expressed through PAM and CAM. The combination of the two enables the network to effectively process image features in both spatial and channel dimensions, improving the segmentation accuracy and robustness of complex targets.

[0186] In order to objectively evaluate the performance of the present invention on the specified data set, the precision-recall curve (PRC) and average precision (AP) are used as the main and authoritative evaluation indicators to quantitatively analyze various drone image segmentation methods. The calculation formula is as follows:

[0187]

[0188] Among them, TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.

[0189]

[0190] Among them, GT represents the true value box and PR represents the predicted box.

[0191]

[0192] Among them, P and R represent Precision and Recall respectively, N cls The network of the present invention is directly trained using the above dataset and evaluated using the aforementioned indicators.

[0193] Attachment Figure 1 To further explain, the network architecture of the entire multimodal drone image segmentation method is divided into multiple modules, and the specific functions of each part are as follows:

[0194] 1.RGB-Transformer encoder module:

[0195] The module on the left shows the workflow of the RGB-Transformer encoder. This module contains 12 layers of Transformer layers, which model the global context information of the input sequence through the multi-head self-attention mechanism (MHSA) and the cross attention mechanism (CA).

[0196] Input sequence: The features from the RGB image are first transformed into a one-dimensional sequence through image-to-sequence (Img2Seq) conversion, and then enter the RGB-Transformer.

[0197] MSA (Multi-Head Self-Attention): Responsible for capturing the global dependencies between different positions in the sequence, which helps to extract long-range spatial features.

[0198] CA (Cross Attention): Focuses on the feature relationship in the channel dimension and enhances the feature expression ability of the network by giving more weights to important channels.

[0199] 2.CNN feature extraction module:

[0200] In the middle of the figure, the CNN module is used to extract the basic features of RGB and T images. This module captures local features in the image, such as edges, textures, and local structures, through multi-layer convolution operations.

[0201] PCAM-Block (Dual Attention Mechanism Module): Located after the CNN feature map, it is used to establish dependencies between spatial positions and channels. It combines the functions of PAM (Position Attention Module) and CAM (Channel Attention Module), further optimizes the feature expression ability, and enhances the network's understanding of complex image structures.

[0202] 3. Multi-scale feature fusion and downsampling:

[0203] The figure shows the further processing of features after 12 layers of Transformer layers. Through upsampling and downsampling operations, feature maps of different resolutions are gradually processed and fused to ensure that target features are captured at multiple scales. The purpose of this design is to handle targets of different sizes and shapes, especially in drone images. This multi-scale processing is especially important for small target detection.

[0204] Conv3x3, ReLU: Convolutional layers and ReLU activation functions are used to perform further nonlinear transformation and spatial detail optimization on features.

[0205] 4.PAM&CAM module:

[0206] PAM (Position Attention Module): Captures the dependencies between different spatial positions in the feature map, ensuring that the model can find associated regions in the global space, thereby better understanding the location of objects in the image.

[0207] CAM (Channel Attention Module): It highlights important features by weighting different channels in the channel dimension, ensuring that the most discriminative features are used for segmentation tasks.

[0208] PCAM-Block combines the advantages of PAM and CAM by combining them, improving the feature extraction capabilities in the spatial and channel dimensions respectively.

[0209] 5. Upsampling and output segmentation results:

[0210] The right part shows how the original resolution is gradually restored through the upsampled feature map, and finally the final segmentation result is generated by the segmentation head. Multiple upsampling ensures that image details are restored at a higher resolution.

[0211] Finally, the segmentation map output by the network can accurately segment the target object in the image.

[0212] Attached Figure 1 The network architecture presented combines the global modeling capabilities of convolutional neural networks (CNN) and Transformer, and uses multiple attention mechanisms (MSA, CA, PAM, CAM) to extract and fuse image features at multiple levels. RGB images and T images are fused through cross-attention mechanisms, and finally generate high-precision semantic segmentation results, which are particularly suitable for processing complex scenes and multi-scale targets in drone images.

[0213] In the present invention, multimodality refers to the use of multiple different types of sensors or data sources to obtain and process image information in image processing. In the present invention, the multimodality used includes RGB images and T (thermal imaging) images. RGB images capture color information within the visible spectrum and are usually used to record the appearance and details of objects; while T images can capture infrared thermal radiation information of objects, and are particularly suitable for object detection and recognition in weak light or complex environments.

[0214] By fusing the information of these two modalities, RGB-T multimodal image segmentation can supplement the temperature feature information provided by thermal imaging while maintaining the target appearance information, enhancing the target segmentation effect under harsh conditions, thereby improving the segmentation accuracy and robustness of UAV images.

[0215] Now, the present invention is described in more detail in conjunction with an experimental example, and the specific steps are as follows:

[0216] 1. Dataset selection

[0217] The present invention uses the following two datasets in the UAV image semantic segmentation task:

[0218] The Semantic Drone dataset focuses on semantic understanding of urban scenes to improve the safety of autonomous drone flight and landing procedures. The image depicts more than 20 houses taken from the lowest point (bird's eye view) perspective at an altitude of 5 to 30 meters above the ground. A high-resolution camera is used to acquire images of size 6000x4000px (24Mpx), and the semantic segmentation dataset includes 20 categories. The training set contains 400 publicly available images and the test set contains 200 private images.

[0219] The self-made drone image RGB-T dataset includes images captured using drones at altitudes ranging from 5 to 100 meters. The dataset provides 10,000 8000x6000px house building RGB images and corresponding T images, including ground-truth masks for 2 categories.

[0220] 2. Selection of evaluation indicators

[0221] In order to objectively evaluate the performance of the present invention on a given dataset, the precision-recall curve (PRC) and average precision (AP) are used as standard and authoritative evaluation indicators to quantitatively evaluate different UAV image semantic segmentation methods. Precision, Recall, I"oU", AP, and mAP are calculated as follows:

[0222]

[0223] 3. Construct a feature extraction network based on a U-shaped architecture with a mixture of convolution and transformer and pre-training instructions

[0224] The Aeroscapes dataset is a new large-scale benchmark dataset containing 3269 high-resolution images taken by drones, mainly covering urban and rural scenes. This paper uses all the images in the Aeroscapes dataset for self-supervised pre-training. During the pre-training process, the batch size is set to 128, and the AdamW optimizer is used, which is evenly distributed on two 4090 GPUs. If not otherwise specified, the model is trained for 50 epochs according to the default settings of MAE.

[0225] 4. Based on the pre-trained model, complete model fine-tuning in downstream segmentation tasks.

[0226] 5. In the data preprocessing of the downstream segmentation task, the present invention performs unified processing on the input image. First, the image is resized to 256x256 by image scaling and edge padding, and a horizontal flip with a probability of 0.5 is applied for data enhancement. Then, the image is normalized to map the pixel values ​​between 0 and 1. Finally, the pixel distribution of the image is converted from RGB to RRRGGGBBB through color space conversion.

[0227] As a de facto standard, all models were trained using the AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.1. The momentum parameter was 0.9, which was used to assign weights to the last updated gradient. In order to fully train the models, a 6-fold training schedule was used, with a total of 72 epochs and a batch size of 1. The learning rate was adjusted using a multi-step scheduler, which reduced the learning rate to 1 / 10 at epochs 48 and 66. A layered learning rate decay strategy was also used with a decay rate of 0.7.

[0228] 6. Comparison of results

[0229] To illustrate the superiority of the present invention, the present invention is compared with five relatively advanced methods, including "Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedicalimage segmentation [C] / / Medical image computing and computer-assisted intervention – MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015: 234-241.", "He K, Gkioxari G, Dollár P, et al. Mask r-cnn [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.", "Liu M, Fu B, Xie S, et al. Comparison of multi-source satellite images for classifying marshvegetation using DeepLabV3 Plus deep learning algorithm[J].EcologicalIndicators, 2021, 125:107562.","Chen J, Lu Y, Yu Q, et al.Transunet: Transformersmake strong encoders for medical image segmentation[J].arXiv preprint arXiv:2102.04306, 2021.","Liu Z, Lin Y, Cao Y, et al.Swin transformer: Hierarchicalvision transformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.".

[0230] To ensure fair comparison, all experimental results are generated under the same settings and tested on the split test sets of the two datasets at mAP@50. The average accuracy of the RGB-TransUNet network on the Semantic Drone dataset is 82.68%, 1.16% higher than Deeplabv3+; the average accuracy on the self-made drone image dataset is 84.57%, 1.25% higher than Transunet, which proves the superiority of the RGB-TransUNet of the present invention.

[0231] The multimodal segmentation network proposed in the present invention combines the local feature extraction capability of convolutional neural network (CNN) and the global modeling capability of transformer (Transformer), which can better capture the spatial relationship of the target in the image, thereby improving the accuracy of segmentation.

[0232] The network proposed in this paper can capture the detailed information of different targets under different lighting conditions through RGB-T multimodal fusion, and solve the segmentation problem of multi-scale targets by utilizing the long-range dependency modeling capability of the transformer block.

[0233] The invention introduces thermal imaging (T-image), so that the system can still accurately segment the target in low light or bad weather. The cross-attention mechanism is used to fuse RGB and T-image information, effectively improving the segmentation effect in complex environments.

[0234] The present invention optimizes the feature extraction efficiency by introducing a dual attention module (PCAM-Block) and balances the accuracy and speed in the combination of convolution and transformer to meet the real-time requirements of drone image segmentation.

[0235] Through these designs, the multimodal UAV image segmentation method of the present invention can adapt to the special challenges in UAV images and improve segmentation accuracy and robustness, especially under complex and changeable environmental conditions.

Claims

1. A multimodal drone image segmentation method, characterized in that: The steps include: Step 1: Based on the UNet model, a hybrid encoder is introduced to fuse convolution and transformer to form a feature extraction framework and build a base network for semantic segmentation of drone images; Step 2: Through the convolutional neural network (CNN) module, high-resolution local features are captured in the shallow layer, and the global features extracted from the deep features of the image through the multi-head self-attention mechanism in the transformer module are fused with the local features through jump connections; Step 3: Through the parallel convolution module and multi-head self-attention mechanism, the features are optimized from the local and global perspectives respectively. The parallel convolution module extracts detailed local information, and the multi-head self-attention mechanism captures long-range global dependencies; Step 4: Based on the local and global features extracted in steps 2 and 3, the feature information is fused into RGB-T multimodal form through a cross-attention mechanism module to improve the segmentation effect. Step 5: Optimize the feature expression of the basic feature extraction framework constructed by combining convolution and transformer in steps 1 and 2, and build a dual attention module to integrate position and channel feature information, thereby optimizing the local and global information processing of the image.

2. A multimodal drone image segmentation method according to claim 1, characterized in that: The base network for semantic segmentation of drone images in step 1 is used to extract global and local features in drone images, process multi-scale targets, improve segmentation accuracy in complex environments, and realize information fusion in multi-modal images, i.e., RGB images and T images; The base network for semantic segmentation of drone images includes a convolution module, a Transformer block, a cross-attention module and a PCAM module; the convolution module is at the front end of the network, used for preliminary local feature extraction of the input image, and is fused with the output of the deep transformer in the jump connection; the Transformer block fuses convolution and transformer, and realizes accurate modeling of targets of different scales through multi-level feature transmission and fusion; the cross-attention module integrates the complementary information of RGB images and T images to ensure the coordinated processing of multimodal images; the PCAM module ensures that targets of different scales and categories can be accurately segmented by enhancing spatial and channel features; Input of the drone image semantic segmentation network: RGB images: ordinary visible light images from drones, containing color and detail information of the scene, used to capture visual features in the image; T-image: Infrared image acquired by the drone using thermal imaging equipment, used for target detection in weak light and complex background environments; These two types of images are used as multimodal inputs and enter the network for feature extraction and processing; Output of the drone image semantic segmentation network: Segmentation Mask: It is an image of the same size as the input image, where each pixel corresponds to a class label in the image.

3. The multimodal drone image segmentation method according to claim 1, characterized in that: In step 2, local and global features in RGB and T images are extracted through the combination of convolution and transformer, and the features are fused through U-shaped structure and jump connection to ensure the retention of image features of different scales; The extraction of local features is used to accurately capture small targets and details in the image, and the extraction of global features ensures the segmentation accuracy of large-scale targets and backgrounds; The feature extraction network is divided into four stages. The first two stages use convolution to extract local features, and the last two stages use transformers for global modeling, which are then fused through jump connections. In the first two stages, ResNetV2 convolutional blocks were used; F1=”ResNetV””2”1(X) (1) F2=”ResNetV””2”2(F1) (2) Next, in the third stage, the Patch Embedding stage, the input is flattened (Flatten), positioned (PositionEmbedding), and dropout is applied using convolution operations, ultimately generating a one-dimensional vector containing position information; F3="Flatten"("Conv"(F2)) (3) F3=”PositionEmbedding”(F3) (4) F3=”Dropout” (F3) (5) In the last stage, a transformer block containing a parallel convolution module is used to increase the transformer's ability to perceive local information. After 12 repetitions, a one-dimensional vector with the same size as the original is output. T0=F3 (6) T i =”TransformerBlock”(T i-1 )”for”i=1,2,…,12 (7) TransformerBlock(T)="LN"(T+"CNT"("MHSA"("LN"(T)))) (8) LN refers to Layer Normalization, and MHSA is used to capture the dependencies between different positions in the input sequence. MHSA simultaneously processes information from different representation spaces through a parallel multi-head self-attention mechanism.

4. The multimodal drone image segmentation method according to claim 3, characterized in that: The features obtained in the four stages are: "Embeddings" = [F1, F2, F3, T 12 ]; Embeddings represents the embedded features after network feature extraction, including the combination of shallow convolution features and deep transformer features. By fusing features from different stages, it ensures accurate modeling of objects of different scales in the image.

5. The multimodal drone image segmentation method according to claim 4, characterized in that: F1 represents the first layer of features obtained by convolution operation, whose spatial resolution is one quarter of the input image resolution, and the feature dimension is C1: Here, H and W are the height and width of the input image, respectively, and C1 is the number of feature channels of this layer; F2 represents the second layer of features obtained through convolution operation, whose spatial resolution is one sixteenth of the input image resolution, and the feature dimension is C2: H and W are the height and width of the input image, respectively, and C2 is the number of feature channels of this layer; F3 represents the third layer of features obtained through deep convolution and transformer operations. Its spatial resolution is also one sixteenth of the input image. After flattening, the embedding dimension obtained is C3: HW represents the total number of pixels of the input image, and C3 is the number of channels of the deep features; T 12 represents the features after being processed by the transformer module, and its embedding dimension is the same as F3: This layer of features is used to strengthen global feature modeling, especially the long-range dependency modeling of multi-scale objects.

6. A multimodal drone image segmentation method according to claim 5, characterized in that: In step 3, let the i-th transformer block f i The input features are: For parallel convolution modules and multi-head self-attention mechanisms, f i is the same feature input, the parallel convolution module (CNT) and the multi-head self-attention (MSHA) mechanism in step 3 share the same input f i , and then process the features differently in different branches; f i It has two branches, one of which is a multi-head self-attention mechanism used to i Modeling long-range global information; The other branch is a parallel convolution module used to extract f i In order to integrate the local information in the network, the multi-head self-attention mechanism is applied to the one-dimensional sequence, while the parallel convolution module is applied to the two-dimensional image.

7. A multimodal drone image segmentation method according to claim 6, characterized in that: In the multi-head self-attention mechanism, the input mark f is first transformed into i Projected onto query (Q), key (K) and value (V), i.e. Q, K, V = f i W Q , f i W K , f i W V ,in Denote the projection matrices of Q, K, and V respectively, and the self-attention operation is calculated as follows: K T : represents the transpose operation of the key vector, which is used to perform dot product operation with the query vector Q to calculate the correlation between each query vector and all key vectors, with a dimension of R D×N ; represents the scaled dot-product attention mechanism, which is used for QK T To zoom, use As a scaling factor to balance the dot product result, is the scaling factor of the dot product result, where D is the dimension of the query and key vectors; Through the softmax function, the correlation value between the query vector and each key vector is converted into a weighting coefficient, which is used to weight the value vector V. The weighted sum of all key vectors is 1. The attention module is iterated h times to construct a multi-head self-attention mechanism (MHSA), where h represents the number of self-attention heads. The features from h attention heads are connected along the channel dimension to form the output of MHSA. After passing through MHSA, f is obtained. i The global information of i g =MHSA(f i ).

8. The multimodal drone image segmentation method according to claim 6, characterized in that: In the parallel convolution module, first, the input features of the transformer module are flattened into a one-dimensional sequence and then reconverted into the corresponding two-dimensional image format (seq2img). Then, the spatial local information is extracted through the convolution operation, and the convolution result is converted back into a one-dimensional sequence (img2seq) to obtain f i The local information f i l , expressed as: f i l =CNT i (f i )=img2seq(Conv i (seq2img(f i ))) (11) Among them, CNT i (·) denotes a parallel convolutional module, which consists of an img2seq(·), three consecutive convolutional layers, and a seq2img(·) operation, which integrates the label features with the local context information; By directly adding the global features generated by the multi-head self-attention module (MHSA) and the local features extracted by the parallel convolution module (CNT), the fusion of global information and local information is achieved. The formula is as follows: f i gl =f i g +f i l f i gl : The fused features in the i-th transformer block, f i g represents the global features extracted by the transformer self-attention mechanism (MHSA), f i l Represents the local features extracted by the parallel convolution module (CNT).

9. The multimodal drone image segmentation method according to claim 1, characterized in that: In step 4, the fusion of local features: the local features extracted by the convolutional network and the parallel convolutional module, i.e., the features at the detail level in the RGB and T images, can be combined through the cross attention mechanism, so that the detail information of the RGB image and the supplementary information captured in the T image under weak light or complex background work together; Fusion of global features: The global features extracted by the transformer block complement the information between modalities through the cross-attention mechanism. The RGB image provides global information under visible light conditions, and the T image enhances the global background understanding in low-light environments. The cross attention mechanism combines the query (Query) of the RGB image with the key (Key) and value (Valu e ) to identify the relevant parts of the RGB image and the T image, and the query of the T image is also associated with the key and value of the RGB image to achieve two-way information fusion; The calculation process of cross attention is as follows: Query: representation from RGB image; Key and Value: Feature representation from another modality T image. Through cross attention, the query features of the RGB image are dot-producted with the key and value features of the T image to find the parts of the T image that are related to the RGB image, and vice versa. Where Q RGB : represents the query vector (Query) extracted from the RGB image, K T : represents the key vector (Key) extracted from the T image, d is the dimension of the key vector, V T : Represents the value vector in the T image. Cross attention can dynamically adjust the information fusion between RGB and T images, improving the segmentation accuracy of targets in complex environments. f RGB-T =CA(f i ) (13) Feature fusion is achieved by directly adding the global features after MHSA, the local features after CNT, and the multimodal features after CA: f i =f i gl +f RGB-T (14) After the fused features pass through the FNN, they are input into the next transformer block: f i+1 =FFN(f i ) + f i (15) 10. The multimodal drone image segmentation method according to claim 1, characterized in that: In step 5, the dual attention module (PCAM-Block) is used as a feature extractor, including a position attention module (PAM) and a channel attention module (CAM), which are responsible for extracting position and channel features respectively; The position attention module (PAM) can capture the spatial dependency between any two points in the feature map and update a specific feature by taking a weighted sum of all position features, where the weight is calculated based on the feature similarity between the two positions; Input local feature A∈R C×H×W , where C is the number of channels, H is the height, and W is the width. Three new feature maps are obtained through convolution operations: B, C, D∈R C×H×W , reshape B and C into R C×N , where N = H × W represents the number of pixels, and performs matrix multiplication C T B. Use the softmax function to normalize the elements in the matrix S to the attention weights: Reshape the matrix D into R C×N , perform matrix multiplication D T S, and reshape the result into R C×H×W , multiply the result by the parameter, and perform element-wise summation with feature A to get the final output: E=α·(D T S)+A (17) The weight α is initially set to 0 and is gradually learned through training; In the CAM (channel attention), the original feature A∈R C×H×W Reshape to R C×N Then perform matrix multiplication M=AA between A and its transpose T , then, apply the softmax layer to obtain the channel attention map X∈R C×C : Perform matrix multiplication Y = X between the transpose of X and A T +A, reshape the result Y into R C×H×W , multiply the result Y by the scale parameter β, and perform element-by-element sum operation with the original feature A to obtain the final output E∈R C×H×W : E=β·Y+A (19) Similar to α, β is also learned through the training process.

Images