Remote sensing image ship detection method based on multi-modal large model prompt generation

Abstract

The invention provides a remote sensing image ship detection method based on multi-modal large model prompt generation. The method comprises the following steps: step 1, obtaining a remote sensing ship image data set; step 2, inputting training set data into a multi-modal large model to generate text prompt features; step 3, inputting the training set into the improved RT-DETR model to obtain multi-scale fusion image features; step 4, designing a cross-modal fusion network to obtain cross-modal fusion features; 5, generating a bounding box and a ship type; step 6, training the improved RT-DETR model to obtain a trained real-time remote sensing image ship detection model; and 7, inputting the test set into the real-time remote sensing image ship detection model obtained in the step 6 to obtain the position information and category of a target ship. According to the method, effective fusion between image and text modals can be realized, so that useful multi-modal information is fully utilized.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image target detection and recognition technology, and particularly relates to a method for ship detection in remote sensing images based on multimodal large model prompting generation. Background Technology

[0002] Remote sensing image target detection and recognition technology refers to the automatic detection and recognition of targets in remote sensing images using computer vision and deep learning techniques. Target detection and recognition in remote sensing images has always been a research focus in image processing and pattern recognition. Deep learning-based ship detection and recognition methods enable large-scale sea surface monitoring and have significant application value. Specifically, this technology can be widely applied to tasks such as marine resource management, monitoring illegal fishing activities, and providing maritime search and rescue support.

[0003] Currently, deep learning-based object detection and recognition methods can be divided into two categories: two-stage object detection algorithms represented by R-CNN (Region-based Convolutional Neural Networks) and single-stage object detection algorithms represented by YOLO (YouOnly Look Once). However, in practical applications, because remote sensing images cover a large area of ​​the Earth's surface, ship targets are relatively small and sparsely distributed in the images. Most current object detection and recognition methods are more suitable for images of natural scenes and struggle to accurately identify ship targets in remote sensing images, which is detrimental to ship detection tasks in remote sensing images. Furthermore, the shape and spatial distribution of ships in remote sensing images are easily affected by factors such as lighting and weather, leading to low accuracy in ship detection tasks. Summary of the Invention

[0004] Objective of the Invention: The technical problem to be solved by this invention is to address the shortcomings of existing technologies by providing a method for ship detection in remote sensing images based on multimodal large model prompting, comprising the following steps:

[0005] Step 1: Obtain the remote sensing ship image dataset and divide it into training set, validation set and test set according to the proportions;

[0006] Step 2: Input the training set data into the multimodal large model and use the multimodal large model to generate text prompt features;

[0007] Step 3: Using the Real-Time Detection Transformer as the base model, improve the feature interaction network and multi-scale feature fusion network in the Real-Time Detection Transformer (RT-DETR) model, and input the training set data into the improved RT-DETR model to obtain multi-scale fused image features.

[0008] Step 4: Design a cross-modal fusion network to fuse the multi-scale fused image features with the text prompt features obtained in Step 2 to obtain cross-modal fusion features;

[0009] Step 5: Input the cross-modal fusion features obtained in Step 4 into the Transformer decoder and the detection head network. The features decoded by the Transformer decoder are then optimized by the detection head to generate bounding boxes and ship categories.

[0010] Step 6: Train the improved RT-DETR model based on the training set and validation set to obtain the trained real-time remote sensing image ship detection model.

[0011] Step 7: Input the test set into the ship detection model of the real-time remote sensing image obtained in Step 6 to obtain the position information and category of the target ship.

[0012] In step 1, the acquired remote sensing ship images are preprocessed by scaling the original images to a standard size of 640×640 pixels and labeling the ship targets in the remote sensing ship image data to obtain the location category labels of the ship images.

[0013] Step 2 includes the following steps:

[0014] Step 2-1: Input the processed remote sensing ship image and the prompt word "Detect Ship" from Step 1 into the pre-trained multimodal large model PaliGemma;

[0015] Step 2-2: First, the image is converted into a series of image tags by an image encoder. The image tags are then projected through a linear projection layer.

[0016] Steps 2-3: The SentencePiece segmenter of the pre-trained language model Gemma is used to convert the prompt word Detect Ship into a series of text tags. The text tags are then embedded as prefixes through the lexical embedding layer of the pre-trained language model Gemma.

[0017] Steps 2-4: Create the tag sequence input to the decoder, represented as:

[0018] Tag sequence = [image tag, BOS, prefix tag, SEP, suffix tag, EOS],

[0019] Among them, the BOS mark indicates the beginning of a text mark, the SEP mark is used as a separator, with \n as the SEP mark, and the EOS mark indicates the end of a text mark;

[0020] Steps 2-5: Decode the suffix marker sequence F using the decoder.p F p That is, text prompt features.

[0021] In step 3, RT-DETR is used as the base model; ResNet50 is selected as the backbone network of the RT-DETR model.

[0022] The SwiftFormer is used as the encoder, and an additive attention block is introduced to replace the quadratic matrix multiplication operation of the traditional self-attention mechanism in the RT-DETR model.

[0023] Improve the multi-scale feature fusion network and establish a small target feature enhancement pyramid network;

[0024] A small target detection layer is introduced, and features rich in small target information are obtained through the spatial depth transformation convolutional network SPDConv and fed into the feature map P3 for fusion.

[0025] We introduce the concept of Cross Stage Partial (CSP) and the full kernel OmniKernel network, and design the CSP-OmniKernel module for feature fusion.

[0026] The Cross-Stage Partial Connection (CSP) concept involves splitting the network's feature map into two parts. One part is fed into the network for feature extraction, while the other part is directly concatenated to the output of the first part.

[0027] The full-kernel OmniKernel network consists of three branches: the global branch, the big kernel branch, and the local branch.

[0028] Step 3 includes the following steps:

[0029] Step 3-1: Input the remote sensing ship image processed in Step 1 into the backbone network ResNet50 to obtain the feature maps {P2,P3,P4,P5} after processing by the last four layers of ResNet50.

[0030] Step 3-2: Input feature map P5 into the feature interaction network to obtain attention-weighted feature map F5. The feature interaction network uses a single-layer SwiftFormer encoder.

[0031] The SwiftFormer encoder performs the following operations:

[0032] Step 3-2-11: Feed feature map P5 into the first encoding block. The first encoding block includes a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer to obtain feature map P5. 1 ;

[0033] Step 3-2-12, transfer feature map P5 1 The feature map P5 is obtained by feeding it into the second encoding block, namely the additive attention block. 2 ;

[0034] Step 3-2-13, transfer feature map P5 2 The third encoding block, namely the linear block, is fed into the system. The linear block consists of two 1×1 pointwise convolutional layers, batch normalization, and GeLU activation function to obtain the output feature map F5.

[0035] The SwiftFormer encoder is represented as:

[0036] P5 1 =Conv1(DWConv BN (P5)

[0037] P5 2 =QK(P5) 1 )+P5 1

[0038] F5 = Conv1(Conv BN,1,G (P5 2 ))+P5 2

[0039] Among them, depthwise separable convolution DWConv BN Conv1 represents a 3×3 depthwise convolutional layer, and Conv2 represents a 1×1 pointwise convolutional layer. BN,1,G This represents batch normalization, a 1×1 pointwise convolutional layer, and the GeLU activation function; QK represents an effective additive attention block.

[0040] The additive attention block performs the following operations:

[0041] Step 3-2-21, input feature map P5 1 By querying the transformation matrix W q Bond transformation matrix W k Transform into a query matrix Q and a key matrix K, where the query matrix Q is multiplied by the learnable parameter vector w. a The global attention query vector α is generated by learning the attention weights of the query, and is represented as follows:

[0042]

[0043] Where d is the dimension of the input feature matrix;

[0044] Step 3-2-22: Pool the query matrix according to the learned attention weights to obtain the global query vector. As shown below:

[0045]

[0046] in Let n represent the real number space, where n is the number of input feature matrices, and α is the number of input feature matrices. i Q is the global attention query vector of the i-th feature matrix. i It is the query matrix of the i-th feature matrix;

[0047] Step 3-2-23: Perform element-wise product encoding of the global query vector q and the key matrix K to obtain the global context matrix W. c The output of the additive attention block, P5 2 Described as:

[0048]

[0049] in, Let T represent the normalized query matrix, and T represent the linear transformation.

[0050] Step 3-2-24: Input the attention-weighted feature map F5 and feature maps {P2,P3,P4} together into the multi-scale feature fusion network to obtain multi-scale fused image features. The multi-scale feature fusion network adopts a small target feature enhancement pyramid network.

[0051] The small target feature enhancement pyramid network performs the following operations:

[0052] Step 3-2-31, P2, P3, P4, F5 represent feature maps at different resolutions, from high to low;

[0053] First, feature map P2 is transformed into feature X1 through spatial depth transformation convolution operation;

[0054] Step 3-2-32: Feature map F5 is upsampled to achieve the same resolution as P4.

[0055] Step 3-2-33: Feature map P4 is concatenated with the upsampled output of F5 after passing through a 3×3 convolutional layer to obtain feature X2;

[0056] Step 3-2-34: After feature X2 is processed by RepC3 residual block, it is passed through a 3×3 convolutional layer to obtain feature X3. The output of feature X3 after upsampling is concatenated with X1 and P3 to obtain feature X4.

[0057] Step 3-2-35: Instruct X4 to first input the cross-stage partial connection and full kernel network CSP-Omnikernel module, and the output of the cross-stage partial connection and full kernel network CSP-Omnikernel module is used to obtain feature X5 through the RepC3 residual block;

[0058] Step 3-2-36: Feature X5 is concatenated with feature X3 in the channel dimension after passing through a 3×3 convolutional layer to obtain feature X6. Feature X6 is then passed through a RepC3 residual block to obtain feature X7.

[0059] Step 3-2-37: After feature X7 passes through a 3×3 convolutional layer, it is concatenated with feature map F5 in the channel dimension. The concatenated output is notified to X8 through the RepC3 residual block.

[0060] Step 3-2-38, finally, notify X5, X7, and X8 to be concatenated to obtain the output feature X;

[0061] The cross-stage partial connectivity and full kernel network CSP-Omnikernel module performs the following operations:

[0062] Step 3-2-41, Cross-stage partial connection and full kernel network CSP-Omnikernel module introduces CSP structure, divides input into two branches; one branch first performs convolution and then passes through Omnikernel block, the other branch performs convolution directly;

[0063] Step 3-2-42: The two branches are concatenated and then convolved to obtain the output.

[0064] The Omnikernel block performs the following operations:

[0065] Step 3-2-51: Input feature X4 is processed through a 1×1 convolution operation to obtain the feature. feature They are fed into three branches: the local branch, the big kernel branch, and the global branch, to enhance multi-scale representation;

[0066] Step 3-2-52: Add the outputs of the three branches together, and then perform a 1×1 convolution operation to obtain the output.

[0067] In the large kernel branch, input features Three convolution operations are performed respectively: a 31×31 depth-separable convolution operation is used to obtain the receptive field, and a 31×1 and a 1×31 bar depth-separable convolution operation is used to obtain the context information of the bar.

[0068] The global branch includes a two-domain channel attention module and a frequency-based spatial attention module;

[0069] The dual-domain channel attention module consists of a frequency channel attention module and a spatial channel attention module;

[0070] Given input features The dual-channel attention module first processes the input feature X GlobalApplying frequency channel attention, expressed as:

[0071]

[0072] in, and These are the Fast Fourier Transform (FFT) and its inverse; X FCA W 1×1 GAP and GAP represent the output of the frequency channel attention module, the 1×1 convolutional layer, and the global average pooling, respectively. This represents an element-wise multiplication operation;

[0073] After Fourier transform processing, the global features, after global modulation in the spectral domain, are further input into the spatial channel attention module. The specific operation is as follows:

[0074]

[0075] Among them, X DCAM It is the output of the dual-domain channel attention module;

[0076] Subsequently, X DCAM The input is fed into a frequency-based spatial attention module, which further refines the spectrum in the spatial dimension. The specific operation is represented as follows:

[0077]

[0078] Among them, X FSAM It is the output of a frequency-based spatial attention module;

[0079] The local branch only passes the input through a 1×1 deep convolutional layer for the modulation of local features.

[0080] Step 4 includes the following steps:

[0081] Step 4-1, Set the text prompt feature F T =F p Multi-scale fusion image features F I =X, inputting the text cue features and multi-scale fused image features into the Co-Transformer, a common attention transformer for the text modality and the image modality respectively, to obtain the visual attention-weighted text features F. K Text attention weighted visual features F L The specific operation is as follows:

[0082] F K =Co-Transformer T (F T ,F I ;θ T )

[0083] F L =Co-Transformer I (F I ,F T ;θ I )

[0084] Where, θ T In the text modality Co-Transformer, θ represents the parameters of a fully connected layer with the hyperbolic tangent Tanh activation function. I The parameters of the fully connected layer in the image modality Co-Transformer are represented by the hyperbolic tangent Tanh function as the activation function; Co-Transformer represents the common attention transformer.

[0085] Step 4-2, F K and F L Connecting them together yields the cross-modal fusion feature F. Z .

[0086] Step 5 includes the following steps:

[0087] Step 5-1, fuse the cross-modal features F Z The input is fed into the head network, and a fixed number of features are selected as the initial object queries for the decoder through a minimum uncertainty query scheme;

[0088] Step 5-2: The features selected by the query party with minimum uncertainty are fed into the Transformer decoder and connected to a multilayer perceptron network as the detection head to generate bounding boxes and ship categories.

[0089] Step 5-1 includes: determining the characteristic uncertainty Defined as location reliability With classification confidence The difference between the predicted distributions is expressed as:

[0090]

[0091] in, Indicates cross-modal fusion features;

[0092] To minimize query uncertainty, this uncertainty is integrated into a gradient-based optimization loss function, and a cross-union (CUI) soft label is introduced into the class loss. That is, the CUI between the predicted bounding box and the ground truth is used as the label for the class prediction. The loss function is expressed as follows:

[0093]

[0094] in, This indicates the predicted category of the detection model for ship targets. and predicted bounding boxes This indicates the correct ship category c and the bounding box b. Represents the bounding box loss function. Represents the category loss function;

[0095] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0096] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0097] Beneficial effects: 1) This invention leverages the powerful zero-shot learning capability of multimodal large models, introduces multimodal large models to generate prompts to obtain more discriminative information contained in image modalities, and designs a cross-modal fusion network to achieve effective fusion between image and text modalities, thereby making full use of useful multimodal information.

[0098] 2) This invention improves upon the original multi-scale feature fusion network of RT-DETR by proposing a small target feature enhancement pyramid network. The P2 feature layer is added and subjected to spatial depth transformation convolution to obtain features rich in small target information, which are then fed to P3 for fusion. The idea of ​​cross-stage partial connection (CSP) and full kernel (OmniKernel) network are introduced, and the CSP-OmniKernel module is designed to perform feature fusion, so as to effectively learn feature representations from global to local, thereby improving the detection performance of small targets. Attached Figure Description

[0099] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0100] Figure 1 This is a flowchart of the steps of the method of the present invention.

[0101] Figure 2 This is a schematic diagram of the improved RT-DETR algorithm in the implementation of this invention.

[0102] Figure 3 This is a schematic diagram of the feature interaction network in the embodiment of the present invention.

[0103] Figure 4 This is a schematic diagram of the multi-scale feature fusion network in an embodiment of the present invention.

[0104] Figure 5 This is a schematic diagram of the structure of the OmniKernel, a full-kernel network module in the small target feature enhancement pyramid network of this invention.

[0105] Figure 6 This is a schematic diagram of the cross-modal fusion network in the implementation of this invention. Detailed Implementation

[0106] like Figure 1 As shown, this embodiment of the invention provides a method for ship detection in remote sensing images based on multimodal large model prompts, comprising the following steps:

[0107] S1. Obtain the remote sensing ship image dataset and divide it into training set, validation set and test set according to the proportion;

[0108] S2. Input the training set data of remote sensing ship images into the multimodal large model, and use the multimodal large model to generate text prompt features;

[0109] The specific operation method of S2 is as follows:

[0110] 1) Input the processed training set of remote sensing ship images and the prompt 'Detect Ship' from S1 into the pre-trained multimodal large model PaliGemma;

[0111] 2) First, the image is converted into a series of image tags by an image encoder, and the image tags are then projected through a linear projection layer;

[0112] 3) The prompt word text is converted into a series of text tokens using the SentencePiece tokenizer of the pre-trained language model Gemma. These tokens are then embedded as prefixes through Gemma's lexical embedding layer.

[0113] 4) Then, create the tag sequence input to the decoder in a certain way, as shown below:

[0114] Tag sequence = [image tag, BOS, prefix tag, SEP, suffix tag, EOS]

[0115] Among them, the BOS mark indicates the beginning of a text mark, the SEP mark is used as a separator, with \n as the SEP mark, and the EOS mark indicates the end of a text mark;

[0116] 5) Finally, the suffix marker sequence F is obtained by decoding. p This refers to text prompt features.

[0117] S3, such as Figure 2As shown, based on the Real-Time Detection Transformer (RT-DETR) model, the feature interaction network and multi-scale feature fusion network in the RT-DETR model are improved. Remote sensing ship image data is input into the improved RT-DETR model to obtain multi-scale fused image features.

[0118] The specific operation method of S3 is as follows:

[0119] 1) Input the processed remote sensing ship image in S1 into the backbone network ResNet50 to obtain the feature maps {P2,P3,P4,P5} after the last four layers of ResNet50 are processed. Among them, the P2 feature map is the small target detection layer. P2 has fewer convolution operations and a larger feature map size, which is more conducive to small target detection.

[0120] 2) such as Figure 3 As shown, the P5 feature map with richer semantic features is input into the feature interaction network to obtain the attention-weighted feature map F5. The feature interaction network adopts a single-layer SwiftFormer encoder.

[0121] Furthermore, the specific steps of the SwiftFormer encoder are as follows:

[0122] Step 1) Feed the P5 feature map into the first encoding block, which consists of a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer to obtain the feature map P5. 1 ;

[0123] Step 2) Transfer feature map P5 1 The feature map P5 is obtained by feeding it into the second encoding block, namely the additive attention block. 2 ;

[0124] Step 3) Transfer feature map P5 2 The third encoding block, namely the linear block, is fed into the system. The linear block consists of two 1×1 pointwise convolutional layers, batch normalization, and the GeLU activation function, resulting in the output feature map F5.

[0125] The SwiftFormer encoder is represented as follows:

[0126] P5 1 =Conv1(DWConv BN (P5)

[0127] P5 2 =QK(P5) 1 )+P5 1

[0128] F5 = Conv1(ConvBN,1,G (P5 2 ))+P5 2

[0129] Where DWConv represents a 3×3 depthwise convolutional layer, Conv1 represents a 1×1 pointwise convolutional layer, and Conv... BN,1,G represents batch normalization, 1×1 pointwise convolutional layer and GeLU activation function, and QK represents effective additive attention block.

[0130] Furthermore, the specific steps of the additive attention block are as follows:

[0131] Step 1) Input feature matrix P5 1 By querying the transformation matrix W q Bond transformation matrix W k Transform into a query matrix Q and a key matrix K, where the query matrix Q is multiplied by the learnable parameter vector w. a Using the attention weights of the learned queries, a global attention query vector α is generated, represented as follows:

[0132]

[0133] Where d is the dimension of the input feature matrix.

[0134] Step 2) Pool the query matrix according to the learned attention weights to obtain the global query vector. As shown below:

[0135]

[0136] Where n is the number of input feature matrices, α i Q is the global attention query vector of the i-th feature matrix. i It is the query matrix of the i-th feature matrix.

[0137] Step 3) Encode the global query vector q and the key matrix K element-wise to obtain the global context matrix W. c This matrix is ​​similar to the attention matrix in multi-head self-attention, used to capture global contextual information for each feature point. The output P5 of the additive attention block 2 It can be described as:

[0138]

[0139] in, Let T represent the normalized query matrix, and T represent the linear transformation.

[0140] 3) The attention-weighted feature map F5 and feature maps {P2, P3, P4} are input together into a multi-scale feature fusion network to obtain multi-scale fused image features. The multi-scale feature fusion network uses a small target feature enhancement pyramid network, such as... Figure 4 As shown (where CSP-OmniKernel is a cross-stage full kernel network, OmniKernel is a full kernel network, and RepC3 is a reparameterized convolutional module used to build efficient neural network structures).

[0141] The specific steps of the small target feature enhancement pyramid network are as follows:

[0142] Step 1) P2, P3, P4, and F5 represent feature maps at different resolutions, from high to low.

[0143] First, the P2 feature map is transformed into X1 through a spatial depth transformation convolution operation. This operation enhances the representation of spatial information by transforming the dimension of the feature map.

[0144] Step 2) The F5 feature map is upsampled to bring its resolution up to the same level as P4.

[0145] Step 3) The P4 feature map is concatenated with the upsampled output of F5 after passing through a 3×3 convolutional layer to obtain X2;

[0146] Step 4) After X2 is processed by the RepC3 residual block, it is passed through a 3×3 convolutional layer to obtain X3. The output of X3 after upsampling is concatenated with X1 and P3 to obtain X4.

[0147] Step 5) X4 is first input into the CSP-Omnikernel module, which is used to extract more complex spatial and contextual features, and then the output is passed through the RepC3 residual block to obtain X5;

[0148] Step 6) After X5 passes through a 3×3 convolutional layer, it is concatenated with X3 in the channel dimension to obtain X6. X6 is then passed through a RepC3 residual block to obtain X7.

[0149] Step 7) After X7 passes through a 3×3 convolutional layer, it is concatenated with F5 in the channel dimension, and the output is passed through the RepC3 residual block to obtain X8;

[0150] Step 8) Finally, concatenate X5, X7, and X8 to obtain the output feature X.

[0151] Furthermore, the specific steps of the CSP-Omnikernel module are as follows:

[0152] Step 1) The CSP-Omnikernel module introduces the CSP structure, dividing the input into two branches;

[0153] Step 2) One branch performs convolution first and then passes through an Omnikernel block, while the other branch performs convolution directly;

[0154] Step 3) The last two branches are concatenated, and then convolution is performed to obtain the output;

[0155] Furthermore, such as Figure 5 As shown, the specific steps of the Omnikernel block are as follows:

[0156] Step 1) Input feature X4 is obtained after a 1×1 convolution operation. They are fed into three branches: the local branch, the big kernel branch, and the global branch, to enhance multi-scale representation;

[0157] Step 2) Then, the outputs of the three branches are added together, and then a 1×1 convolution operation is performed to obtain the output.

[0158] Furthermore, in the three branches:

[0159] a) In the large kernel branch, input Three convolution operations are performed: a 31×31 depth-separable convolution operation to obtain a larger receptive field, and 31×1 and 1×31 bar depth-separable convolution operations to obtain the contextual information of the bars.

[0160] b) The global branch includes a dual-domain channel attention module (DCAM) and a frequency-based spatial attention module (FSAM), which enhances the global modeling capability through dual-domain processing;

[0161] Given input features The dual-domain channel attention module first focuses on X Global The specific operation of applying Frequency Channel Attention (FCA) is as follows:

[0162]

[0163] in, and These are the Fast Fourier Transform and its inverse transform; X FCA W 1×1 GAP and GAP represent the output of the frequency channel attention module, the 1×1 convolutional layer, and the global average pooling, respectively. This represents an element-wise multiplication operation;

[0164] After Fourier transform processing, the global features, after global modulation in the spectral domain, are further input into the Spatial Channel Attention (SCA) module, and its specific operation is represented as follows:

[0165]

[0166] Among them, X DCAM It is the output of the Dual-Domain Channel Attention Module (DCAM), which enhances dual-domain features at the channel level with coarse granularity.

[0167] Subsequently, X DCAM The input is fed into a frequency-based spatial attention module, which further refines the spectrum in the spatial dimension. The specific operation is represented as follows:

[0168]

[0169] Among them, X FSAM It is the output of the frequency-based spatial attention module (FSAM).

[0170] c) Local branches pass the input through only a 1×1 deep convolutional layer for modulation of local features.

[0171] S4. Design a cross-modal fusion network to fuse multi-scale fused image features with text prompt features obtained in S2 to obtain cross-modal features;

[0172] The specific operation method of S4 is as follows:

[0173] 1) such as Figure 6 As shown, assuming the text prompt feature F T =F p Multi-scale fusion image features F I =X, and input them into the common attention Transformer (Co-Transformer) of the text modality and the image modality respectively to obtain the output visual attention-weighted text features F. K Text attention weighted visual features F L The specific operation is as follows:

[0174] F K =Co-Transformer T (F T ,F I ;θ T )

[0175] F L =Co-Transformer I (F I ,F T ;θ I )

[0176] Where, θ T In the text modality Co-Transformer, θ represents the parameters of a fully connected layer with the hyperbolic tangent Tanh activation function. IThe Co-Transformer represents the parameters of a fully connected layer in the image modality Co-Transformer, where the activation function is the hyperbolic tangent Tanh function; Co-Transformer represents the common attention transformer to obtain F of the same dimension. I and F T ;

[0177] 2) Finally, F K and F L Connecting them together yields the cross-modal fusion feature F. Z .

[0178] S5. Input the cross-modal fusion features obtained in S4 into the Transformer decoder and the detection head network. The features decoded by the Transformer decoder are then optimized by the detection head to generate more accurate bounding boxes and ship categories.

[0179] The specific operation method of S5 is as follows:

[0180] 1) Input cross-modal fusion features into the head network, and select a fixed number of features as the initial object query for the decoder through a minimum uncertainty query scheme;

[0181] Characteristic uncertainty Defined as location reliability With classification confidence The difference between the predicted distributions is expressed as follows:

[0182]

[0183] in, This indicates cross-modal fusion features.

[0184] To minimize query uncertainty, this uncertainty is integrated into a gradient-based optimization loss function, and a cross-union soft label (CUI) is introduced into the class loss. This means the CUI between the predicted bounding box and the ground truth is used as the label for the class prediction. The loss function is expressed as follows:

[0185]

[0186] in, This indicates the predicted category of the detection model for ship targets. and predicted bounding boxes This indicates the correct ship category c and the bounding box b. Represents the bounding box loss function. 1) Describe the category loss function; 2) Feed the features selected by the minimum uncertainty query into the Transformer decoder and connect several multilayer perceptron networks as detection heads to generate accurate bounding boxes and ship categories.

[0187] S6. Based on the training set and validation set obtained in S1, the improved RT-DETR model is trained to obtain the trained Real-Time Remote Sensing Images Ship Detection Transformer (RT-RSDETR).

[0188] S7. Finally, the test set image data obtained in S1 is input into the RT-RSDETR model obtained in S6 to obtain the position information and category of the target ship.

[0189] In summary, this invention proposes a remote sensing image ship detection method based on multimodal large model cue generation. It improves upon the RT-DETR model by introducing multimodal large model cue generation and designing a cross-modal fusion network. SwiftFormer is selected as the encoder, and an efficient additive attention block is introduced to enhance the model's feature extraction capability and detection speed. A small target feature enhancement pyramid network is proposed to improve the detection performance of small targets. Ultimately, this method improves both the detection accuracy and speed of remote sensing image ship detection.

[0190] This embodiment was tested on HRSC2016. First, the HRSC2016 remote sensing ship image dataset was acquired and divided into training and test sets. The training set data was input into a multimodal large-scale model to generate text prompt features, and simultaneously input into an improved RT-DETR model to obtain multi-scale fused image features. A cross-modal fusion network was designed to fuse these two features, thereby generating bounding boxes and ship categories. After training, a real-time detection model was obtained. Inputting the test set allowed for the acquisition of target ship location and category information. Experimental results show that this method performs excellently on the HRSC2016 dataset, achieving an accuracy of 0.983 and a mean AP50 of 0.971, outperforming mainstream models. The reason for the good results of this method is that, on the one hand, it achieves effective fusion of image and text modalities, giving full play to the complementary advantages of multimodal information, enabling the model to acquire more comprehensive and richer ship feature information; on the other hand, it improves the RT-DETR model, combines it with a cross-modal fusion network, optimizes the feature extraction and processing process, and improves the model's detection accuracy and efficiency of ship targets in remote sensing images, thus achieving good performance in various indicators.

[0191] This invention provides a method for ship detection in remote sensing images based on multimodal large model prompting. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for ship detection in remote sensing images based on multimodal large model prompting, characterized in that, Includes the following steps: Step 1: Obtain the remote sensing ship image dataset and divide it into training set, validation set and test set according to the proportions; Step 2: Input the training set data into the multimodal large model and use the multimodal large model to generate text prompt features; Step 3: Using the real-time detection Transformer as the base model, improve the feature interaction network and multi-scale feature fusion network in the real-time detection transformer RT-DETR model, and input the training set data into the improved RT-DETR model to obtain multi-scale fused image features. Step 4: Design a cross-modal fusion network to fuse the multi-scale fused image features with the text prompt features obtained in Step 2 to obtain cross-modal fusion features; Step 5: Input the cross-modal fusion features obtained in Step 4 into the Transformer decoder and the detection head network. The features decoded by the Transformer decoder are then optimized by the detection head to generate bounding boxes and ship categories. Step 6: Train the improved RT-DETR model based on the training set and validation set to obtain the trained real-time remote sensing image ship detection model. Step 7: Input the test set into the ship detection model of the real-time remote sensing image obtained in Step 6 to obtain the position information and category of the target ship.

2. The method according to claim 1, characterized in that, In step 1, the acquired remote sensing ship images are preprocessed by scaling the original images to a standard size of 640×640 pixels and labeling the ship targets in the remote sensing ship image data to obtain the location category labels of the ship images.

3. The method according to claim 2, characterized in that, Step 2 includes the following steps: Step 2-1: Input the processed remote sensing ship image and the prompt word "Detect Ship" from Step 1 into the pre-trained multimodal large model PaliGemma; Step 2-2: First, the image is converted into a series of image tags by an image encoder. The image tags are then projected through a linear projection layer. Steps 2-3: The SentencePiece segmenter of the pre-trained language model Gemma is used to convert the prompt word Detect Ship into a series of text tags. The text tags are then embedded as prefixes through the lexical embedding layer of the pre-trained language model Gemma. Steps 2-4: Create the tag sequence input to the decoder, represented as: Tag sequence = [image tag, BOS, prefix tag, SEP, suffix tag, EOS], Among them, the BOS mark indicates the beginning of a text mark, the SEP mark is used as a separator, with \n as the SEP mark, and the EOS mark indicates the end of a text mark; Steps 2-5: Decode the suffix marker sequence F using the decoder. p F p That is, text prompt features.

4. The method according to claim 3, characterized in that, In step 3, RT-DETR is used as the base model; ResNet50 is selected as the backbone network of the RT-DETR model. We use the SwiftFormer as the encoder and introduce an additive attention block to replace the quadratic matrix multiplication operation of the traditional self-attention mechanism in the RT-DETR model. Improve the multi-scale feature fusion network and establish a small target feature enhancement pyramid network; A small target detection layer is introduced, and features rich in small target information are obtained through the spatial depth transformation convolutional network SPDConv and fed into the feature map P3 for fusion. By introducing the concept of cross-stage partial connection (CSP) and the full kernel OmniKernel network, a CSP-OmniKernel module is designed for feature fusion. The Cross-Stage Partial Connection (CSP) concept involves splitting the network's feature map into two parts. One part is fed into the network for feature extraction, while the other part is directly concatenated to the output of the first part. The full-kernel OmniKernel network consists of three branches: the global branch, the big kernel branch, and the local branch.

5. The method according to claim 4, characterized in that, Step 3 includes the following steps: Step 3-1: Input the remote sensing ship image processed in Step 1 into the backbone network ResNet50 to obtain the feature maps {P2,P3,P4,P5} after processing by the last four layers of ResNet50. Step 3-2: Input feature map P5 into the feature interaction network to obtain attention-weighted feature map F5. The feature interaction network uses a single-layer SwiftFormer encoder. The SwiftFormer encoder performs the following operations: Step 3-2-11: Feed the feature map P5 into the first encoding block. The first encoding block includes a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer to obtain the feature map. Step 3-2-12, transfer the feature map The feature map P5 is obtained by feeding it into the second encoding block, namely the additive attention block. 2 ; Step 3-2-13, transfer the feature map The third encoding block, namely the linear block, is fed into the system. The linear block consists of two 1×1 pointwise convolutional layers, batch normalization, and GeLU activation function to obtain the output feature map F5. The SwiftFormer encoder is represented as: F5=Conv1(Conv BN,1,G (P5 2 ))+P5 2 Among them, depthwise separable convolution DWConv BN Conv1 represents a 3×3 depthwise convolutional layer, and Conv2 represents a 1×1 pointwise convolutional layer. BN,1,G This represents batch normalization, a 1×1 pointwise convolutional layer, and the GeLU activation function; QK represents an effective additive attention block. The additive attention block performs the following operations: Step 3-2-21, input feature map By querying the transformation matrix W q Bond transformation matrix W k Transform into a query matrix Q and a key matrix K, where the query matrix Q is multiplied by the learnable parameter vector w. a The global attention query vector α is generated by learning the attention weights of the query, and is represented as follows: Where d is the dimension of the input feature matrix; Step 3-2-22: Pool the query matrix according to the learned attention weights to obtain the global query vector. As shown below: in Let n represent the real number space, where n is the number of input feature matrices, and α is the number of input feature matrices. i Q is the global attention query vector of the i-th feature matrix. i It is the query matrix of the i-th feature matrix; Step 3-2-23: Perform element-wise product encoding of the global query vector q and the key matrix K to obtain the global context matrix W. c The output of the additive attention block, P5 2 Described as: in, Let T represent the normalized query matrix, and T represent the linear transformation. Step 3-2-24: Input the attention-weighted feature map F5 and feature maps {P2,P3,P4} together into the multi-scale feature fusion network to obtain multi-scale fused image features. The multi-scale feature fusion network adopts a small target feature enhancement pyramid network. The small target feature enhancement pyramid network performs the following operations: Step 3-2-31, P2, P3, P4, F5 represent feature maps at different resolutions, from high to low; First, feature map P2 is transformed into feature X1 through spatial depth transformation convolution operation; Step 3-2-32: Feature map F5 is upsampled to achieve the same resolution as P4. Step 3-2-33: Feature map P4 is concatenated with the upsampled output of F5 after passing through a 3×3 convolutional layer to obtain feature X2; Step 3-2-34: After feature X2 is processed by RepC3 residual block, it is passed through a 3×3 convolutional layer to obtain feature X3. The output of feature X3 after upsampling is concatenated with X1 and P3 to obtain feature X4. Step 3-2-35: Instruct X4 to first input the cross-stage partial connection and full kernel network CSP-Omnikernel module, and the output of the cross-stage partial connection and full kernel network CSP-Omnikernel module is used to obtain feature X5 through the RepC3 residual block; Step 3-2-36: Feature X5 is concatenated with feature X3 in the channel dimension after passing through a 3×3 convolutional layer to obtain feature X6. Feature X6 is then passed through a RepC3 residual block to obtain feature X7. Step 3-2-37: After feature X7 passes through a 3×3 convolutional layer, it is concatenated with feature map F5 in the channel dimension. The concatenated output is notified to X8 through the RepC3 residual block. Step 3-2-38, finally, notify X5, X7, and X8 to be concatenated to obtain the output feature X; The cross-stage partial connectivity and full kernel network CSP-Omnikernel module performs the following operations: Step 3-2-41, Cross-stage partial connection and full kernel network CSP-Omnikernel module introduces CSP structure, divides input into two branches; one branch first performs convolution and then passes through Omnikernel block, the other branch performs convolution directly; Step 3-2-42: The two branches are concatenated and then convolved to obtain the output. The Omnikernel block performs the following operations: Step 3-2-51: Input feature X4 is processed through a 1×1 convolution operation to obtain the feature. feature They are fed into three branches: the local branch, the big kernel branch, and the global branch, to enhance multi-scale representation; Step 3-2-52: Add the outputs of the three branches together, and then perform a 1×1 convolution operation to obtain the output. In the large kernel branch, input features Three convolution operations are performed respectively: a 31×31 depth-separable convolution operation is used to obtain the receptive field, and a 31×1 and a 1×31 bar depth-separable convolution operation is used to obtain the context information of the bar. The global branch includes a two-domain channel attention module and a frequency-based spatial attention module; The dual-domain channel attention module consists of a frequency channel attention module and a spatial channel attention module; Given input features The dual-channel attention module first processes the input feature X Global Applying frequency channel attention, expressed as: in, and These are the Fast Fourier Transform (FFT) and its inverse; X FCA W 1×1 GAP and GAP represent the output of the frequency channel attention module, the 1×1 convolutional layer, and the global average pooling, respectively. This represents an element-wise multiplication operation; After Fourier transform processing, the global features, after global modulation in the spectral domain, are further input into the spatial channel attention module. The specific operation is as follows: Among them, X DCAM It is the output of the dual-domain channel attention module; Subsequently, X DCAM The input is fed into a frequency-based spatial attention module, which further refines the spectrum in the spatial dimension. The specific operation is represented as follows: Among them, X FSAM It is the output of a frequency-based spatial attention module; The local branch only passes the input through a 1×1 deep convolutional layer for the modulation of local features.

6. The method according to claim 5, characterized in that, Step 4 includes the following steps: Step 4-1, set the text prompt feature F T =F p Multi-scale fusion image features F I =X, inputting the text cue features and multi-scale fused image features into the Co-Transformer, a common attention transformer for the text modality and the image modality respectively, to obtain the visual attention-weighted text features F. K Text attention weighted visual features F L The specific operation is as follows: F K =Co-Transformer T (F T ,F I ;θ T ) F L =Co-Transformer I (F I ,F T ; i I ) Where, θ T In the text modality Co-Transformer, θ represents the parameters of a fully connected layer with the hyperbolic tangent Tanh activation function. I The parameters of the fully connected layer in the image modality Co-Transformer are represented by the hyperbolic tangent Tanh function as the activation function; Co-Transformer represents the common attention transformer. Step 4-2, F K and F L Connecting them together yields the cross-modal fusion feature F. Z .

7. The method according to claim 6, characterized in that, Step 5 includes the following steps: Step 5-1, fuse the cross-modal features F Z The input is fed into the head network, and a fixed number of features are selected as the initial object queries for the decoder through a minimum uncertainty query scheme; Step 5-2: The features selected by the query party with minimum uncertainty are fed into the Transformer decoder and connected to a multilayer perceptron network as the detection head to generate bounding boxes and ship categories.

8. The method according to claim 7, characterized in that, Step 5-1 includes: determining the characteristic uncertainty Defined as location reliability With classification confidence The difference between the predicted distributions is expressed as: in, Indicates cross-modal fusion features; To minimize query uncertainty, this uncertainty is integrated into a gradient-based optimization loss function, and a cross-union (CUI) soft label is introduced into the class loss. That is, the CUI between the predicted bounding box and the ground truth is used as the label for the class prediction. The loss function is expressed as follows: in, This indicates the predicted category of the detection model for ship targets. and predicted bounding boxes This indicates the correct ship category c and the bounding box b. Represents the bounding box loss function. This represents the category loss function.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 8.

Images