Target detection method and device for multi-modal remote sensing image

Abstract

The application relates to the technical field of remote sensing image processing, in particular to a multi-modal remote sensing image target detection method and device, wherein the method comprises the following steps: based on a first modal remote sensing image and a second modal remote sensing image of a detection target, acquiring corresponding multi-scale extraction features; based on the multi-scale extraction features, acquiring scale extraction features of the two modal remote sensing images respectively; inputting the first scale extraction features and the second scale extraction features into a pre-constructed Mamba interaction module respectively, so as to acquire high-dimensional interaction features of the two modal remote sensing images; inputting the multi-scale extraction features and the high-dimensional interaction features into a pre-constructed offset guide fusion module respectively, so as to acquire multi-dimensional fusion features between the two modal remote sensing images, and then generating a final target detection result. Thus, the problem that, in the related art, due to the difference between the object position and the features, the precision of the detection model is not good and the overall performance is poor and the like is solved.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for target detection in multimodal remote sensing images. Background Art

[0002] Object detection is a task that rapidly interprets images and determines the location of objects. It is a key task in computer vision and has been applied in a variety of fields, including autonomous driving, remote sensing, and medical imaging. However, single-modal object detection is limited in its ability to capture salient features of objects in complex scenarios, such as lighting, weather conditions, and occlusions, resulting in reduced accuracy. Multimodal visual data, comprised of images acquired by different sensors, can provide richer information about object attributes. Recent advances in multimodal fusion technology have significantly improved object detection performance.

[0003] Related technologies can effectively combine data from different modalities, including RGB images, infrared images, lidar, and radar, through methods such as feature-level fusion, decision-level fusion, and pixel-level fusion. These methods can leverage the complementarity of multimodal data to improve the ability to detect objects even in low-light conditions, obstructions, or inclement weather.

[0004] However, in related technologies, due to the data offset of different sensors, there will be differences between the object position and features. If feature association and fusion are performed directly, it will make it difficult for the detection model to accurately align the information in each modality, thereby affecting the detection accuracy and reducing the overall performance of the model. Improvement is urgently needed. Summary of the Invention

[0005] The present application provides a method and device for target detection in multimodal remote sensing images to address the problems in related technologies, such as data offset between different sensors, which leads to differences between object positions and features, making it difficult for the detection model to accurately align information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0006] The first aspect embodiment of the present application provides a target detection method of multi-modal remote sensing images, comprising the following steps: based on a first modal remote sensing image and a second modal remote sensing image of a detection target, obtaining a first multi-scale extracted feature of the first modal remote sensing image and a second multi-scale extracted feature of the second modal remote sensing image; based on the first multi-scale extracted feature and the second multi-scale extracted feature, respectively obtaining a first scale extracted feature of the first multi-scale extracted feature and a second scale extracted feature of the second multi-scale extracted feature; inputting the first scale extracted feature and the second scale extracted feature into a pre-constructed Mamba interaction module respectively, to obtain a first high-dimensional interaction feature of the first modal remote sensing image and a second high-dimensional interaction feature of the second modal remote sensing image; inputting the first multi-scale extracted feature, the second multi-scale extracted feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature into a pre-constructed offset guided fusion module respectively, to obtain a multi-dimensional fusion feature of the first modal remote sensing image and the second modal remote sensing image, and generate a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion feature.

[0007] Optionally, in an embodiment of the present application, based on the first modal remote sensing image and the second modal remote sensing image of the detection target, the first multi-scale extracted feature of the first modal remote sensing image and the second multi-scale extracted feature of the second modal remote sensing image are obtained, comprising: inputting the first modal remote sensing image and the second modal remote sensing image into a target convolutional neural network to extract the first multi-scale extracted feature and the second multi-scale extracted feature.

[0008] Optionally, in one embodiment of the present application, the first-scale extraction features and the second-scale extraction features are respectively input into a pre-constructed Mamba interaction module to obtain the first high-dimensional interaction features of the first modality remote sensing image and the second high-dimensional interaction features of the second modality remote sensing image, including: performing a global scan on the first-scale extraction features and the second-scale extraction features respectively to obtain the first modality discrete data of the first-scale extraction features and the second modality discrete data of the second-scale extraction features; inputting the first modality discrete data and the second modality discrete data into a pre-constructed single Mamba module to obtain the first modality discrete extraction features of the first modality discrete data and the second modality discrete extraction features of the second modality discrete data; combining the first modality discrete extraction features, the second modality discrete extraction features, the local scan and the pre-constructed cross-Mamba module to obtain the first modality discrete significant features of the first modality discrete extraction features and the second modality discrete significant features of the second modality discrete extraction features; and reconstructing the first modality discrete significant features and the second modality discrete significant features to obtain the first high-dimensional interaction features and the second high-dimensional interaction features.

[0009] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be, but is not limited to, as follows:

[0010]

[0011] in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0012] Optionally, in one embodiment of the present application, the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature are respectively input into a pre-built offset-guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, including: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain the multi-dimensional fusion feature.

[0013] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be, but is not limited to, as follows:

[0014]

[0015] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0016] According to a second aspect of the present application, a target detection device for a multimodal remote sensing image is provided, comprising: an extraction module for obtaining, based on a first modal remote sensing image and a second modal remote sensing image of a detection target, a first multiscale extraction feature of the first modal remote sensing image and a second multiscale extraction feature of the second modal remote sensing image; an acquisition module for obtaining, based on the first multiscale extraction feature and the second multiscale extraction feature, a first scale extraction feature of the first multiscale extraction feature and a second scale extraction feature of the second multiscale extraction feature; an input module for inputting the first scale extraction feature and the second scale extraction feature into a pre-built Mamba interaction module, respectively, to obtain a first high-dimensional interaction feature of the first modal remote sensing image and a second high-dimensional interaction feature of the second modal remote sensing image; and a generation module for inputting the first multiscale extraction feature, the second multiscale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature into a pre-built offset-guided fusion module, respectively, to obtain multidimensional fusion features of the first modal remote sensing image and the second modal remote sensing image, and generating a first target detection result and a second target detection result of the detection target based on the multidimensional fusion features.

[0017] Optionally, in one embodiment of the present application, the extraction module includes: an extraction unit, used to input the first modality remote sensing image and the second modality remote sensing image into a target convolutional neural network to extract the first multi-scale extraction features and the second multi-scale extraction features.

[0018] Optionally, in one embodiment of the present application, the input module includes: a scanning unit for performing a global scan on the first-scale extraction feature and the second-scale extraction feature respectively to obtain the first modal discrete data of the first-scale extraction feature and the second modal discrete data of the second-scale extraction feature; an input unit for inputting the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain the first modal discrete extraction feature of the first modal discrete data and the second modal discrete extraction feature of the second modal discrete data; an acquisition unit for combining the first modal discrete extraction feature, the second modal discrete extraction feature, the local scan and the pre-built cross-Mamba module to obtain the first modal discrete significant feature of the first modal discrete extraction feature and the second modal discrete significant feature of the second modal discrete extraction feature; a reconstruction unit for reconstructing the first modal discrete significant feature and the second modal discrete significant feature to obtain the first high-dimensional interaction feature and the second high-dimensional interaction feature.

[0019] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be, but is not limited to, as follows:

[0020]

[0021] in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0022] Optionally, in one embodiment of the present application, the generation module includes: a fusion unit, used to use a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain the multi-dimensional fusion feature.

[0023] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be, but is not limited to, as follows:

[0024]

[0025] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0026] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method for multimodal remote sensing images as described in the above embodiment.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned target detection method for multimodal remote sensing images.

[0028] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method for multimodal remote sensing images.

[0029] The embodiment of the present application can perform feature extraction on multimodal remote sensing images to obtain multi-scale extraction features corresponding to remote sensing images of different modalities, and perform feature interaction based on the multi-scale extraction features to realize the construction of complementary data bodies, and obtain high-dimensional interactive features of multimodal remote sensing images by constructing a Mamba interaction module, and obtain multi-dimensional fusion features of multimodal remote sensing images by constructing an offset-guided fusion module, thereby generating the final target detection result of the detection target, fully considering the offset problem between multimodal remote sensing images, and using the constructed Mamba interaction module to realize the calculation of multimodal remote sensing images, achieving the effect of complementary information bodies between modalities, thereby improving the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to reduce the offset effect between multimodal remote sensing images, ensuring that the rich information of multimodal remote sensing images can be maximized during the fusion process, not only improving the accuracy of multimodal target detection, but also significantly enhancing the robustness of the model. Thus, it solves the problem in the related art that due to the data offset of different sensors, there will be differences between object positions and features, which makes it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0030] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 This is a flowchart of a method for detecting targets in multimodal remote sensing images according to an embodiment of the present application;

[0033] Figure 2A block diagram illustrating the working principle of a method for detecting an object in a multimodal remote sensing image according to an embodiment of the present application;

[0034] Figure 3 Schematic diagram of a block diagram of a target detection device for multimodal remote sensing images according to an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the target detection method and device for multimodal remote sensing images of the embodiment of the present application with reference to the accompanying drawings. In response to the problem mentioned in the above background technology that due to the data offset of different sensors, there will be differences between the position and features of the object, which makes it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model, the present application provides a target detection method for multimodal remote sensing images. In this method, feature extraction can be performed on the multimodal remote sensing image to obtain multi-scale extracted features corresponding to remote sensing images of different modalities, and feature interaction can be performed based on the multi-scale extracted features to realize the construction of complementary data bodies, and high-dimensional interactive features of the multimodal remote sensing image can be obtained by constructing a Mamba interaction module. Construct an offset-guided fusion module to obtain the multi-dimensional fusion features of the multimodal remote sensing image, and then generate the final target detection result of the detection target, fully consider the offset problem between the multimodal remote sensing images, and use the constructed Mamba interaction module to realize the calculation of the multimodal remote sensing image, so as to achieve the effect of complementary information body between modalities, thereby improving the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to reduce the offset effect between multimodal remote sensing images, ensuring that the rich information of the multimodal remote sensing image can be maximized during the fusion process, which not only improves the accuracy of multimodal target detection, but also significantly enhances the robustness of the model. Thus, the problem in the related art that due to the data offset of different sensors, there will be differences between the object position and features, which makes it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model is solved.

[0038] Specifically, Figure 1 The present invention provides a flowchart of a method for detecting targets in multimodal remote sensing images according to an embodiment of the present application.

[0039] likeFigure 1 As shown, the target detection method of the multi-modal remote sensing image includes the following steps:

[0040] In step S101, based on the first modal remote sensing image and the second modal remote sensing image of the detected target, the first multi-scale extraction feature of the first modal remote sensing image and the second multi-scale extraction feature of the second modal remote sensing image are obtained.

[0041] It can be understood that the first modal remote sensing image of the embodiment of the present application can be a visible light remote sensing image, which can be represented as D rgb , and the second modal remote sensing image can be an infrared remote sensing image, which can be represented as D ir , wherein D rgb and D ir have the same data label, and then the two modal remote sensing images are read into the operation device.

[0042] In some embodiments, the embodiment of the present application can obtain the first multi-scale extraction feature of the visible light remote sensing image based on the visible light remote sensing image, and obtain the second multi-scale extraction feature of the infrared remote sensing image using the infrared remote sensing image.

[0043] For example, the embodiment of the present application can be written in PYTHON language, implemented by using a classic deep learning framework PYTORCH, and the Dataset function and the DataLoader function in PYTORCH are used as data reading methods to read the multi-modal remote sensing image dataset respectively. The PYTORCH data reading and loading function is a well-known technology in the technical field, and will not be described here.

[0044] Optionally, in an embodiment of the present application, based on the first modal remote sensing image and the second modal remote sensing image of the detected target, the first multi-scale extraction feature of the first modal remote sensing image and the second multi-scale extraction feature of the second modal remote sensing image are obtained, including: inputting the first modal remote sensing image and the second modal remote sensing image into a target convolutional neural network to extract the first multi-scale extraction feature and the second multi-scale extraction feature.

[0045] It should be noted that the embodiment of the present application can use the target convolutional neural network to extract the multi-scale extraction feature of the multi-modal remote sensing image. In the embodiment of the present application, the target convolutional neural network can be but not limited to CSPDarknet53, which can be set by a person skilled in the art according to the actual situation, and the present application does not make specific limitation.

[0046] As a possible implementation manner, the embodiment of the present application can construct a target convolutional neural network, and use the target convolutional neural network to extract the multi-scale extraction feature of the two modal remote sensing images respectively.

[0047] Exemplarily, an embodiment of the present application constructs two CSPDarknet53s with the same structure to extract deep features of visible light remote sensing images and infrared remote sensing images respectively. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained as the first multi-scale extraction features of the visible light remote sensing image and the second multi-scale extraction features of the infrared remote sensing image, respectively.

[0048] For example, an embodiment of the present application constructs two CSPDarknet53s with the same structure based on PYTHON and PYTORCH to extract deep features of visible light remote sensing images and infrared remote sensing images respectively. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained.

[0049] In step S102 , based on the first multi-scale extracted features and the second multi-scale extracted features, first scale extracted features of the first multi-scale extracted features and second scale extracted features of the second multi-scale extracted features are respectively obtained.

[0050] In some embodiments, the embodiment of the present application may select the feature least affected by the offset in the first multi-scale extracted features as the fusion feature to obtain the first scale extracted feature, thereby achieving the effect of reducing the impact of the offset.

[0051] In some embodiments, the embodiment of the present application may select the feature least affected by the offset in the second multi-scale extracted features as the fusion feature to obtain the second scale extracted feature, thereby achieving the effect of reducing the impact of the offset.

[0052] It is understandable that in the embodiment of the present application, due to the existence of a certain offset phenomenon in the multimodal data, if it is not taken into account, the accuracy of feature extraction and multimodal detection will be reduced, and the impact of the offset is different at different scales, which can be specifically expressed as but not limited to:

[0053] A intersection =|w-Δx|×|h-Δy|,

[0054] Among them, Δx and Δy are offsets, and both values ​​are fixed for multimodal image pairs; w and h are the width and height of the image patch, with different degrees of downsampling.

[0055] Furthermore, in the embodiment of the present application, since the offset is fixed, larger w and h are required to obtain a larger intersection area A. textintersection At the same time, using only high-level features for multimodal interaction can greatly reduce the amount of computation and improve the real-time performance of the network.

[0056] In step S103, the first scale extracted features and the second scale extracted features are respectively input into a pre-built Mamba interaction module to obtain first high-dimensional interaction features of the first modality remote sensing image and second high-dimensional interaction features of the second modality remote sensing image.

[0057] It can be understood that the Mamba interaction module constructed in the embodiment of the present application may include, but is not limited to, a single Mamba module that only processes a single modality remote sensing image and a cross-Mamba module that processes multimodal remote sensing images. The specific setting can be made by technicians in this field according to actual conditions, and this application does not impose any specific restrictions.

[0058] As a possible implementation method, the embodiment of the present application can input the first-scale extraction features and the second-scale extraction features into a pre-built Mamba interaction module respectively, thereby obtaining the first high-dimensional interaction features of the first modality remote sensing image and the second high-dimensional interaction features of the second modality remote sensing image.

[0059] For example, the embodiment of the present application can input the obtained first-scale extraction features of the visible light remote sensing image and the second-scale extraction features of the infrared remote sensing image into the pre-built Mamba interaction module, thereby obtaining the first high-dimensional interaction features of the visible light remote sensing image and the second high-dimensional interaction features of the infrared remote sensing image.

[0060] Optionally, in one embodiment of the present application, the first scale extraction feature and the second scale extraction feature are respectively input into a pre-built Mamba interaction module to obtain the first high-dimensional interaction feature of the first modality remote sensing image and the second high-dimensional interaction feature of the second modality remote sensing image, including: performing a global scan on the first scale extraction feature and the second scale extraction feature respectively to obtain the first modality discrete data of the first scale extraction feature and the second modality discrete data of the second scale extraction feature; inputting the first modality discrete data and the second modality discrete data into a pre-built single Mamba module to obtain the first modality discrete extraction feature of the first modality discrete data and the second modality discrete extraction feature of the second modality discrete data; combining the first modality discrete extraction feature, the second modality discrete extraction feature, the local scan and the pre-built cross-Mamba module to obtain the first modality discrete significant feature of the first modality discrete extraction feature and the second modality discrete significant feature of the second modality discrete extraction feature; reconstructing the first modality discrete significant feature and the second modality discrete significant feature to obtain the first high-dimensional interaction feature and the second high-dimensional interaction feature. Wherein, the calculation process of the cross-Mamba module can be, but is not limited to:

[0061]

[0062] in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0063] In some embodiments, the embodiments of the present application can perform global scans on the first-scale extracted features and the second-scale extracted features respectively to discretely serialize the feature data, and input these two modal discrete data into a pre-built single Mamba module to achieve the effect of further feature extraction and obtain the corresponding modal discrete extraction features.

[0064] Specifically, the embodiment of the present application is for the input feature S in , the feature matrix F can be constructed using adaptive maximum pooling and mean pooling in ∈R H×W×C , to ensure consistent dimensions across image sizes:

[0065]

[0066] Furthermore, the embodiments of the present application are directed to F in A deep feature map is performed and a Drop layer is added to make the module adaptive:

[0067]

[0068] Among them, h is the channel of hidden features in the mapping process, It is a linear mapping operation, and Drop means randomly dropping neurons with a certain probability.

[0069] Then, the embodiment of the present application flattens the obtained matrix into a tag sequence, that is, Simulate the sequential data of the state space model algorithm. In order to reduce the loss of two-dimensional spatial information, the embodiment of the present application also adds a learnable position embedding It provides a clear position encoding. In addition, the embodiment of the present application establishes a parallel data stream for operation, maintaining the integrity of the original feature extraction. Further, the embodiment of the present application I in Scanning can be performed in four directions, thereby expanding the serialization method with position encoding, thereby expanding the data distribution. In this embodiment of the present application, the scanning results in each direction will be subjected to S6 for sequence feature state space model feature extraction to obtain four output results, which can be represented by yi respectively, which can be represented by but not limited to:

[0070] x i=crossscan i (I in ),

[0071] y i =S6 i (x i ),

[0072]

[0073] Among them, i=1,2,3,4. represents the four global scanning directions.

[0074] For example, in the embodiment of the present application, features can be extracted at two modal scales for global scanning, and feature data can be discretely serialized. The two modal discrete data can be input into a pre-built single Mamba module to achieve further feature extraction and obtain corresponding modal discrete extraction features, which may include:

[0075] In this embodiment of the application, for the input feature S in First, the feature matrix F is constructed using adaptive maximum pooling and mean pooling. in ∈R H×W×C , to ensure that the dimensions of different image sizes remain consistent; then, in A deep feature map is performed and a Drop layer is added to make the model adaptive; then, the resulting matrix is ​​flattened into a tag sequence, i.e. Simulates sequential data of the state space model algorithm. In order to reduce the loss of two-dimensional spatial information, a learnable position embedding is added. It provides an explicit positional encoding and also establishes a parallel data flow for operations, preserving the integrity of the original feature extraction. in The scan will be performed in four directions, extending the serialization method with position encoding to expand the data distribution. Finally, the scan results in each direction will be processed by S6 to extract the sequence feature state space model features, and four output results will be obtained, which can be used to represent the y i express.

[0076] In some embodiments, the embodiments of the present application can fuse the first modal discrete extraction features and the second modal discrete extraction features, specifically by globally and locally scanning and serializing the two modal discrete extraction features, and then simultaneously inputting the two modal discrete extraction features into a pre-built cross-Mamba module to achieve the effect of deep interaction between modalities, thereby fusing the significant features of the two modalities, increasing the amount of information, and obtaining discrete significant features corresponding to the two modal discrete extraction features.

[0077] Specifically, the calculation process of the cross Mamba module in the embodiments of the present application can be, but is not limited to, the following:

[0078]

[0079] wherein, is the multi-modal input of the cross Mamba module, CS6 is the core calculation method of the cross Mamba module, cross scan i is a cross scan mechanism for serializing two-dimensional features and realizing two-modal information interaction, reverse scan i is a reverse scan mechanism for two-dimensional reconstruction of the serialized fusion features.

[0080] In the embodiments of the present application, CS6 is the core calculation method of the cross Mamba module, and the core idea is to take the input of the first mode as the historical state result, and utilize the result to interact with the input data of the second mode to obtain cross data with deep interconnection, and realize construction of complementary data bodies. The calculation process can be, but is not limited to, the following:

[0081]

[0082] wherein, represents the serialized state input of the two modal data.

[0083] Further, the embodiments of the present application splice the outputs of the two CS6 blocks to obtain the final output first discrete salient feature and the second discrete salient feature

[0084] For example, the embodiments of the present application perform global and local scan serialization on the two modal discrete extraction features, and then input the two modal discrete extraction features into the pre-constructed cross Mamba module to obtain the discrete salient features corresponding to the two modal discrete extraction features. CS6 is the core calculation method of the cross Mamba module, and then the outputs of the two CS6 blocks are spliced to obtain the final output first discrete salient feature and the second discrete salient feature

[0085] In some embodiments, the discrete salient features obtained are reconstructed so that their shapes regress to the original two-dimensional features, and then high-dimensional interaction features are obtained.

[0086] Specifically, the discrete salient features and Perform residual linking and add them to the input modal data to get the final feature interaction module. Since the data is still serialized at this time, it cannot be directly operated with low-dimensional features and needs to be reconstructed to get two-dimensional feature results. Therefore, the F with a shape of [batcH-size,H×W,dim] is converted to out Reconstruct high-dimensional interaction features and Its shape can be, but is not limited to, [batch-size, dim, H, W], where batch-size is the batch size of data, dim is the dimension of the feature, and H and W are the height and width of the feature.

[0087] For example, the embodiment of the present application is to serialize discrete significant features and Perform residual linking and add them to the input modal data to obtain the final feature interaction module. Since the data is still the result of serialization at this time, it cannot be directly operated with low-dimensional features. It is necessary to convert F out Reconstruct and reconstruct to obtain high-dimensional interaction features and

[0088] In step S104, the first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interaction features, and the second high-dimensional interaction features are respectively input into a pre-built offset guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and a first target detection result and a second target detection result of the detection target are generated based on the multi-dimensional fusion features.

[0089] As a possible implementation method, the embodiment of the present application can fuse the obtained multi-scale extraction features and high-dimensional interactive features to achieve the effect of high-dimensional non-offset features guiding the fusion of low-dimensional offset features, thereby maximizing the amount of information while reducing the impact of offset. Specifically, the embodiment of the present application can input the first multi-scale extraction features, the second multi-scale extraction features, the first high-dimensional interactive features and the second high-dimensional interactive features into a pre-built offset-guided fusion module respectively to obtain multi-dimensional fusion features. Furthermore, the embodiment of the present application detects the multi-dimensional fusion features and outputs the final target detection results, which include a total of three dimensions to achieve comprehensive detection of targets in remote sensing images.

[0090] Optionally, in one embodiment of the present application, the first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interaction features, and the second high-dimensional interaction features are respectively input into a pre-built offset-guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, including: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interaction features, and the second high-dimensional interaction features to obtain a multi-dimensional fusion feature. The calculation process of the multi-dimensional fusion feature can be, but is not limited to, as follows:

[0091]

[0092] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0093] In some embodiments, the content of generating multi-dimensional fusion features in the embodiments of the present application may include: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain a multi-dimensional fusion feature.

[0094] For example, the embodiment of the present application fuses the obtained multi-scale extracted features and high-dimensional interactive features, and uses high-dimensional features with or without offset to guide the fusion of multi-scale low-dimensional features. Specifically, Figure 2 As shown, the offset-guided fusion module pre-built in the embodiment of the present application receives three types of input data: high-dimensional features and low-dimensional features from two different modes. This method establishes a dual-branch feature fusion model by implementing channel reconstruction and channel residual preservation, maximizing information flow and achieving offset-guided fusion. The calculation process of the multi-dimensional fusion feature can be, but is not limited to, as follows:

[0095]

[0096] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0097] For example, the embodiments of the present application are verified through comparative experiments.

[0098] (1) Dataset Introduction

[0099] The data used in the comparative test of the embodiment of the present application are the DroneVehicle dataset from the perspective of drones, the LLVIP dataset from the perspective of traffic monitoring, and the VEDAI dataset from the perspective of aerial remote sensing.

[0100] Among them, the DroneVehicle dataset is a large-scale dataset that contains images captured by drones in various scenarios, making it highly representative. It provides 28,439 pairs of RGB and infrared images. It includes five categories of targets: cars, trucks, buses, vans, and trucks. Due to the position offset between the two modes, special consideration is required to achieve the best detection results. The dataset includes two annotation formats: horizontal box annotation and rotated box annotation, with separate labels for each modality. For training, the experiment selected 17,990 image pairs, and for testing, 1,469 image pairs were used.

[0101] The LLVIP dataset is a challenging dataset that contains images in both infrared and visible light modes captured by road monitoring cameras in low-light conditions. The dataset includes 16,836 RGB and thermal image pairs. The LLVIP dataset has lower viewing angles, closer objects, and only contains one pedestrian category, making detection slightly less difficult than for drone vehicles. However, the main challenge of the LLVIP dataset is that it consists entirely of nighttime scenes, which significantly reduces the information in the visible modality. Furthermore, the lower viewing angle causes objects to frequently occlude each other. To achieve the best results on this dataset, it is necessary to effectively fuse the infrared data with the visible light data while capturing the key features of the target. This allows for accurate detection even in situations where occlusion occurs.

[0102] The VEDAI dataset consists of RGB and infrared images captured by aircraft, including eight vehicle categories and over 3,700 annotated objects in over 1,200 images. The dataset provides images at two resolutions: 1024×1024 and 512×512. Table 1 provides an introduction to the dataset provided in the examples of this application.

[0103] Table 1

[0104]

[0105] (2) Introduction to comparison method

[0106] The examples of this application respectively use YOLOrs (method 1), CFT (method 2), SuperYOLO (method 3), ICAFusion (method 4), GM-DETR (method 5), DaFF (method 6), and CMADet (method 7) to perform comparative verification of target detection tasks.

[0107] (3) Introduction to evaluation indicators

[0108] The embodiment of the present application adopts a quantitative evaluation method for classification evaluation, and the mAP coefficient can be used as an evaluation indicator.

[0109] In the embodiment of the present application, the mAP coefficient is an authoritative evaluation index for evaluating target detection problems. The larger the mAP coefficient, the higher the accuracy. The calculation process can be:

[0110] First, calculate the precision PRecision and recall Recall, whose expressions can be but are not limited to:

[0111]

[0112] Among them, n TP 、N FP and N FN They represent the number of true positives (TP), false positives (FP), and false negatives (FN), respectively. TP means that the object is accurately identified, FP means that the object is incorrectly detected, and FN is the object instance that is missed in the detection result.

[0113] Then calculate the average precision AP, which can be expressed as but not limited to:

[0114]

[0115] Finally, the mAP coefficient is calculated, and its expression can be but not limited to:

[0116]

[0117] Among them, AP is determined by averaging the accuracy of a set of recall times S. Among them, the embodiment of the present application selects S = {0, 0.1, ..., 1}11 numbers to calculate AP. c represents the category of the data set. When calculating AP, if the intersection of the predicted result labels exceeds 0.5, it is classified as TP; otherwise, it is classified as FP. In addition, when the IOU (Intersection over Union) value of multiple objects and the ground truth is high, the object with the highest detection confidence is usually designated as TP. Subsequently, the AP is calculated as the average accuracy within the range of equally spaced recall rates represented by the set S. Finally, the AP results of multiple categories can be obtained, and then the AP of each category is averaged to obtain the mAP coefficient.

[0118] (4) Introduction of comparative test results

[0119] Among them, as shown in Table 2, the embodiment of the present application provides the comparative verification results of Method 1, Method 2, Method 3, Method 4, Method 5, Method 6, Method 7 and the embodiment of the present application. As can be seen from Table 2, compared with other methods, the embodiment of the present application can obtain a higher mAP coefficient, has a stronger detection capability, has a higher detection accuracy, can overcome the information bottleneck problem of a single modality, break through its detection limit, and achieve a higher accuracy and more robust detection method. Among them, Table 2 is a schematic table of comparative test results provided according to an embodiment of the present application.

[0120] Table 2

[0121]

[0122]

[0123] The working principle of the target detection method for multimodal remote sensing images proposed in the embodiment of the present application is described in detail below with reference to an embodiment.

[0124] Figure 2 A schematic diagram illustrating the working principle of a target detection method for multimodal remote sensing images provided according to one embodiment of the present application.

[0125] Step S201: Acquire visible light remote sensing images and infrared remote sensing images.

[0126] Among them, the embodiment of the present application is based on PYTHON and PYTORCH to construct two CSPDarknet53s with the same structure, which respectively extract the deep features of visible light remote sensing images and infrared remote sensing images. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained.

[0127] Step S202: Constructing a Mamba interaction module.

[0128] Among them, the embodiment of the present application can input the obtained first-scale extraction features of the visible light remote sensing image and the second-scale extraction features of the infrared remote sensing image into the pre-built Mamba interaction module respectively, and then obtain high-dimensional interaction features of the visible light remote sensing image and the infrared remote sensing image respectively.

[0129] Step S203: Construct an offset-guided fusion module.

[0130] In this embodiment of the present application, the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature can be respectively input into a pre-built offset-guided fusion module to obtain a multi-dimensional fusion feature.

[0131] Step S204: Output the final target detection result.

[0132] Among them, the embodiment of the present application can detect multi-dimensional fusion features and output the final target detection result, which contains a total of three dimensions and can achieve comprehensive detection of targets in remote sensing images.

[0133] Furthermore, the embodiments of the present application were verified through comparative tests. As can be seen from Table 2, compared with other methods, the embodiments of the present application can obtain a higher mAP coefficient, have stronger detection capabilities, and have higher detection accuracy. It can overcome the information bottleneck problem of a single modality and break through its detection limit, thereby achieving a detection method with higher accuracy and greater robustness.

[0134] According to the target detection method for multimodal remote sensing images proposed in the embodiment of the present application, feature extraction can be performed on the multimodal remote sensing images to obtain multi-scale extracted features corresponding to remote sensing images of different modalities, and feature interaction is performed based on the multi-scale extracted features to realize the construction of complementary data bodies. The high-dimensional interactive features of the multimodal remote sensing images are obtained by constructing a Mamba interaction module, and the multi-dimensional fusion features of the multimodal remote sensing images are obtained by constructing an offset-guided fusion module, thereby generating the final target detection result of the detected target. The offset problem between multimodal remote sensing images is fully considered, and the constructed Mamba interaction module is used to realize the calculation of the multimodal remote sensing images, achieving the effect of complementary information bodies between modalities, thereby increasing the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to reduce the offset effect between multimodal remote sensing images to ensure that the rich information of the multimodal remote sensing images can be maximized during the fusion process, which not only improves the accuracy of multimodal target detection but also significantly enhances the robustness of the model. This solves the problem in related technologies where data offsets between different sensors lead to differences between object positions and features, making it difficult for the detection model to accurately align information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0135] Next, a target detection device for multimodal remote sensing images proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0136] Figure 3 Schematic diagram of a block diagram of a target detection device for multimodal remote sensing images provided according to an embodiment of the present application.

[0137] like Figure 3 As shown, the target detection device 10 for multimodal remote sensing images includes: an extraction module 100 , an acquisition module 200 , an input module 300 and a generation module 400 .

[0138] The extraction module 100 is configured to obtain first multi-scale extraction features of the first modality remote sensing image and second multi-scale extraction features of the second modality remote sensing image based on the first modality remote sensing image and the second modality remote sensing image of the detection target.

[0139] The acquisition module 200 is configured to respectively acquire a first scale extraction feature of the first multi-scale extraction feature and a second scale extraction feature of the second multi-scale extraction feature based on the first multi-scale extraction feature and the second multi-scale extraction feature.

[0140] The input module 300 is used to input the first scale extracted features and the second scale extracted features into a pre-built Mamba interaction module respectively to obtain the first high-dimensional interaction features of the first modality remote sensing image and the second high-dimensional interaction features of the second modality remote sensing image.

[0141] The generation module 400 is used to input the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature into a pre-built offset-guided fusion module, respectively, to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and generate a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion features.

[0142] Optionally, in one embodiment of the present application, the extraction module 100 includes: an extraction unit.

[0143] Among them, the extraction unit is used to input the first modality remote sensing image and the second modality remote sensing image into the target convolutional neural network to extract the first multi-scale extraction feature and the second multi-scale extraction feature.

[0144] Optionally, in one embodiment of the present application, the input module 300 includes: a scanning unit, an input unit, an acquisition unit, and a reconstruction unit.

[0145] The scanning unit is used to perform global scanning on the first-scale extracted features and the second-scale extracted features respectively to obtain first modal discrete data of the first-scale extracted features and second modal discrete data of the second-scale extracted features.

[0146] The input unit is used to input the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain the first modal discrete extracted features of the first modal discrete data and the second modal discrete extracted features of the second modal discrete data.

[0147] An acquisition unit is used to combine the first modal discrete extraction feature, the second modal discrete extraction feature, the local scan and the pre-built cross Mamba module to acquire the first modal discrete salient feature of the first modal discrete extraction feature and the second modal discrete salient feature of the second modal discrete extraction feature.

[0148] The reconstruction unit is used to reconstruct the first modal discrete salient feature and the second modal discrete salient feature to obtain a first high-dimensional interaction feature and a second high-dimensional interaction feature.

[0149] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be, but is not limited to:

[0150]

[0151] in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0152] Optionally, in one embodiment of the present application, the generation module 400 includes: a fusion unit.

[0153] Among them, the fusion unit is used to use a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain a multi-dimensional fusion feature.

[0154] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be, but is not limited to, as follows:

[0155]

[0156] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0157] It should be noted that the aforementioned explanation of the embodiment of the target detection method for multimodal remote sensing images is also applicable to the target detection device for multimodal remote sensing images of this embodiment, and will not be repeated here.

[0158] According to the target detection device for multimodal remote sensing images proposed in the embodiment of the present application, feature extraction can be performed on the multimodal remote sensing images to obtain multi-scale extracted features corresponding to remote sensing images of different modalities, and feature interaction is performed based on the multi-scale extracted features to realize the construction of complementary data bodies. The high-dimensional interactive features of the multimodal remote sensing images are obtained by constructing a Mamba interaction module, and the multi-dimensional fusion features of the multimodal remote sensing images are obtained by constructing an offset-guided fusion module, thereby generating the final target detection result of the detected target. The offset problem between multimodal remote sensing images is fully considered, and the constructed Mamba interaction module is used to realize the calculation of the multimodal remote sensing images, achieving the effect of complementary information bodies between modalities, thereby increasing the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to reduce the offset effect between multimodal remote sensing images to ensure that the rich information of the multimodal remote sensing images can be maximized during the fusion process, which not only improves the accuracy of multimodal target detection but also significantly enhances the robustness of the model. This solves the problem in related technologies where data offsets between different sensors lead to differences between object positions and features, making it difficult for the detection model to accurately align information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0159] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0160] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0161] When the processor 402 executes the program, the target detection method for the multimodal remote sensing image provided in the above embodiment is implemented.

[0162] Furthermore, the electronic device further includes:

[0163] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0164] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0165] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0166] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0167] Optionally, in a specific implementation, if the memory 401 , the processor 402 and the communication interface 403 are integrated on a chip, the memory 401 , the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0168] The processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for detecting targets in multimodal remote sensing images.

[0170] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method for multimodal remote sensing images.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0172] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0173] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0174] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0175] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0176] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0177] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0178] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A target detection method for multimodal remote sensing images, characterized in that: The following steps are involved: Based on a first modality remote sensing image and a second modality remote sensing image of the detection target, obtaining a first multi-scale extracted feature of the first modality remote sensing image and a second multi-scale extracted feature of the second modality remote sensing image; Based on the first multi-scale extracted features and the second multi-scale extracted features, respectively obtaining a first scale extracted feature of the first multi-scale extracted features and a second scale extracted feature of the second multi-scale extracted features; Performing a global scan on the first-scale extracted features and the second-scale extracted features respectively to obtain first modal discrete data of the first-scale extracted features and second modal discrete data of the second-scale extracted features; Inputting the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain first modal discrete extracted features of the first modal discrete data and second modal discrete extracted features of the second modal discrete data; Combining the first modal discrete extracted features, the second modal discrete extracted features, local scanning, and a pre-built cross Mamba module to obtain first modal discrete salient features of the first modal discrete extracted features and second modal discrete salient features of the second modal discrete extracted features; Reconstructing the first modality discrete salient features and the second modality discrete salient features to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image; Inputting the first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interaction features, and the second high-dimensional interaction features into a pre-built offset-guided fusion module, respectively, to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and generating a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion features; The calculation process of the cross Mamba module is as follows: in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scan mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions; The calculation process of the multi-dimensional fusion feature is as follows: Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

2. The method according to claim 1, characterized in that The step of acquiring a first multi-scale extracted feature of the first modality remote sensing image and a second multi-scale extracted feature of the second modality remote sensing image based on the first modality remote sensing image and the second modality remote sensing image of the detected target includes: The first modality remote sensing image and the second modality remote sensing image are input into a target convolutional neural network to extract the first multi-scale extraction features and the second multi-scale extraction features.

3. The method according to claim 1, characterized in that The first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interactive features, and the second high-dimensional interactive features are respectively input into a pre-built offset-guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, including: The first multi-scale extracted features, the second multi-scale extracted features, the first high-dimensional interaction features and the second high-dimensional interaction features are fused using a pre-built dual-branch feature fusion model to obtain the multi-dimensional fusion feature.

4. A target detection device for multimodal remote sensing images, characterized in that: include: an extraction module, configured to obtain, based on a first modality remote sensing image and a second modality remote sensing image of a detected target, a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image; A first acquisition module is configured to respectively acquire a first scale extraction feature of the first multi-scale extraction feature and a second scale extraction feature of the second multi-scale extraction feature based on the first multi-scale extraction feature and the second multi-scale extraction feature; a first generating module, configured to globally scan the first-scale extracted features and the second-scale extracted features respectively, to obtain first modal discrete data of the first-scale extracted features and second modal discrete data of the second-scale extracted features; a second acquisition module, configured to input the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain first modal discrete extracted features of the first modal discrete data and second modal discrete extracted features of the second modal discrete data; a third acquisition module, configured to combine the first modal discrete extracted features, the second modal discrete extracted features, a local scan, and a pre-built cross-Mamba module to acquire first modal discrete significant features of the first modal discrete extracted features and second modal discrete significant features of the second modal discrete extracted features; a second generating module, configured to reconstruct the first modal discrete salient features and the second modal discrete salient features to obtain a first high-dimensional interaction feature of the first modal remote sensing image and a second high-dimensional interaction feature of the second modal remote sensing image; a third generation module, configured to input the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature into a pre-built offset-guided fusion module, respectively, to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and generate a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion features; The calculation process of the cross Mamba module is as follows: in, It is the multimodal input of the cross Mamba module. CS6 is the core calculation method of the cross Mamba module. i It is a cross-scan mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions; The calculation process of the multi-dimensional fusion feature is as follows: Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method for multimodal remote sensing images according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the target detection method for multimodal remote sensing images as described in any one of claims 1 to 3.

7. A computer program product, characterized in that The invention comprises a computer program, which, when executed, is used to implement the target detection method for multimodal remote sensing images as described in any one of claims 1 to 3.

Images