Target detection method and device for multi-modal remote sensing image

Abstract

The invention relates to the technical field of remote sensing image processing, in particular to a multi-modal remote sensing image target detection method and device, and the method comprises the steps: obtaining corresponding multi-scale extraction features based on a first modal remote sensing image and a second modal remote sensing image of a detection target; respectively acquiring scale extraction features of the two modal remote sensing images based on the multi-scale extraction features; respectively inputting the first scale extraction feature and the second scale extraction feature into a pre-constructed Mamba interaction module to obtain high-dimensional interaction features of the two modal remote sensing images; and inputting the multi-scale extraction features and the high-dimensional interaction features into a pre-constructed offset guide fusion module to obtain multi-dimensional fusion features between the two modal remote sensing images so as to generate a final target detection result. Therefore, the problems of poor precision, poor overall performance and the like of the detection model due to differences between the positions and the features of the objects in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for target detection of multimodal remote sensing images. Background Art

[0002] The object detection task can quickly understand an image and determine the location of an object. It is one of the important tasks in the field of computer vision and has been used in many fields such as autonomous driving, remote sensing, and medical imaging. However, the single-modal object detection task is not capable of capturing the salient features of objects in complex scenarios such as lighting, weather conditions, and occlusion, resulting in reduced accuracy. Visual multimodal data consists of images acquired by different sensors and can provide richer information on object feature attributes. Recent advances in multimodal fusion technology have greatly improved object detection performance.

[0003] In related technologies, data from different modalities, including RGB images, infrared images, lidar, and radar, can be effectively combined through methods such as feature-level fusion, decision-level fusion, and pixel-level fusion. These methods can take advantage of the complementarity of multimodal data and improve the ability to detect objects even in low light, occlusion, or bad weather conditions.

[0004] However, in the related technology, due to the data offset of different sensors, there will be differences between the object position and features. If feature association and fusion are performed directly, it will make it difficult for the detection model to accurately align the information in each modality, thereby affecting the detection accuracy and reducing the overall performance of the model. Improvement is urgently needed. Summary of the invention

[0005] The present application provides a method and device for target detection in multimodal remote sensing images to solve the problem in the related art that, due to data offset from different sensors, there will be differences between the position and features of the object, making it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0006] The first aspect of the present application provides a method for detecting a target in a multimodal remote sensing image, comprising the following steps: based on a first modality remote sensing image and a second modality remote sensing image of a detection target, obtaining a first multiscale extraction feature of the first modality remote sensing image and a second multiscale extraction feature of the second modality remote sensing image; based on the first multiscale extraction feature and the second multiscale extraction feature, respectively obtaining a first scale extraction feature of the first multiscale extraction feature and a second scale extraction feature of the second multiscale extraction feature; respectively inputting the first scale extraction feature and the second scale extraction feature into a pre-constructed Mamba interaction module to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image; respectively inputting the first multiscale extraction feature, the second multiscale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature into a pre-constructed offset-guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and generating a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion feature.

[0007] Optionally, in one embodiment of the present application, the first multi-scale extraction features of the first modality remote sensing image and the second multi-scale extraction features of the second modality remote sensing image are obtained based on the first modality remote sensing image and the second multi-scale extraction features of the second modality remote sensing image, including: inputting the first modality remote sensing image and the second modality remote sensing image into a target convolutional neural network to extract the first multi-scale extraction features and the second multi-scale extraction features.

[0008] Optionally, in one embodiment of the present application, the first scale extraction feature and the second scale extraction feature are respectively input into a pre-constructed Mamba interaction module to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image, including: performing a global scan on the first scale extraction feature and the second scale extraction feature respectively to obtain first modality discrete data of the first scale extraction feature and second modality discrete data of the second scale extraction feature; inputting the first modality discrete data and the second modality discrete data into a pre-constructed single Mamba module to obtain first modality discrete extraction features of the first modality discrete data and second modality discrete extraction features of the second modality discrete data; combining the first modality discrete extraction features, the second modality discrete extraction features, local scanning and a pre-constructed cross Mamba module to obtain first modality discrete significant features of the first modality discrete extraction features and second modality discrete significant features of the second modality discrete extraction features; reconstructing the first modality discrete significant features and the second modality discrete significant features to obtain the first high-dimensional interaction features and the second high-dimensional interaction features.

[0009] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be but is not limited to:

[0010]

[0011] in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0012] Optionally, in one embodiment of the present application, the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature are respectively input into a pre-built offset guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, including: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain the multi-dimensional fusion feature.

[0013] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be but is not limited to:

[0014]

[0015] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0016] According to a second aspect of the present application, a target detection device for a multimodal remote sensing image is provided, comprising: an extraction module, which is used to obtain a first multiscale extraction feature of the first modal remote sensing image and a second multiscale extraction feature of the second modal remote sensing image based on the first modal remote sensing image and the second modal remote sensing image of the detection target; an acquisition module, which is used to obtain a first scale extraction feature of the first multiscale extraction feature and a second scale extraction feature of the second multiscale extraction feature based on the first multiscale extraction feature and the second multiscale extraction feature; an input module, which is used to input the first scale extraction feature and the second scale extraction feature into a pre-constructed Mamba interaction module, respectively, to obtain a first high-dimensional interaction feature of the first modal remote sensing image and a second high-dimensional interaction feature of the second modal remote sensing image; a generation module, which is used to input the first multiscale extraction feature, the second multiscale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature into a pre-constructed offset guided fusion module, respectively, to obtain multi-dimensional fusion features of the first modal remote sensing image and the second modal remote sensing image, and generate a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion features.

[0017] Optionally, in one embodiment of the present application, the extraction module includes: an extraction unit, used to input the first modality remote sensing image and the second modality remote sensing image into a target convolutional neural network to extract the first multi-scale extraction features and the second multi-scale extraction features.

[0018] Optionally, in one embodiment of the present application, the input module includes: a scanning unit, used to perform a global scan on the first scale extraction feature and the second scale extraction feature respectively to obtain the first modal discrete data of the first scale extraction feature and the second modal discrete data of the second scale extraction feature; an input unit, used to input the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain the first modal discrete extraction feature of the first modal discrete data and the second modal discrete extraction feature of the second modal discrete data; an acquisition unit, used to combine the first modal discrete extraction feature, the second modal discrete extraction feature, local scanning and the pre-built cross Mamba module to obtain the first modal discrete significant feature of the first modal discrete extraction feature and the second modal discrete significant feature of the second modal discrete extraction feature; a reconstruction unit, used to reconstruct the first modal discrete significant feature and the second modal discrete significant feature to obtain the first high-dimensional interaction feature and the second high-dimensional interaction feature.

[0019] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be but is not limited to:

[0020]

[0021] in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0022] Optionally, in one embodiment of the present application, the generation module includes: a fusion unit, used to use a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain the multi-dimensional fusion feature.

[0023] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be but is not limited to:

[0024]

[0025] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0026] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method for multimodal remote sensing images as described in the above embodiments.

[0027] A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the above-mentioned method for detecting targets in multimodal remote sensing images.

[0028] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method for multimodal remote sensing images.

[0029] The embodiment of the present application can perform feature extraction on multimodal remote sensing images to obtain multiscale extraction features corresponding to remote sensing images of different modalities, and perform feature interaction based on multiscale extraction features to realize complementary data body construction, and obtain high-dimensional interactive features of multimodal remote sensing images by constructing a Mamba interaction module, and obtain multidimensional fusion features of multimodal remote sensing images by constructing an offset-guided fusion module, thereby generating the final target detection result of the detection target, taking full account of the offset problem between multimodal remote sensing images, and using the constructed Mamba interaction module to realize the calculation of multimodal remote sensing images, so as to achieve the effect of complementary information body between modalities, thereby improving the overall characteristic information volume, in addition, the embodiment of the present application also starts from how to weaken the offset effect between multimodal remote sensing images, ensuring that the rich information of multimodal remote sensing images can be maximized in the fusion process, not only improving the accuracy of multimodal target detection, but also significantly enhancing the robustness of the model. Thus, the problem that the data offset of different sensors in the related art will cause differences between object positions and features, resulting in the difficulty of the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0030] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A flowchart of a method for detecting a target in a multimodal remote sensing image according to an embodiment of the present application;

[0033] Figure 2A block diagram of the working principle of a method for detecting a target in a multimodal remote sensing image according to an embodiment of the present application;

[0034] Figure 3 A block diagram of a target detection device for a multimodal remote sensing image provided according to an embodiment of the present application;

[0035] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the target detection method and device for multimodal remote sensing images of the embodiment of the present application with reference to the accompanying drawings. In view of the problem mentioned in the above background technology that due to the data offset of different sensors, there will be differences between the position and features of the object, which makes it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model, the present application provides a target detection method for multimodal remote sensing images. In this method, feature extraction can be performed on multimodal remote sensing images to obtain multi-scale extracted features corresponding to remote sensing images of different modalities, and feature interaction is performed based on the multi-scale extracted features to realize the construction of complementary data bodies, and high-dimensional interactive features of multimodal remote sensing images are obtained by constructing a Mamba interaction module. Construct an offset-guided fusion module to obtain the multi-dimensional fusion features of multimodal remote sensing images, and then generate the final target detection result of the detection target, fully consider the offset problem between multimodal remote sensing images, and use the constructed Mamba interaction module to realize the calculation of multimodal remote sensing images, so as to achieve the effect of complementary information body between modalities, thereby improving the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to weaken the offset effect between multimodal remote sensing images, ensuring that the rich information of multimodal remote sensing images can be maximized during the fusion process, which not only improves the accuracy of multimodal target detection, but also significantly enhances the robustness of the model. Thus, the problem in the related art that due to the data offset of different sensors, there will be differences between the position and features of the object, which makes it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0038] Specifically, Figure 1 The present invention provides a flowchart of a method for detecting targets in a multimodal remote sensing image according to an embodiment of the present application.

[0039] like Figure 1 As shown, the target detection method of the multimodal remote sensing image includes the following steps:

[0040] In step S101, based on a first modality remote sensing image and a second modality remote sensing image of a detection target, a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image are acquired.

[0041] It can be understood that the first modality remote sensing image in the embodiment of the present application may be a visible light remote sensing image, which can be represented by D rgb , the second modality remote sensing image may be an infrared remote sensing image, which may be expressed as D ir , where D rgb and D ir Having the same data tags, the two modal remote sensing images are then read into the computing device.

[0042] In some embodiments, the embodiments of the present application can obtain first multi-scale extraction features of visible light remote sensing images based on visible light remote sensing images, and use infrared remote sensing images to obtain second multi-scale extraction features of infrared remote sensing images.

[0043] For example, the embodiment of the present application can be written in PYTHON language, implemented using the classic deep learning framework PYTORCH, and use the Dataset function and DataLoader function in PYTORCH as data reading methods to read multimodal remote sensing image datasets respectively. Among them, the PYTORCH data reading and loading functions are well-known technologies in the technical field and will not be described in detail here.

[0044] Optionally, in one embodiment of the present application, based on the first modality remote sensing image and the second modality remote sensing image of the detected target, a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image are obtained, including: inputting the first modality remote sensing image and the second modality remote sensing image into the target convolutional neural network to extract the first multi-scale extraction feature and the second multi-scale extraction feature.

[0045] It should be noted that the embodiment of the present application can utilize the target convolutional neural network to extract multi-scale extraction features of multimodal remote sensing images. In the embodiment of the present application, the target convolutional neural network can be but is not limited to CSPDarknet53, which can be specifically set by technical personnel in this field according to actual conditions, and this application does not make any specific restrictions.

[0046] As a possible implementation method, the embodiment of the present application can construct a target convolutional neural network and use the target convolutional neural network to extract multi-scale extraction features of the two-modal remote sensing images respectively.

[0047] Exemplarily, the embodiment of the present application constructs two CSPDarknet53s with the same structure to extract deep features of visible light remote sensing images and infrared remote sensing images respectively. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained as the first multi-scale extraction features of the visible light remote sensing image and the second multi-scale extraction features of the infrared remote sensing image, respectively.

[0048] For example, the embodiment of the present application constructs two CSPDarknet53s with the same structure based on PYTHON and PYTORCH to extract deep features of visible light remote sensing images and infrared remote sensing images respectively. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained.

[0049] In step S102, based on the first multi-scale extracted features and the second multi-scale extracted features, first scale extracted features of the first multi-scale extracted features and second scale extracted features of the second multi-scale extracted features are respectively obtained.

[0050] In some embodiments, the embodiment of the present application may select the feature least affected by the offset in the first multi-scale extraction feature as the fusion feature to obtain the first scale extraction feature, thereby achieving the effect of reducing the impact of the offset.

[0051] In some embodiments, the embodiments of the present application may select the feature least affected by the offset in the second multi-scale extraction feature as the fusion feature to obtain the second scale extraction feature, thereby achieving the effect of reducing the impact of the offset.

[0052] It can be understood that in the embodiment of the present application, due to the existence of a certain offset phenomenon in the multimodal data, if it is not taken into account, the accuracy of feature extraction and multimodal detection will be reduced, and the impact of the offset at different scales is different, which can be specifically expressed as but not limited to:

[0053] A intersection =|w-Δx|×|h-Δy|,

[0054] Among them, Δx and Δy are offsets, and both values ​​are fixed for multimodal image pairs; w and h are the width and height of the image block, with different degrees of downsampling.

[0055] Furthermore, in the embodiment of the present application, since the offset is fixed, larger w and h are required to obtain a larger intersection area A. textintersection At the same time, using only high-level features for multimodal interaction can greatly reduce the amount of computation and improve the real-time performance of the network.

[0056] In step S103, the first scale extracted features and the second scale extracted features are respectively input into a pre-built Mamba interaction module to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image.

[0057] It can be understood that the Mamba interaction module constructed in the embodiment of the present application may include but is not limited to a single Mamba module that only processes a single modality remote sensing image and a cross Mamba module that processes multimodal remote sensing images. It can be specifically set by technicians in this field according to actual conditions, and this application does not make specific restrictions.

[0058] As a possible implementation method, the embodiment of the present application can input the first scale extraction features and the second scale extraction features into a pre-built Mamba interaction module respectively, thereby obtaining the first high-dimensional interaction features of the first modality remote sensing image and the second high-dimensional interaction features of the second modality remote sensing image.

[0059] For example, the embodiment of the present application can input the obtained first-scale extraction features of the visible light remote sensing image and the second-scale extraction features of the infrared remote sensing image into a pre-built Mamba interaction module, thereby obtaining the first high-dimensional interaction features of the visible light remote sensing image and the second high-dimensional interaction features of the infrared remote sensing image.

[0060] Optionally, in one embodiment of the present application, the first scale extraction feature and the second scale extraction feature are respectively input into a pre-constructed Mamba interaction module to obtain the first high-dimensional interaction feature of the first modality remote sensing image and the second high-dimensional interaction feature of the second modality remote sensing image, including: performing a global scan on the first scale extraction feature and the second scale extraction feature respectively to obtain the first modality discrete data of the first scale extraction feature and the second modality discrete data of the second scale extraction feature; inputting the first modality discrete data and the second modality discrete data into a pre-constructed single Mamba module to obtain the first modality discrete extraction feature of the first modality discrete data and the second modality discrete extraction feature of the second modality discrete data; combining the first modality discrete extraction feature, the second modality discrete extraction feature, the local scan and the pre-constructed cross-Mamba module to obtain the first modality discrete significant feature of the first modality discrete extraction feature and the second modality discrete significant feature of the second modality discrete extraction feature; reconstructing the first modality discrete significant feature and the second modality discrete significant feature to obtain the first high-dimensional interaction feature and the second high-dimensional interaction feature. Among them, the calculation process of the cross-Mamba module can be but is not limited to:

[0061]

[0062] in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0063] In some embodiments, the embodiments of the present application can perform global scans on the first scale extraction features and the second scale extraction features respectively to discretize the serialized feature data, and input the two modal discrete data into a pre-built single Mamba module to achieve the effect of further feature extraction and obtain the corresponding modal discrete extraction features.

[0064] Specifically, the embodiment of the present application is for the input feature S in , the feature matrix F can be constructed using adaptive maximum pooling and mean pooling in ∈R H×W×C , to ensure that the dimensions remain consistent across image sizes:

[0065]

[0066] Furthermore, the embodiments of the present application are directed to F in A deep feature map is performed and a Drop layer is added to make the module adaptive:

[0067]

[0068] Among them, h is the channel of hidden features in the mapping process, It is a linear mapping operation, and Drop means randomly discarding neurons with a certain probability.

[0069] Then, the present embodiment flattens the obtained matrix into a tag sequence, that is, Simulate the sequential data of the state space model algorithm. In order to reduce the loss of two-dimensional spatial information, the embodiment of the present application also adds a learnable position embedding It provides clear position encoding. In addition, the embodiment of the present application establishes a parallel data stream for operation, maintaining the integrity of the original feature extraction. Further, the embodiment of the present application I in Scanning can be performed in four directions, thereby expanding the serialization method with position coding, thereby expanding the data distribution. In the embodiment of the present application, the scanning results in each direction will be subjected to S6 for sequence feature state space model feature extraction to obtain four output results, which can be represented by yi respectively, which can be represented by but not limited to:

[0070] x i=crossscan i (I in ),

[0071] y i =S6 i (x i ),

[0072]

[0073] Among them, i=1, 2, 3, 4. represents the directions of four global scans.

[0074] For example, the embodiment of the present application can perform global scanning by extracting features at two modal scales, and discretely serialize the feature data, and respectively input the two modal discrete data into a pre-built single Mamba module to achieve the effect of further feature extraction, and obtain the corresponding modal discrete extraction features, the content of which may include:

[0075] In this embodiment of the present application, for the input feature S in First, the feature matrix F is constructed using adaptive maximum pooling and mean pooling. in ∈R H×W×C , to ensure that the dimensions of different image sizes remain consistent; then, in A deep feature map is performed and a Drop layer is added to make the model adaptive; then, the resulting matrix is ​​flattened into a labeled sequence, i.e. Sequential data for simulating state-space model algorithms. In order to reduce the loss of two-dimensional spatial information, learnable position embedding is added It provides an explicit positional encoding and also establishes a parallel data flow for operations, preserving the integrity of the original feature extraction. in The scan will be performed in four directions to expand the serialization method with position encoding, thereby expanding the data distribution. Finally, the scan results in each direction will be respectively subjected to S6 for sequence feature state space model feature extraction to obtain four output results, which can be used to represent the state space model of the sequence feature state space model. i express.

[0076] In some embodiments, the embodiments of the present application can fuse the first modal discrete extraction features and the second modal discrete extraction features, specifically by globally and locally scanning and serializing the two modal discrete extraction features, and then simultaneously inputting the two modal discrete extraction features into a pre-built cross-Mamba module to achieve the effect of deep interaction between modalities, thereby fusing the significant features of the two modalities, increasing the amount of information, and obtaining discrete significant features corresponding to the two modal discrete extraction features.

[0077] Specifically, the calculation process of the cross-Mamba module in the embodiment of the present application may be, but is not limited to, as follows:

[0078]

[0079] in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0080] Among them, in the embodiment of the present application, CS6 is the core calculation method of the cross Mamba module, and its core idea is to use the input of the first modality as the historical state result, and use the result to interact with the input data of the second modality to obtain deeply interconnected cross data and realize the construction of complementary data body. Its calculation process can be but not limited to:

[0081]

[0082] in, Represents the serialized state input of the two modal data.

[0083] Furthermore, the embodiment of the present application concatenates the outputs of the two CS6 blocks to obtain the final output first discrete salient feature and the second discrete salient feature

[0084] For example, the embodiment of the present application performs global and local scanning serialization on the two modal discrete extraction features, and then simultaneously inputs the two modal discrete extraction features into the pre-built cross-Mamba module to obtain the discrete salient features corresponding to the two modal discrete extraction features. Among them, CS6 is the core calculation method of the cross-Mamba module, and then the outputs of the two CS6 blocks are spliced ​​to obtain the final output first discrete salient feature and the second discrete salient feature

[0085] In some embodiments, the embodiments of the present application reconstruct the obtained discrete significant features so that their shapes return to the original two-dimensional features, thereby obtaining high-dimensional interactive features.

[0086] Specifically, the present application embodiment provides a discrete salient feature of the sequence and Perform residual linking and add them to the input modal data to get the final feature interaction module. Since the data is still the result of serialization at this time, it cannot be directly operated with low-dimensional features and needs to be reconstructed to get two-dimensional feature results. Therefore, the F with a shape of [batcH-size,H×W,dim] is out Reconstruct high-dimensional interactive features and Its shape can be, but is not limited to, [batch-size, dim, H, W], where batch-size is the batch size of data, dim is the dimension of the feature, and H and W are the height and width of the feature.

[0087] For example, the present invention provides a discrete salient feature of the sequence and Perform residual linking and add them to the input modal data to obtain the final feature interaction module. Since the data is still the result of serialization at this time, it cannot be directly operated with low-dimensional features. out Reconstruct and reconstruct to obtain high-dimensional interaction features and

[0088] In step S104, the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature are respectively input into a pre-built offset guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and a first target detection result and a second target detection result of the detection target are generated based on the multi-dimensional fusion features.

[0089] As a possible implementation method, the embodiment of the present application can fuse the obtained multi-scale extraction features and high-dimensional interactive features to achieve the effect of high-dimensional non-offset features guiding the fusion of low-dimensional offset features, thereby maximizing the amount of information while reducing the impact of offset. Specifically, the embodiment of the present application can input the first multi-scale extraction features, the second multi-scale extraction features, the first high-dimensional interactive features, and the second high-dimensional interactive features into a pre-built offset guided fusion module respectively to obtain multi-dimensional fusion features. Furthermore, the embodiment of the present application detects the multi-dimensional fusion features and outputs the final target detection results, which include a total of three dimensions to achieve comprehensive detection of targets in remote sensing images.

[0090] Optionally, in one embodiment of the present application, the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature are respectively input into a pre-built offset-guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, including: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain a multi-dimensional fusion feature. The calculation process of the multi-dimensional fusion feature can be, but is not limited to, as follows:

[0091]

[0092] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0093] In some embodiments, the content of generating multidimensional fusion features in the embodiments of the present application may include: using a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain a multi-dimensional fusion feature.

[0094] Exemplarily, the embodiment of the present application fuses the obtained multi-scale extracted features and high-dimensional interactive features, and uses high-dimensional features with or without offsets to guide the fusion of multi-scale low-dimensional features. Specifically, Figure 2 As shown, the offset-guided fusion module pre-built in the embodiment of the present application receives three types of input data: high-dimensional features and low-dimensional features from two different modes. The method establishes a dual-branch feature fusion model by realizing channel reconstruction and channel residual preservation, maximizes information flow, and realizes offset-guided fusion. The calculation process of the multi-dimensional fusion feature can be, but is not limited to, as follows:

[0095]

[0096] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0097] For example, the embodiments of the present application are verified through comparative experiments.

[0098] (1) Dataset Introduction

[0099] The data used in the comparative test of the embodiment of the present application are the DroneVehicle dataset from the perspective of a drone, the LLVIP dataset from the perspective of traffic monitoring, and the VEDAI dataset from the perspective of aerial remote sensing.

[0100] Among them, the DroneVehicle dataset is a large-scale dataset that contains images captured by drones in various scenarios, making it highly representative. It provides 28,439 pairs of RGB and infrared images. It includes five categories of targets: cars, trucks, buses, vans, and trucks. Due to the position offset between the two modes, special consideration is required to achieve the best detection results. The dataset includes two annotation formats: horizontal box annotations and rotated box annotations, with separate labels for each modality. For training, 17,990 image pairs were selected in the experiment, and for testing, 1,469 image pairs were used.

[0101] The LLVIP dataset is a challenging dataset that contains images in both infrared and visible light modes captured by road monitoring cameras in low-light conditions. The dataset includes 16,836 RGB and thermal image pairs. The LLVIP dataset has lower viewing angles and closer objects, and contains only one pedestrian category, which is slightly less difficult to detect compared to drone vehicles. However, the main challenge of the LLVIP dataset is that it consists entirely of night scenes, which makes the information of the visible modality significantly reduced. In addition, the lower viewing angle causes objects to frequently occlude each other. In order to achieve the best results on this dataset, it is necessary to effectively fuse the infrared data with the visible light data while capturing the key features of the target. This allows accurate detection even in situations where occlusion occurs.

[0102] The VEDAI dataset consists of RGB and infrared images captured by aircraft, including 8 vehicle categories, with more than 3700 annotated objects in more than 1200 images. The dataset provides images with two resolutions: 1024×1024 and 512×512. Table 1 is an introduction table of the dataset provided according to the embodiments of the present application.

[0103] Table 1

[0104]

[0105] (2) Introduction to comparison method

[0106] The embodiments of the present application respectively use YOLOrs (method 1), CFT (method 2), SuperYOLO (method 3), ICAFusion (method 4), GM-DETR (method 5), DaFF (method 6), and CMADet (method 7) to perform comparative verification of target detection tasks.

[0107] (3) Introduction to evaluation indicators

[0108] The embodiment of the present application adopts a quantitative evaluation method for classification evaluation, and the mAP coefficient can be used as an evaluation indicator.

[0109] Among them, in the embodiment of the present application, the mAP coefficient is an authoritative evaluation index used to evaluate the target detection problem. The larger the mAP coefficient, the higher the accuracy. The calculation process can be:

[0110] First, calculate the precision PRecision and recall Recall, whose expressions can be but are not limited to:

[0111]

[0112] Among them, n TP 、N FP and N FN They represent the number of true positives (TP), false positives (FP), and false negatives (FN), respectively. TP means that the object is accurately identified, FP means that the object is incorrectly detected, and FN is the object instance missed in the detection result.

[0113] Then calculate the average precision AP, which can be expressed as but not limited to:

[0114]

[0115] Finally, the mAP coefficient is calculated, and its expression can be but not limited to:

[0116]

[0117] Among them, AP is determined by averaging the accuracy of a set of recall times S. Among them, the embodiment of the present application selects S = {0, 0.1, ..., 1} 11 numbers to calculate AP. c represents the category of the data set. When calculating AP, if the intersection of the predicted result labels exceeds 0.5, it is classified as TP; otherwise, it is classified as FP. In addition, when the IOU (Intersection over Union) value of multiple objects with the ground truth is high, the object with the highest detection confidence is usually designated as TP. Subsequently, the AP is calculated as the average accuracy within the range of equally spaced recall rates represented by the set S. Finally, the AP results of multiple categories can be obtained, and then the AP of each category is averaged to obtain the mAP coefficient.

[0118] (4) Introduction of comparative test results

[0119] Among them, as shown in Table 2, the embodiment of the present application provides the comparative verification results of method 1, method 2, method 3, method 4, method 5, method 6, method 7 and the embodiment of the present application. It can be seen from Table 2 that compared with other methods, the embodiment of the present application can obtain a higher mAP coefficient, has a stronger detection capability, has a higher detection accuracy, can overcome the information bottleneck problem of a single modality, break through its detection limit, and achieve a higher accuracy and more robust detection method. Among them, Table 2 is a schematic table of comparative test results provided according to an embodiment of the present application.

[0120] Table 2

[0121]

[0122]

[0123] The working principle of the target detection method for multimodal remote sensing images proposed in the embodiment of the present application is described in detail below in conjunction with an embodiment.

[0124] Figure 2 A schematic diagram of the working principle of a target detection method for a multimodal remote sensing image provided according to an embodiment of the present application.

[0125] Step S201: Acquire visible light remote sensing images and infrared remote sensing images.

[0126] Among them, the embodiment of the present application is based on PYTHON and PYTORCH to construct two CSPDarknet53s with the same structure, which respectively extract the deep features of visible light remote sensing images and infrared remote sensing images. Through this network, multi-scale extraction features of different downsampling dimensions can be obtained, and only the last three scale feature extraction results of the two modal remote sensing images are retained.

[0127] Step S202: constructing a Mamba interaction module.

[0128] Among them, the embodiment of the present application can input the obtained first-scale extraction features of the visible light remote sensing image and the second-scale extraction features of the infrared remote sensing image into the pre-built Mamba interaction module respectively, and then obtain the high-dimensional interaction features of the visible light remote sensing image and the infrared remote sensing image respectively.

[0129] Step S203: construct an offset-guided fusion module.

[0130] Among them, the embodiment of the present application can input the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature into a pre-built offset guided fusion module to obtain a multi-dimensional fusion feature.

[0131] Step S204: Output the final target detection result.

[0132] Among them, the embodiment of the present application can detect multi-dimensional fusion features and output the final target detection result, which contains a total of three dimensions and can achieve comprehensive detection of targets in remote sensing images.

[0133] Furthermore, the embodiments of the present application are verified through comparative tests. It can be seen from Table 2 that, compared with other methods, the embodiments of the present application can obtain a higher mAP coefficient, have a stronger detection capability, have a higher detection accuracy, can overcome the information bottleneck problem of a single modality, break through its detection limit, and achieve a higher accuracy and more robust detection method.

[0134] According to the target detection method for multimodal remote sensing images proposed in the embodiment of the present application, feature extraction can be performed on the multimodal remote sensing images to obtain multi-scale extracted features corresponding to remote sensing images of different modalities, and feature interaction is performed based on the multi-scale extracted features to realize the construction of complementary data bodies, and high-dimensional interactive features of multimodal remote sensing images are obtained by constructing a Mamba interaction module, and multi-dimensional fusion features of multimodal remote sensing images are obtained by constructing an offset-guided fusion module, thereby generating a final target detection result of the detected target, fully considering the offset problem between multimodal remote sensing images, and using the constructed Mamba interaction module to realize the calculation of multimodal remote sensing images, so as to achieve the effect of complementary information bodies between modalities, thereby improving the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to reduce the offset effect between multimodal remote sensing images to ensure that the rich information of multimodal remote sensing images can be maximized in the fusion process, which not only improves the accuracy of multimodal target detection, but also significantly enhances the robustness of the model. This solves the problem in the related art that, due to data offset from different sensors, there will be differences between the position and features of the object, making it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0135] Next, the target detection device for multimodal remote sensing images proposed in accordance with the embodiments of the present application will be described with reference to the accompanying drawings.

[0136] Figure 3 A block diagram of a target detection device for a multimodal remote sensing image provided according to an embodiment of the present application.

[0137] like Figure 3 As shown, the target detection device 10 for multimodal remote sensing images includes: an extraction module 100, an acquisition module 200, an input module 300 and a generation module 400.

[0138] The extraction module 100 is used to obtain first multi-scale extraction features of the first modality remote sensing image and second multi-scale extraction features of the second modality remote sensing image based on the first modality remote sensing image and the second modality remote sensing image of the detected target.

[0139] The acquisition module 200 is used to respectively acquire a first scale extraction feature of the first multi-scale extraction feature and a second scale extraction feature of the second multi-scale extraction feature based on the first multi-scale extraction feature and the second multi-scale extraction feature.

[0140] The input module 300 is used to input the first scale extraction feature and the second scale extraction feature into the pre-built Mamba interaction module respectively to obtain the first high-dimensional interaction feature of the first modality remote sensing image and the second high-dimensional interaction feature of the second modality remote sensing image.

[0141] The generation module 400 is used to input the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature into a pre-built offset guided fusion module, respectively, to obtain the multi-dimensional fusion feature of the first modality remote sensing image and the second modality remote sensing image, and generate the first target detection result and the second target detection result of the detection target based on the multi-dimensional fusion feature.

[0142] Optionally, in one embodiment of the present application, the extraction module 100 includes: an extraction unit.

[0143] Among them, the extraction unit is used to input the first modality remote sensing image and the second modality remote sensing image into the target convolutional neural network to extract the first multi-scale extraction feature and the second multi-scale extraction feature.

[0144] Optionally, in one embodiment of the present application, the input module 300 includes: a scanning unit, an input unit, an acquisition unit and a reconstruction unit.

[0145] The scanning unit is used to perform global scanning on the first-scale extracted features and the second-scale extracted features respectively to obtain first modal discrete data of the first-scale extracted features and second modal discrete data of the second-scale extracted features.

[0146] The input unit is used to input the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain the first modal discrete extracted features of the first modal discrete data and the second modal discrete extracted features of the second modal discrete data.

[0147] The acquisition unit is used to combine the first modal discrete extraction features, the second modal discrete extraction features, the local scan and the pre-built cross Mamba module to acquire the first modal discrete significant features of the first modal discrete extraction features and the second modal discrete significant features of the second modal discrete extraction features.

[0148] The reconstruction unit is used to reconstruct the first modal discrete salient feature and the second modal discrete salient feature to obtain a first high-dimensional interactive feature and a second high-dimensional interactive feature.

[0149] Optionally, in one embodiment of the present application, the calculation process of the cross Mamba module may be but is not limited to:

[0150]

[0151] in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scanning mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

[0152] Optionally, in one embodiment of the present application, the generation module 400 includes: a fusion unit.

[0153] Among them, the fusion unit is used to use a pre-built dual-branch feature fusion model to fuse the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature to obtain a multi-dimensional fusion feature.

[0154] Optionally, in one embodiment of the present application, the calculation process of the multi-dimensional fusion feature may be, but is not limited to, as follows:

[0155]

[0156] Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

[0157] It should be noted that the aforementioned explanation of the embodiment of the target detection method for multimodal remote sensing images is also applicable to the target detection device for multimodal remote sensing images of this embodiment, and will not be repeated here.

[0158] According to the target detection device for multimodal remote sensing images proposed in the embodiment of the present application, feature extraction can be performed on the multimodal remote sensing images to obtain multi-scale extraction features corresponding to remote sensing images of different modalities, and feature interaction can be performed based on the multi-scale extraction features to realize the construction of complementary data bodies, and high-dimensional interaction features of multimodal remote sensing images can be obtained by constructing a Mamba interaction module, and multi-dimensional fusion features of multimodal remote sensing images can be obtained by constructing an offset-guided fusion module, thereby generating a final target detection result of the detection target, fully considering the offset problem between multimodal remote sensing images, and using the constructed Mamba interaction module to realize the calculation of multimodal remote sensing images, so as to achieve the effect of complementary information bodies between modalities, thereby improving the overall characteristic information volume. In addition, the embodiment of the present application also starts from how to weaken the offset effect between multimodal remote sensing images to ensure that the rich information of multimodal remote sensing images can be maximized in the fusion process, which not only improves the accuracy of multimodal target detection, but also significantly enhances the robustness of the model. This solves the problem in the related art that, due to data offset from different sensors, there will be differences between the position and features of the object, making it difficult for the detection model to accurately align the information in each modality, thereby reducing the accuracy and overall performance of the detection model.

[0159] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0160] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0161] When the processor 402 executes the program, the target detection method for the multimodal remote sensing image provided in the above embodiment is implemented.

[0162] Furthermore, the electronic device further comprises:

[0163] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0164] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0165] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0166] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0167] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0168] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned target detection method for multimodal remote sensing images.

[0170] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method for multimodal remote sensing images.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0172] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0173] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0174] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0175] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0176] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0178] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for target detection in multimodal remote sensing images, characterized in that: The following steps are involved: Based on a first modality remote sensing image and a second modality remote sensing image of the detection target, obtaining a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image; Based on the first multi-scale extraction features and the second multi-scale extraction features, respectively obtaining a first scale extraction feature of the first multi-scale extraction features and a second scale extraction feature of the second multi-scale extraction features; Inputting the first scale extracted features and the second scale extracted features into a pre-built Mamba interaction module respectively to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image; The first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature are respectively input into a pre-built offset guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and a first target detection result and a second target detection result of the detection target are generated based on the multi-dimensional fusion features.

2. The method according to claim 1, characterized in that The method of acquiring a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image based on the first modality remote sensing image and the second modality remote sensing image of the detected target includes: The first modality remote sensing image and the second modality remote sensing image are input into a target convolutional neural network to extract the first multi-scale extraction features and the second multi-scale extraction features.

3. The method according to claim 1, characterized in that The step of inputting the first scale extracted features and the second scale extracted features into a pre-built Mamba interaction module to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image comprises: Performing a global scan on the first scale extracted features and the second scale extracted features respectively to obtain first modal discrete data of the first scale extracted features and second modal discrete data of the second scale extracted features; Inputting the first modal discrete data and the second modal discrete data into a pre-built single Mamba module to obtain first modal discrete extracted features of the first modal discrete data and second modal discrete extracted features of the second modal discrete data; Combining the first modal discrete extraction feature, the second modal discrete extraction feature, local scanning, and a pre-constructed cross Mamba module, obtaining a first modal discrete significant feature of the first modal discrete extraction feature and a second modal discrete significant feature of the second modal discrete extraction feature; The first modal discrete salient features and the second modal discrete salient features are reconstructed to obtain the first high-dimensional interaction features and the second high-dimensional interaction features.

4. The method according to claim 3, characterized in that The calculation process of the cross Mamba module is: in, It is the multimodal input of the cross-Mamba module. CS6 is the core calculation method of the cross-Mamba module. i It is a cross-scan mechanism used to serialize two-dimensional features and realize the interaction of two-modal information. i It is a reverse scanning mechanism used to reconstruct the serialized fusion features into two dimensions.

5. The method according to claim 1, characterized in that The step of inputting the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature, and the second high-dimensional interaction feature into a pre-built offset guided fusion module to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image includes: The first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature are fused using a pre-built dual-branch feature fusion model to obtain the multi-dimensional fusion feature.

6. The method according to claim 5, characterized in that The calculation process of the multi-dimensional fusion feature is as follows: Among them, x is the input feature, ConvBlock is the convolution channel residual preservation block, and RepBlock is the channel reconstruction block.

7. A target detection device for multimodal remote sensing images, characterized in that: include: An extraction module, configured to obtain a first multi-scale extraction feature of the first modality remote sensing image and a second multi-scale extraction feature of the second modality remote sensing image based on the first modality remote sensing image and the second modality remote sensing image of the detected target; an acquisition module, configured to respectively acquire a first scale extraction feature of the first multi-scale extraction feature and a second scale extraction feature of the second multi-scale extraction feature based on the first multi-scale extraction feature and the second multi-scale extraction feature; An input module, used for inputting the first scale extraction feature and the second scale extraction feature into a pre-built Mamba interaction module respectively, so as to obtain a first high-dimensional interaction feature of the first modality remote sensing image and a second high-dimensional interaction feature of the second modality remote sensing image; A generation module is used to input the first multi-scale extraction feature, the second multi-scale extraction feature, the first high-dimensional interaction feature and the second high-dimensional interaction feature into a pre-built offset guided fusion module, respectively, to obtain multi-dimensional fusion features of the first modality remote sensing image and the second modality remote sensing image, and generate a first target detection result and a second target detection result of the detection target based on the multi-dimensional fusion features.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method for a multimodal remote sensing image as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the target detection method for a multimodal remote sensing image as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the target detection method for multimodal remote sensing images as described in any one of claims 1 to 6.

Images