Multi-modal target detection method and device based on feature enhancement and cooperative interaction

Abstract

The application provides a kind of multi-modal target detection method and device based on feature enhancement and collaborative interaction, it is related to image processing technical field, method includes: obtaining at least two modal remote sensing images;Remote sensing image is input into remote sensing target detection model, and remote sensing target detection result output by remote sensing target detection model is obtained;Wherein, remote sensing target detection model is based on the at least two modal remote sensing training image obtained and based on each remote sensing training image corresponding label is combined and obtained The training of combined label;Remote sensing target detection model is used for carrying out feature enhancement to the input at least two modal remote sensing image respectively, and feature fusion is carried out to the feature after feature enhancement, to carry out remote sensing target detection based on the feature after fusion and feature enhancement.The application realizes the multi-modal feature collaborative fusion while reducing the introduction of redundant information, improves the detection precision in complex environment and target scale change scene.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a multimodal target detection method and apparatus based on feature enhancement and collaborative interaction. Background Technology

[0002] The continuous development of sensor technology has provided more high-precision, low-cost, miniaturized, and low-power solutions for various application fields. Target detection that integrates data from multiple sensors, or multimodal target detection, has become a new trend in remote sensing image target detection. By utilizing the complementarity between images from different sensors, multimodal data-based target detection can acquire target information more comprehensively, solve the problems of noise and outliers in single-modal images, and improve the algorithm's anti-interference ability.

[0003] Depending on the fusion strategy, existing remote sensing detection methods based on multimodal images can be divided into pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion directly fuses the original images at the input level, maintaining the spectral fidelity of the original data, improving spatial resolution, and enhancing edge and texture details. Feature-level fusion transforms different modal data into a set of basic features in the same feature space and fuses them through cascaded fusion methods, such as addition or stitching, enhancing the representational power of the features. Decision-level fusion first involves each sensor processing its own data and making decisions, then using methods such as voting and statistics to fuse the individual decisions, exhibiting strong anti-interference capabilities.

[0004] However, pixel-level fusion introduces uncertainty in target location due to slight differences in the poses of different sensors during image acquisition and potential pixel misalignment between images. Feature-level fusion, on the other hand, is prone to introducing redundant information due to the large differences between images of different modalities, which contaminates the extracted original features and reduces the robustness of the algorithm. Decision-level fusion is much more computationally intensive than single-modal or feature-level fusion, and inappropriate decision-making algorithms may cause data to deviate from reality. Summary of the Invention

[0005] This invention provides a multimodal target detection method and apparatus based on feature enhancement and collaborative interaction, which addresses the shortcomings of existing technologies such as data misalignment, redundant information, and excessive computation. While leveraging the complementary advantages of different modal features, it reduces the impact of redundant information on the fusion of complementary information, thereby achieving stable and accurate multimodal remote sensing image target detection adaptable to various extreme scenarios.

[0006] This invention provides a multimodal target detection method based on feature enhancement and collaborative interaction, comprising: acquiring remote sensing images of at least two modalities; inputting the remote sensing images into a remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein, the remote sensing target detection model is trained based on the acquired remote sensing training images of at least two modalities and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the enhanced features, so as to perform remote sensing target detection based on the fused features and the enhanced features.

[0007] According to the present invention, a multimodal target detection method based on feature enhancement and collaborative interaction is provided. The remote sensing target detection model includes: a feature extraction layer, which extracts features from at least two input remote sensing images to obtain extracted features for the corresponding remote sensing images; a feature enhancement layer, which enhances each extracted feature to obtain enhanced features for each remote sensing image; a feature fusion layer, which fuses the enhanced features to obtain fused features; and a target detection layer, which performs remote sensing target detection on each enhanced feature and the fused feature, and combines non-maximum suppression to obtain the remote sensing target detection result.

[0008] According to the present invention, a multimodal target detection method based on feature enhancement and collaborative interaction performs remote sensing target detection on various enhanced features and fused features, and combines non-maximum suppression to obtain remote sensing target detection results. The method includes: upsampling deep features in each enhanced feature, and according to a first preset stitching order, stitching the upsampled features with shallow features in the corresponding enhanced features to obtain enhanced stitched features corresponding to each enhanced feature; performing remote sensing target detection on the enhanced stitched features corresponding to each enhanced feature to obtain corresponding first target detection results; upsampling deep features in the fused features, and according to the first preset stitching order, stitching the upsampled features with shallow features in the fused features to obtain fused stitched features corresponding to the fused features; performing remote sensing target detection on the fused stitched features to obtain second target detection results; stitching all first target detection results and second target detection results according to a second preset stitching order to obtain a stitched result; and using non-maximum suppression based on the stitched result to obtain the remote sensing target detection result.

[0009] According to the multimodal target detection method based on feature enhancement and collaborative interaction provided by the present invention, feature enhancement is performed on each extracted feature to obtain enhanced features corresponding to each remote sensing image. The method includes: for the extracted features of a single remote sensing image, using a global channel attention mechanism to modulate the high-level semantic features in the extracted features from top to bottom to obtain deep features, and using a point-by-point channel attention mechanism to modulate the low-level detail features of the extracted features from bottom to top to obtain shallow features; and obtaining the enhanced features of the corresponding remote sensing image based on the deep features and shallow features.

[0010] According to the multimodal target detection method based on feature enhancement and collaborative interaction provided by the present invention, feature fusion is performed on each enhancement feature to obtain fused features, including: for each individual enhancement feature, using a multi-scale convolutional attention (MSCA) module to obtain corresponding key features; and obtaining fused features based on the attention weights of each key feature and the other enhancement features corresponding to the previously obtained key features.

[0011] According to the present invention, a multimodal target detection method based on feature enhancement and collaborative interaction is provided to train a remote sensing target detection model, comprising: acquiring remote sensing training images of at least two modalities and labels corresponding to each remote sensing training image; merging the labels corresponding to each remote sensing training image and deleting duplicate information in the merged labels to obtain merged labels; inputting the acquired remote sensing training images of at least two modalities into the model to be trained to obtain a first target training result corresponding to each modal remote sensing training image and a second target training result obtained after feature fusion of the remote sensing training images of at least two modalities; constructing a first loss function corresponding to each remote sensing training image based on each first target training result and the labels of the remote sensing training images corresponding to the first target training results; constructing a second loss function based on the second target training result and the merged labels; obtaining a total loss function based on the first loss function and the second loss function of all remote sensing training images, and converging based on the total loss function to end the training.

[0012] According to the multimodal target detection method based on feature enhancement and collaborative interaction provided by the present invention, the first target training results include bounding box training results, confidence training results, classification training results, and angle classification results. The labels of the remote sensing training images include bounding box ground truth, confidence ground truth, classification labels, and angle classification labels. Based on each first target training result and the labels of the corresponding remote sensing training images, a first loss function is constructed for each remote sensing training image, including: constructing a bounding box regression loss using CIoU based on the bounding box training results and the bounding box ground truth; constructing a confidence loss using smoothed binary cross-entropy based on the confidence training results and the confidence ground truth; constructing a classification loss using smoothed binary cross-entropy based on the classification training results and the classification labels; constructing an angle loss using CSL circular smoothed labels and smoothed binary cross-entropy based on the angle classification results and the angle classification labels; and obtaining the first loss function for the corresponding remote sensing training image based on the bounding box regression loss, confidence loss, classification loss, and angle loss.

[0013] The present invention also provides a multimodal target detection device based on feature enhancement and collaborative interaction, comprising: an image acquisition unit for acquiring remote sensing images of at least two modalities; and a remote sensing target detection unit for inputting the remote sensing images into a remote sensing target detection model to obtain remote sensing target detection results output by the remote sensing target detection model; wherein the remote sensing target detection model is trained based on the acquired remote sensing training images of at least two modalities and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the feature-enhanced features, so as to perform remote sensing target detection based on the fused features and the feature-enhanced features.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described multimodal target detection methods based on feature enhancement and collaborative interaction.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described multimodal target detection methods based on feature enhancement and cooperative interaction.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described multimodal target detection methods based on feature enhancement and collaborative interaction.

[0017] The present invention provides a multimodal target detection method and apparatus based on feature enhancement and collaborative interaction. By using a remote sensing target detection model to enhance the features of at least two modalities of remote sensing images, the method strengthens the semantic features of small targets while effectively aggregating spatial and channel detail information of the targets. This solves the problem of feature loss of small targets caused by network deepening or the use of only unidirectional cross-layer fusion. By performing feature fusion on the enhanced features, the method achieves collaborative fusion of multimodal features while reducing the introduction of redundant information, thereby improving the detection accuracy in complex environments and scenarios with varying target scales, while ensuring high inference speed. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the multimodal target detection method based on feature enhancement and collaborative interaction provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the architecture of the remote sensing target detection model provided by the present invention;

[0021] Figure 3 This is a schematic diagram of the feature enhancement process for extracted features provided by the present invention;

[0022] Figure 4 This is a schematic diagram of the process of feature fusion of various enhancement features provided by the present invention;

[0023] Figure 5 This is a feature visualization diagram provided by the present invention;

[0024] Figure 6 This is a schematic diagram illustrating the detection effects of different fusion strategies provided by the present invention;

[0025] Figure 7 This is a schematic diagram of the structure of the multimodal target detection device based on feature enhancement and collaborative interaction provided by the present invention;

[0026] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Figure 1 The diagram illustrates a flowchart of a multimodal target detection method based on feature enhancement and collaborative interaction according to the present invention. The method includes:

[0029] S11, acquire remote sensing images in at least two modalities;

[0030] S12, input the remote sensing image into the remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein, the remote sensing target detection model is trained based on at least two modalities of remote sensing training images and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the feature-enhanced features, so as to perform remote sensing target detection based on the fused features and the feature-enhanced features.

[0031] It should be noted that S1N in this specification does not represent the order of multimodal target detection methods based on feature enhancement and cooperative interaction. The following will explain this in detail. Figures 2-6 This invention describes a multimodal target detection method based on feature enhancement and collaborative interaction.

[0032] Step S11: Acquire remote sensing images in at least two modalities.

[0033] It should be noted that remote sensing images include visible light images, infrared images, and radar / laser images, etc. The following explanation uses visible light and infrared images as examples to illustrate methods for multimodal target detection. This article does not further limit the acquisition of specific modalities of remote sensing images. Referring to Table 1, the characteristics and applicable scenarios of three typical modalities of images are shown. Under sufficient lighting conditions, visible light images can effectively capture information such as the color, shape, and size of the target's spatial distribution, clearly capturing the target's texture features and details. However, under insufficient lighting conditions, the accuracy is greatly reduced, and it is easily affected by the environment, unable to penetrate clouds, fog, or obstructed areas. Infrared images can solve the problems of visible light images. Based on the difference in infrared radiation brightness between the target and its surrounding environment, it senses the target by measuring the target's infrared radiation capability. It can still perform relatively accurate detection and positioning under low-light conditions such as nighttime and rainy days. However, infrared images are not sensitive to changes in target shape and cannot capture the target's color and outline information. In addition, due to heat crosstalk in infrared imaging equipment, false objects resembling real objects may appear in locations where there are no targets. Similarly, radar laser imaging determines the position and shape of a target by calculating the time difference and intensity of reflected signals. It can obtain high-precision measurement information under extreme conditions such as low visibility and low illumination, but it suffers from problems such as high equipment cost and long data acquisition time.

[0034] Table 1. Characteristics and applicable scenarios of different modal images

[0035] Visible light images √ √ Infrared images √ √ Radar laser images √ √

[0036] Step S12: Input the remote sensing image into the remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein, the remote sensing target detection model is trained based on at least two modalities of remote sensing training images and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the feature-enhanced features, so as to perform remote sensing target detection based on the fused features and the feature-enhanced features.

[0037] In this embodiment, reference Figure 2The remote sensing target detection model includes: a feature extraction layer, which extracts features from at least two modalities of the input remote sensing images to obtain extracted features for the corresponding remote sensing images; a feature enhancement layer, which enhances each extracted feature to obtain enhanced features for each remote sensing image; a feature fusion layer, which fuses the enhanced features to obtain fused features; and a target detection layer, which performs remote sensing target detection on each enhanced feature and the fused feature, and combines non-maximum suppression to obtain the remote sensing target detection result. It should be noted that when the acquired at least two modalities of remote sensing images are visible light images and infrared images, the acquired visible light images and infrared images are respectively fed into different branches of the feature extraction layer for feature extraction. It should also be noted that in this embodiment, the backbone network includes a feature extraction layer and a feature enhancement layer, and CSPDarkNet53 can be used.

[0038] Specifically, firstly, a feature extraction layer extracts features from at least two modalities of the input remote sensing images to obtain the extracted features corresponding to the remote sensing images. It should be noted that the extracted features include high-level semantic features and low-level detail features. High-level semantic features are used to characterize the features extracted by deep convolution, while low-level detail features are used to characterize the features extracted by shallow convolution.

[0039] Secondly, the feature enhancement layer enhances each extracted feature to obtain the enhanced features for each remote sensing image. It should be noted that, to improve the model's ability to detect targets at small scales due to shooting distance, it is necessary to fuse high-level semantic features and low-level detail features. Therefore, to simultaneously utilize top-down global channel attention and bottom-up point-by-point channel attention mechanisms to exchange multi-scale context and obtain richer semantic information and spatial detail encoding, an Asymmetric Contextual Modulation (ACM) module is introduced to bidirectionally enhance features. By using top-down global attention and bottom-up local attention to exchange multi-scale context for cross-layer feature fusion, the spatial and channel detail information of the target is effectively gathered while strengthening the semantic features of small targets, thus solving the problem of small target feature loss caused by deepening the network or using only unidirectional cross-layer fusion.

[0040] In this embodiment, reference Figure 3Each extracted feature is enhanced to obtain the enhanced features for each remote sensing image. This includes: for the extracted features of a single remote sensing image, using a global channel attention mechanism to modulate the high-level semantic features in the extracted features from top to bottom to obtain deep features, and using a point-by-point channel attention mechanism to modulate the low-level detail features of the extracted features from bottom to top to obtain shallow features; and based on the deep features and shallow features, the enhanced features of the corresponding remote sensing image are obtained.

[0041] Specifically, when obtaining deep features through top-down modulation, a global average pooling layer can be used to obtain features of size 1×1×D (only the feature size is changed, not the number of channels). Deep features are represented as:

[0042]

[0043] Among them, X enhance Y represents deep features; X represents high-level semantic features; σ represents the ReLU activation function; W1 and W2 are two fully connected layers; y represents the global context features obtained through global average pooling. This represents element-wise multiplication; This indicates element-wise addition.

[0044] Furthermore, when obtaining shallow features through bottom-up modulation, features of size H×W×1 can be obtained through pointwise convolution (changing only the number of channels, not the feature size). Shallow features are represented as:

[0045]

[0046] Among them, Y enhance PWConv represents shallow features; it is used to highlight small targets point by point. This represents element-wise multiplication.

[0047] It should be added that the point-to-point channel attention mechanism can aggregate the channel feature context information of each spatial location individually. In contrast to top-down modulation, this modulation path propagates context information in a bottom-up manner, thereby enriching high-level semantic features with spatial information of low-level detail features.

[0048] Furthermore, based on the deep and shallow features, the enhancement features of the corresponding remote sensing image are obtained, including: adding the deep and shallow features element-wise to obtain the enhancement features. In this embodiment, the enhancement feature Z is represented as:

[0049]

[0050] It should be noted that the enhanced features Z of the corresponding visible light image can be obtained by referring to the above method for both visible light and infrared images.vis And the corresponding enhanced feature Z of the infrared image inf .

[0051] Secondly, refer to Figure 4 The feature fusion layer fuses the various enhanced features to obtain the fused features. It should be noted that, to leverage the advantages of visible light images in capturing texture and contours, and the advantage of infrared images being unaffected by lighting conditions, while reducing redundant information introduced by differences in shooting pose, a Lightweight Multi-branch Fusion (ALMF) module based on an attention mechanism is introduced. This module uses lightweight multi-branch strip-shaped deep convolutions to obtain feature aggregation weights for the corresponding modalities, achieving multi-scale cross-fusion of features from the two modalities. The ALMF module's attention mechanism adaptively and selectively focuses on important spatial regions or channels based on the features of the input data, improving the model's ability to perceive key features containing the target, thereby suppressing noise and interference.

[0052] Specifically, the enhanced features are fused to obtain fused features, including: for each individual enhanced feature, the corresponding key feature is obtained by using the multi-scale convolutional attention (MSCA) module; and the fused features are obtained based on the attention weights of each key feature and the other enhanced features corresponding to the previously obtained key features.

[0053] It should be noted that when obtaining the attention weights corresponding to different modal remote sensing images, a traditional attention mechanism was not used. Instead, a lightweight multi-scale convolutional attention (MSCA) module was used. The MSCA module focuses on the key features of infrared and visible light from both spatial and channel perspectives, so as to effectively reduce the number of model parameters and computational cost while ensuring model performance.

[0054] Furthermore, the acquired remote sensing images of at least two modalities include visible light images and infrared images. Based on the attention weights of each key feature and the remaining enhancement features corresponding to the previously acquired key features, a fusion feature is obtained, including: element-wise multiplying the attention weights of the key features obtained based on the visible light image enhancement features and the infrared image enhancement features to obtain a first feature; element-wise multiplying the attention weights of the key features obtained based on the infrared image enhancement features and the visible light image enhancement features to obtain a second feature; and element-wise adding the first feature and the second feature to obtain the fusion feature. (Reference) Figure 5 Wherein, (a) represents remote sensing images of different modalities, (b) represents extracted features, (c) represents reweighted features corresponding to the first and second features mentioned above, and (d) represents fused features.

[0055] It should be noted that the fusion feature is represented as follows:

[0056]

[0057] Where Out represents the fusion feature; Z vis Z represents the enhancement features corresponding to the visible light image; inf Indicates the enhancement features of the corresponding infrared image; A vis Attention weights representing the enhancement features of a visible light image; A inf Attention weights representing the enhanced features of an infrared image; This represents element-wise multiplication; This indicates element-wise addition.

[0058] In an optional embodiment, before obtaining the fused features based on the attention weights of each key feature and the other enhancement features corresponding to the previously acquired key features, the method includes: obtaining the attention weights corresponding to the enhancement features of different modal remote sensing images.

[0059] It should be noted that continued reference is necessary. Figure 4 The MSCA module consists of three parts: (1) Depth-separable convolution: used to aggregate local information. By performing separate spatial convolution and pointwise convolution on each channel, the contextual information around each location is captured. On the one hand, the computational cost and parameter quantity of the model can be reduced, and on the other hand, the receptive field of the features can be improved; (2) Multi-branch depth-separable strip convolution: used to capture multi-scale contextual information. By using convolution kernels of different sizes, feature representations of different scales are obtained and the correlation between them is calculated. Using strip convolution has two advantages: on the one hand, it can reduce the computational cost. Only a pair of 1×7 and 7×1 convolutions are needed to simulate the standard 2D convolution with a kernel size of 7×7; on the other hand, in remote sensing datasets, there are some strip-shaped targets, such as trucks and vans in the DroneVehicle dataset and ships in the VEDAI dataset. Therefore, strip convolution can be used as a supplement to grid convolution to help extract strip-shaped features; (3) 1×1 convolution: used to model the relationship between different channels. By performing 1×1 convolution on multi-scale features, they are mapped to a new channel space to obtain attention weights.

[0060] In this embodiment, the attention weights of the visible light image enhancement features and the attention images of the infrared image enhancement features are respectively represented as follows:

[0061]

[0062]

[0063] Among them, A visAttention weights representing the enhancement features of a visible light image; A inf The attention weights representing the enhanced features of an infrared image; Z vis Z represents the enhancement features corresponding to the visible light image; inf Represents the enhancement features corresponding to the infrared image; DW-Conv represents depthwise separable convolution; Scale i Let represent the i-th branch of the multi-branch depthwise separable strip convolution of the MSCA module. Scale0 denotes identity connection. In practical design, for each branch, two depthwise strip convolutions can be used to approximate a standard depthwise convolution with a large kernel. For visible light and infrared images, the multi-branch depthwise separable strip convolution of the MSCA module can use three branches, with the kernel sizes of each branch set to 7, 11, and 21, respectively.

[0064] It should be added that by using convolution to obtain the attention weights for the corresponding enhanced features, the number of model parameters and computational cost are significantly reduced. By cross-fusing the enhanced features of visible light images and infrared images, the advantages of visible light images in capturing texture and contours and the advantages of infrared images in being unaffected by lighting conditions are simultaneously leveraged. In addition, replacing the standard convolution with two strip convolutions helps to extract strip-shaped features, such as those of a long, narrow car.

[0065] Finally, in the target detection layer, remote sensing target detection is performed on each enhanced feature and fused feature, and non-maximum suppression is combined to obtain the remote sensing target detection result. In this embodiment, performing remote sensing target detection on each enhanced feature and fused feature, and combining non-maximum suppression to obtain the remote sensing target detection result, includes: upsampling the deep features in each enhanced feature, and according to a first preset stitching order, stitching the upsampled features with the shallow features in the corresponding enhanced features to obtain the enhanced stitched features corresponding to each enhanced feature; performing remote sensing target detection on the enhanced stitched features corresponding to each enhanced feature to obtain the corresponding first target detection result; upsampling the deep features in the fused feature, and according to the first preset stitching order, stitching the upsampled features with the shallow features in the fused feature to obtain the fused stitched feature corresponding to the fused feature; performing remote sensing target detection on the fused stitched features to obtain the second target detection result; stitching all the first target detection results and the second target detection results according to the second preset stitching order to obtain the stitched result; and using non-maximum suppression based on the stitched result to obtain the remote sensing target detection result.

[0066] Specifically, based on the stitching results, non-maximum suppression (NMS) is used to obtain the remote sensing target detection results, including: Step 1: Based on the stitching results, all bounding boxes are classified according to category; Step 2: The bounding boxes in each category are sorted in descending order of confidence level; Step 3: The bounding box with the highest confidence level (bbox1) in a certain category is selected, and then removed from the list of all bounding boxes in that category and added to the output list; Step 4: The remaining bounding boxes in the list of bounding boxes of that category are calculated sequentially. Step 5: Calculate IOU using ox; Step 6: Compare the IOU with the NMS preset threshold Thre. If the IOU of a certain bounding box with bounding box 1 is greater than Thre, remove the bounding box from the bounding box list of that category, i.e., remove redundant bounding boxes; Step 7: Repeat steps 8 to 9 until the bounding box list of all objects in that category is empty. At this point, the traversal of one object category is complete; Step 8: Repeat steps 9 to 9 to complete the NMS post-processing process for all object categories in sequence; Step 9: The output list is the desired detection box, and the NMS process ends.

[0067] It should be noted that the target detection layer sets up a multi-scale detection head for the enhancement feature branch and fusion feature branch of at least two remote sensing images, which facilitates decision fusion of the first target detection results and second target detection results of different branches. It also facilitates comparison of the results of single modality and feature fusion based on the output results of the three multi-scale detection heads and the remote sensing target detection results obtained by combining the output results of the three multi-scale detection heads with non-maximum suppression, so as to avoid the influence of single errors on the experimental results.

[0068] It should be noted that the first preset concatenation order can be either concatenating the upsampled features after the shallow features or concatenating the upsampled features before the shallow features; no further limitation is made here. Furthermore, through multi-scale fusion, high-level features with low resolution and high semantic information and low-level features with high resolution and low semantic information are concatenated from top to bottom, ensuring that features at all scales possess rich semantic information.

[0069] In one alternative embodiment, before inputting the remote sensing image into the remote sensing target detection model, the method includes: training the remote sensing target detection model. Specifically, training the remote sensing target detection model includes:

[0070] SA1, acquire remote sensing training images of at least two modalities and the corresponding labels for each remote sensing training image.

[0071] In the actual training process, corresponding remote sensing training images and their labels can be obtained from the DroneVehicle and VEDAI datasets. DroneVehicle is a large-scale drone aerial remote sensing dataset used for tasks such as vehicle detection and vehicle counting. It contains 28,439 pairs of visible-infrared images, including 5 categories and 953,087 instances; the regional scenes are divided into urban roads, highways, and residential areas; the lighting conditions are divided into night, evening, and daytime; the shooting heights are divided into 80 meters, 100 meters, and 120 meters; the shooting angles are divided into 15°, 30°, and 45°; and the image resolution is 840x712. In addition, considering the different orientations of objects in the aerial images, oriented bounding boxes (OBBs) are used for the labels to more accurately and compactly represent the object outlines.

[0072] The VEDAI dataset was cropped from the AGRC dataset and contains 1246 pairs of visible-infrared images across 11 categories. The scenes include grasslands, highways, and urban areas, all under daytime lighting conditions, captured from a 90° vertical angle, with image resolutions of 1024×1024 or 512×512. This embodiment uses the 1024×1024 resolution. Labels are represented by horizontal bounding boxes. Classes with fewer than 50 instances were ignored in the experiments. Therefore, only the following eight categories were retained: Car, Pickup, Camping, Truck, Other, Tractor, Boat, and Van. The dataset partitioning referenced SuperYolo, and its fold01test was used for testing.

[0073] Table 2. Comparison of DroneVehicle and VEDAI datasets

[0074]

[0075] SA2 merges the labels corresponding to each remote sensing training image and removes duplicate information from the merged labels to obtain merged labels. It should be noted that in the DroneVehicle dataset, the labels for images of different modalities are not aligned (e.g., in nighttime conditions, due to lack of illumination, only infrared images are labeled; in daytime conditions, due to insufficient detail in infrared images, some very small targets are only labeled in visible light images). To verify the robustness of the proposed method, the union of the bounding boxes of each pair of images is taken. For adjacent boxes with an IoU greater than 0.5, the larger bounding box is retained to remove redundant boxes. By merging the labels of remote sensing training images of different modalities, label misalignment is avoided.

[0076] In one possible implementation, after acquiring remote sensing training images of at least two modalities, the method includes: performing random color distortion and inversion stitching on each of the remote sensing training images to improve data diversity.

[0077] SA3 uses at least two modalities of remote sensing training images as input data for training, and uses the labels and merged labels corresponding to each remote sensing training image as labels for training, to train the model to obtain a remote sensing target detection model for detecting multimodal remote sensing images.

[0078] It should be noted that the network to be trained can be an existing network built into the training device. This existing network usually includes a network structure, or it can be another network specified by the user. The network to be trained usually includes a feature extraction layer for extracting features, a feature enhancement layer for enhancing the extracted features, a feature fusion layer for feature fusion based on the enhanced features, a target detection layer for remote sensing target detection based on the enhanced features and fused features, and a loss function. According to the preset iteration rules, the above remote sensing training images are input into the model to be trained for training to obtain the trained target detection model.

[0079] In this embodiment, training the model to be trained includes: inputting at least two modalities of remote sensing training images into the model to be trained to obtain a first target training result corresponding to each modality of remote sensing training image and a second target training result obtained by fusing features of at least two modalities of remote sensing training images; constructing a first loss function corresponding to each remote sensing training image based on each first target training result and the label of the remote sensing training image corresponding to the first target training result; constructing a second loss function based on the second target training result and the merged label; obtaining a total loss function based on the first loss function and the second loss function of all remote sensing training images, and converging based on the total loss function to end the training.

[0080] Furthermore, the training results for the first objective include bounding box training results, confidence score training results, classification training results, and angle classification results. The labels for the remote sensing training images include ground truth bounding box values, ground truth confidence scores, classification labels, and angle classification labels. Accordingly, based on each first objective training result and the labels of the corresponding remote sensing training images, a first loss function is constructed for each remote sensing training image, including:

[0081] SB1, based on the bounding box training results and ground truth bounding box values, constructs the bounding box regression loss using CIoU. In this embodiment, the bounding box regression loss is expressed as:

[0082]

[0083]

[0084]

[0085] Among them, L box The bounding box regression loss represents the bounding box training result b and the ground truth bounding box value b. gt The intersection-union ratio; ρ represents the training result b of the bounding box and the ground truth b of the bounding box. gt The Euclidean distance between the center points of the two bounding boxes; c represents the training result b of the bounding boxes and the ground truth b of the bounding boxes. gt The diagonal distance of the circumscribed rectangle; v is used to measure the similarity of aspect ratios, w gt h represents the width of the border truth value. gt α represents the length of the border truth value; α represents the weight function.

[0086] SB2 constructs a confidence loss function using smoothed binary cross-entropy, based on the confidence training results and the ground truth confidence values. It should be noted that the confidence loss L... obj The following classification of losses can be referenced, and will not be repeated here.

[0087] SB3, based on the classification training results and classification labels, constructs a classification loss using smoothed binary cross-entropy. In this embodiment, the classification loss is expressed as:

[0088]

[0089]

[0090] Among them, L cls Represents classification loss; x i Indicates the classification training result; y i Indicates category label.

[0091] SB4 constructs an angle loss function based on the angle classification results and angle classification labels, using CSL circular smooth labels and smooth binary cross-entropy.

[0092] It's important to note that the bounding box annotation method needs to be modified according to the shape characteristics of the detected object itself. The more precise the annotation method, the less redundant information is provided to the network during training; the more sufficient the prior information, the fewer learnable schemes the network has, which helps to constrain the training direction and reduce the network's convergence time. Furthermore, when target objects are too densely packed, precise annotation can prevent detected targets from being filtered out by NMS.

[0093] Oriented bounding boxes (OBBs) are a commonly used annotation method that incorporates angle information to make the bounding box fit the target more closely. However, because the learned target parameters are periodic, the loss value increases sharply at the periodically changing boundaries, increasing the learning difficulty of the network. Therefore, to avoid the boundary problems caused by rotating the target annotation method, Circular Smooth Labels (CSL) are used to transform the regression problem into a classification problem. The boundary problem is solved by directly discretizing the continuous problem, and then the angle loss is calculated using Smooth Binary Cross-Entropy (SmoothBCE). The angle loss is specifically represented as follows:

[0094]

[0095]

[0096]

[0097] Among them, L theta Indicates angle loss; g(angle) i () represents the window function, r represents the window radius, and window functions have periodicity, symmetry, monotonicity, and finiteness. Common window functions include impulse functions, rectangular functions, trigonometric functions, and Gaussian functions. The Gaussian function, which yields the best results, can be chosen as the window function; θ represents the acute angle between the box and the x-axis; angle i Indicates the angle classification result; angle i gt Indicates angle classification labels.

[0098] SB5, based on the bounding box regression loss, confidence loss, classification loss, and angle loss, obtains the first loss function for the corresponding remote sensing training image. In this embodiment, the first loss function L is expressed as:

[0099] L = L box +L obj +L cls +L theta

[0100] It should be added that the order of steps SB1-SB4 above does not matter, as long as they are performed before step SB5.

[0101] In addition, the second loss function can be referred to the first loss function mentioned above, and will not be explained further here.

[0102] Furthermore, the total loss function is expressed as:

[0103] L total =αL vis +βLinf +γL fusion

[0104] Among them, L total L represents the total loss function; vis and L inf L represents the first loss function corresponding to different modal remote sensing images; fusion α represents the second loss function; α, β and γ represent weight parameters, which can all be set to 1 to ensure that the weights of each mode are the same.

[0105] It should be noted that in the actual training process, a Stochastic Gradient Descent (SGD) optimizer was used with an initial learning rate of 1e-2, which decayed to 2e-3 as training progressed. The DroneVehicle dataset had 200 training epochs, and the Vedai dataset had 400, with a batch size of 16 for both. To verify the effectiveness of the method, it was compared with other mainstream object detection methods, including one-stage FCOS, two-stage Faster R-CNN, and Mask R-CNN, using the open-source mmdetection library for comparison. To facilitate evaluation and ensure experimental fairness, HBB was used to calculate mAP during testing. All code was implemented in PyTorch and run on a machine configured with two NVIDIA GPUs. The experiment was conducted on an RTX 3090 server with Ubuntu 18.04 operating system and CUDA version 12.0.

[0106] In the above validation process, the mean average precision (mAP) was mainly used to measure the detection performance of different methods. The formulas for calculating precision and recall are as follows:

[0107] AP=∫P(R)dR

[0108]

[0109] Here, AP represents the average precision, which is the area under the PR curve enclosed by the coordinates. The closer the AP value is to 1, the better the detection performance of the algorithm. mAP represents the average AP for each class, used to measure the performance of multi-class object detection tasks. k represents the number of classes for which AP is applied. A predicted bounding box is considered a true positive (TP) when the IoU between it and its nearest labeled bounding box is greater than 0.5.

[0110] In an alternative embodiment, the following ablation experiments were performed on the DroneVehicle dataset to verify the effectiveness of the ACM and ALMF modules.

[0111] For the ACM module: The backbone of the Yolov5 network is used to obtain feature information at different scales and depths. In experiments, the ACM module is added to different modalities and locations to verify its effectiveness and improve its fit in the proposed method.

[0112] Taking infrared modality as an example, an ACM module is introduced. In the Yolov5 configuration file, each module is distinguished and connected by layers. To find the most suitable location to add the ACM module, one or more ACM modules are added to different layers of the feature extraction network. Experiments are conducted to verify the results from two perspectives: 1) Number of enhancements: ACM modules are used to enhance features on one or more pairs of adjacent layer features; 2) Feature distance: ACM modules are used to enhance features on one pair of adjacent or alternating features.

[0113] Table 3. Comparison of ACM module performance at different fusion locations (infrared branch)

[0114]

[0115] Table 3 shows a comparison of the ACM module performance at different fusion positions. The enhancement schemes can be categorized into adjacent single-pass (24th layer), adjacent multiple-pass (24th layer + 46th layer + 68th layer), and alternating single-pass (28th layer). Taking adjacent single-pass as an example, it indicates that layer 2 is used as a shallow feature layer, layer 4 is used as a deep feature layer input to the ACM module, and the result is fed back into the network as the output of layer 4. The results show that single-pass feature enhancement is more effective than multiple-pass enhancement, possibly because multiple interpolations introduce more redundant information. The mAP of features with large distances and adjacent-layer fusion is basically consistent. This is because when the feature map sizes differ significantly, interpolating the smaller feature map will miss more detailed information and introduce a small amount of redundant information.

[0116] Based on the above experiments, in order to verify whether ACM has the same enhancement effect on visible light images, the same processing was performed on the visible light branches. The experiment selected an alternating single-pass enhancement strategy.

[0117] Table 4. Comparison of ACM module performance in different modalities (single phase 28) th layer)

[0118]

[0119] Table 4 presents the results of using the ACM module to enhance different modalities. It can be seen that the ACM module also improves the performance of the visible light branch, increasing mAP by 0.7%. Furthermore, this experiment compared the effects of decision fusion, finding that using the detection results of both modalities simultaneously can improve mAP by 0.7%, further demonstrating the complementarity of the features of the two modalities.

[0120] For the ALMF module: Yolov5 outputs three feature maps at different scales for detection, which aligns with the design of this method using a multi-scale detection head. In the experiments, the ALMF module was applied to the feature maps at the three scales to obtain the fused multi-scale features. To verify the effectiveness of the ALMF module, a cascaded feature fusion method was selected for comparative experiments, including three strategies: concatenation (Cat), addition (Add), and maximum value (Max). In addition, an SE module was added to enhance the correlation between feature channels. In the experiments, the feature maps at the three scales were fused using the above strategies at the same location. Table 5 shows the performance of different fusion strategies:

[0121] Table 5. Comparison of Different Fusion Strategies

[0122]

[0123]

[0124] Depend on Figure 6 It can be seen that the accuracy of cascaded feature fusion is lower than that of single-modality fusion, where gray rectangles represent missed detections and gray circles represent false detections. For example... Figure 6 Cascaded fusion methods cannot handle noise when targets have similar colors or ghosting effects from infrared thermal imaging, introducing excessive redundant information and thus reducing detection accuracy. In contrast, the ALMF proposed in this embodiment uses an attention mechanism to enhance key information, leveraging the complementarity of fused features while reducing the impact of noise on fusion, achieving an mAP of 76.6%, which is 1.6% higher than the best result achieved with a single modality.

[0125] Furthermore, in this embodiment, a dual-stream multimodal target detection network based on feature enhancement and fusion is used. The ACM module is used to improve the feature representation of small targets, and the ALMF module is used to achieve effective fusion of visible light features and infrared features. Experiments have verified the complementarity of the two modules.

[0126] Table 6. ACM and ALM F Module complementarity experiment

[0127]

[0128] The experimental results are shown in Table 6, where the baseline represents the detection results of the infrared branch, and the ACM module uses adjacent single (24) enhancements. It can be seen that using only the ACM module for feature enhancement improves mAP by 1.2%; using only the ALMF module for feature fusion improves mAP by 1.4%; while introducing both modules simultaneously improves mAP by 2.5%, and precision and recall are also significantly improved. The experiment verifies the complementarity of the two modules, that is, using the ACM module to solve the problem that small target features cannot be preserved as the network deepens, and using the ALMF module to solve the problem of excessive redundant information during fusion, thereby improving the detection accuracy in scenarios with complex backgrounds, dense distribution, and small target size.

[0129] Furthermore, the proposed method is compared with other state-of-the-art methods. Since existing methods such as FSCO, Faster R-CNN, Mask R-CNN, and Yolov5s are designed for a single modality, training is performed on a single modality (visible light or infrared modality). To verify the robustness of the proposed method and its effectiveness compared with mainstream remote sensing target detection methods, experiments were conducted on the DroneVehicle and VEDAI datasets.

[0130] To verify the effectiveness of the proposed method, experiments were first conducted on the DroneVehicle remote sensing dataset. Observations revealed that Freight and Van classes exhibit very similar shapes in this dataset. Therefore, many popular detection methods, when experimenting with the DroneVehicle dataset, merge these two classes into three others to eliminate errors caused by fine-grained classification. However, in our experiments, the original classification was preserved. UA-CMDet is the method proposed in the paper that developed this dataset.

[0131] Table 7. Evaluation results of the DroneVehicle dataset

[0132] FCOS Visible 80.9 48.9 35.2 78.2 31.4 54.9 Faster RCNN Visible 82.5 54.1 40.1 80.1 41.2 59.6 Mask RCNN Visible 82.5 53.9 40.5 80.1 40 59.4 Yolov5s Visible 89.7 62 48.8 86.4 46.8 66.7 FCOS Infrared 87.8 57.9 44.5 81.4 33.2 61 Faster RCNN Infrared 89 65 53.1 83.9 42.1 66.6 Mask RCNN Infrared 89 65.2 51.9 83.8 41.5 66.3 Yolov5s Infrared 94.6 73.1 63.4 89.8 55.4 75 UA-CMDet Visible+Infrared 87.5 46.8 60.7 87.1 38.0 64.0 LAIIFusion Visible+Infrared 94.5 54.4 58.0 90.5 33.9 66.2 Dual-YOLO Visible+Infrared 98.1 65.7 52.9 95.8 46.6 71.8 <![CDATA[C 2 Former-S 2 ANet]]> Visible+Infrared 90.2 68.3 63.7 89.5 58.5 74.2 Proposed Visible+Infrared 95.8 76.9 64.3 91.3 59.3 77.5

[0133] Table 7 categorizes the data into visible light and infrared modes. Yolov5s achieves the highest mAP among single-modal target detection methods, with accuracies of 66.7% and 77.5%, respectively. Therefore, this method selects Yolov5s as the basic framework for its proposed approach. Compared to single-modal Yolov5s, the proposed method improves mAP by 10.8% and 2.5% in the visible light and infrared modes, respectively. Since visible light images cannot locate corresponding positions in the DroneVehicle dataset under insufficient illumination, the introduction of infrared information results in a significant improvement. Compared to multimodal methods, the proposed method outperforms the state-of-the-art (SOTA) method by 3.2%, with improvements of 8.6% for Truck, 0.6% for Freight, and 0.8% for Van.

[0134] Compared to the highest accuracy in each category of the visible light mode, this method improves the accuracy of all categories by more than 4.9%; compared to the highest accuracy in each category of the infrared mode, this method improves the accuracy of all categories by more than 0.9%. In the visible light mode, this method shows significant improvements in the detection of Trucks, Freighters, and Vans; specifically, it improves the accuracy of Cars by 6.1%, Trucks by 14.9%, Freighters by 15.5%, Buses by 4.9%, and Vans by 12.5%. In the infrared mode, because the multi-scale detector head can detect targets of different sizes well, the overall improvement is smaller than that in the visible light mode; specifically, it improves the accuracy of Cars by 1.2%, Trucks by 3.8%, Freighters by 0.9%, Buses by 1.5%, and Vans by 3.9%.

[0135] To further verify the effectiveness and robustness of this method, experiments were conducted on the challenging VEDAI dataset, and comparisons were made with mainstream methods.

[0136] Similarly, the data is divided into visible light and infrared modes. Yolov5s also performs well on the VEDAI dataset. Compared with the single-modal Yolov5s, the mAP of this method is increased by 2.8% and 12.4% in the visible light mode and infrared mode, respectively. Since the VEDAI dataset only contains images of daytime scenes and the shooting distance is relatively far, the improvement in the infrared mode is more significant after introducing color information.

[0137] Compared to the highest accuracy in each category of the visible light modal, this method achieves improvements in four categories: Car (0.2%), Other (2.9%), Tractor (2.3%), and Van (5.2%). The remaining categories show a decrease in accuracy of no more than 3.6% from the highest accuracy. Compared to the highest accuracy in each category of the infrared modal, this method achieves improvements in six categories: Car (1.4%), Pickup (0.8%), Other (22.7%), Tractor (27.7%), Boat (11.3%), and Van (13.3%). The significant improvement is attributed to the similar appearance contours of Tractor, Boat, Van, and Other after incorporating color information.

[0138] Table 8. Evaluation Results of the VEDAI Dataset

[0139]

[0140]

[0141] In summary, the embodiments of the present invention enhance the features of at least two modalities of remote sensing images through a remote sensing target detection model. This strengthens the semantic features of small targets while effectively aggregating spatial and channel detail information of the targets, thereby solving the problem of feature loss of small targets caused by network deepening or the use of only unidirectional cross-layer fusion. By performing feature fusion on the enhanced features, multimodal feature synergy is achieved while reducing the introduction of redundant information, thereby improving the detection accuracy in complex environments and scenarios with varying target scales, while ensuring high inference speed.

[0142] The following describes the multimodal target detection device based on feature enhancement and cooperative interaction provided by the present invention. The multimodal target detection device based on feature enhancement and cooperative interaction described below can be referred to in correspondence with the multimodal target detection method based on feature enhancement and cooperative interaction described above.

[0143] Figure 7 A schematic diagram of a multimodal target detection device based on feature enhancement and cooperative interaction is shown. The device includes:

[0144] The image acquisition unit 71 acquires remote sensing images in at least two modalities;

[0145] The remote sensing target detection unit 72 inputs remote sensing images into a remote sensing target detection model to obtain remote sensing target detection results output by the remote sensing target detection model. The remote sensing target detection model is trained based on at least two modalities of remote sensing training images and merged labels obtained by merging the labels corresponding to each remote sensing training image. The remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the enhanced features, so as to perform remote sensing target detection based on the fused features and the enhanced features.

[0146] In this embodiment, the remote sensing target detection unit 72 includes: a feature extraction unit, which inputs at least two modalities of remote sensing images to a feature extraction layer to extract features respectively, thereby obtaining extracted features of the corresponding remote sensing images; a feature enhancement unit, which inputs the extracted features of each remote sensing image to a feature enhancement layer to enhance each extracted feature respectively, thereby obtaining enhanced features of each remote sensing image; a feature fusion unit, which inputs the enhanced features of each remote sensing image to a feature fusion layer to perform feature fusion, thereby obtaining fused features; and a target detection unit, which inputs the enhanced features output by the feature enhancement unit and the fused features output by the feature fusion unit to a target detection layer to perform remote sensing target detection on each enhanced feature and the fused features respectively, and combines non-maximum suppression to obtain remote sensing target detection results.

[0147] Specifically, the feature enhancement unit includes: a first modulation subunit, which modulates the high-level semantic features in the extracted features of a single remote sensing image from top to bottom using the global channel attention mechanism of the feature enhancement layer to obtain deep features; a second modulation subunit, which modulates the low-level detail features of the extracted features from bottom to top using the point-by-point channel attention mechanism of the feature enhancement layer to obtain shallow features; and a feature enhancement subunit, which obtains the enhanced features of the corresponding remote sensing image based on the deep features and the shallow features.

[0148] The feature fusion unit includes: a key feature extraction subunit, which extracts the corresponding key features for each individual enhancement feature using the multi-scale convolutional attention (MSCA) module; and a feature fusion subunit, which obtains the fused features based on the attention weights of each key feature and the other enhancement features corresponding to the previously acquired key features.

[0149] Further, the feature fusion subunit includes: a first feature acquisition subunit, which multiplies the key features obtained based on visible light image enhancement features and the attention weights of infrared image enhancement features element by element to obtain a first feature; a second feature acquisition subunit, which multiplies the key features obtained based on infrared image enhancement features and the attention weights of visible light image enhancement features element by element to obtain a second feature; and a feature fusion subunit, which adds the first feature and the second feature element by element to obtain a fused feature.

[0150] In an optional embodiment, the feature fusion subunit further includes: a weight acquisition subunit, which acquires the attention weights corresponding to the enhancement features of different modal remote sensing images before obtaining the fused features based on the attention weights of each key feature and the other enhancement features corresponding to the previously acquired key features.

[0151] In addition, the target detection unit includes: a first stitching subunit, which upsamples the deep features in each enhancement feature and stitches the upsampled features with the shallow features in the corresponding enhancement feature according to a first preset stitching order to obtain the enhanced stitched features corresponding to each enhancement feature; a first target detection subunit, which performs remote sensing target detection on the enhanced stitched features corresponding to each enhancement feature to obtain the corresponding first target detection result; a second stitching subunit, which upsamples the deep features in the fusion feature and stitches the upsampled features with the shallow features in the fusion feature according to a first preset stitching order to obtain the fusion stitched features corresponding to the fusion feature; a second target detection subunit, which performs remote sensing target detection on the fusion stitched features to obtain the second target detection result; a third stitching subunit, which stitches all the first target detection results and the second target detection results according to a second preset stitching order to obtain the stitched result; and a remote sensing target detection subunit, which uses non-maximum suppression based on the stitched result to obtain the remote sensing target detection result.

[0152] Specifically, the remote sensing target detection subunit includes: distinguishing grandchild units, step 1: based on the stitching results, distinguishing all bounding boxes according to category; sorting grandchild units, step 2: sorting the bounding boxes in each category in descending order of confidence from high to low; selecting grandchild units, step 3: selecting the bounding box bbox1 with the highest confidence among all bounding boxes in a certain category, then removing the bounding box bbox1 with the highest confidence from the list of all bounding boxes in that category and adding it to the output list; acquiring grandchild units, step 4: calculating the IOU between the bbox1 and the remaining bboxes in the list of bounding boxes in that category; comparing grandchild units, step 5: comparing the IOUs. The bounding box is compared with the NMS preset threshold Thre. If the IOU between a bounding box and bounding box1 is greater than Thre, the bounding box is removed from the bounding box list of that category, i.e., redundant bounding boxes are removed. The first step is to loop through the grandchild unit. Step 6: Repeat steps 3 to 5 until the bounding box list of all objects in that category is empty. At this point, the traversal of one object category is complete. The second step is to loop through the grandchild unit. Step 7: Repeat steps 2 to 6 to complete the NMS post-processing process for all object categories in sequence. Step 8: The output list is the desired detection box. The NMS process ends, and the detection box and its corresponding confidence, classification, and angle classification are used as the remote sensing target detection result.

[0153] In an optional embodiment, the apparatus further includes: a training unit for training the remote sensing target detection model before inputting the remote sensing images into the remote sensing target detection model. Specifically, the training unit includes: a training data acquisition unit for acquiring at least two modalities of remote sensing training images and labels corresponding to each remote sensing training image; a label processing unit for merging the labels corresponding to each remote sensing training image and deleting duplicate information in the merged labels to obtain merged labels; and a training unit for using the acquired at least two modalities of remote sensing training images as input data for training, and using the labels corresponding to each remote sensing training image and the merged labels as training labels to train the model to obtain a remote sensing target detection model for detecting multimodal remote sensing images.

[0154] In one possible implementation, the training unit further includes: after acquiring remote sensing training images of at least two modalities, performing random color distortion and inversion stitching processing on each of the remote sensing training images to improve data diversity.

[0155] In this embodiment, the training unit includes: a result acquisition subunit, which inputs at least two modalities of remote sensing training images into the model to be trained to obtain a first target training result corresponding to each modality of remote sensing training image and a second target training result obtained by fusing features of at least two modalities of remote sensing training images; a first loss function construction subunit, which constructs a first loss function corresponding to each remote sensing training image based on each first target training result and the label of the remote sensing training image corresponding to the first target training result; a second loss function construction subunit, which constructs a second loss function based on the second target training result and the merged label; and a total loss function construction subunit, which obtains a total loss function based on the first loss function and the second loss function of all remote sensing training images, and terminates training upon convergence of the total loss function.

[0156] Furthermore, the first target training results include bounding box training results, confidence training results, classification training results, and angle classification results. The labels of the remote sensing training images include ground truth bounding box values, ground truth confidence values, classification labels, and angle classification labels. Correspondingly, the first loss function construction sub-units include: a grandchild unit for bounding box regression loss, which constructs a bounding box regression loss using CIoU based on the bounding box training results and ground truth bounding box values; a grandchild unit for confidence loss, which constructs a confidence loss using smoothed binary cross-entropy based on the confidence training results and ground truth confidence values; a grandchild unit for classification loss, which constructs a classification loss using smoothed binary cross-entropy based on the classification training results and classification labels; and a grandchild unit for angle loss, which constructs an angle loss using CSL circular smoothed labels and smoothed binary cross-entropy based on the angle classification results and angle classification labels. The first loss function construction sub-unit obtains the first loss function for the corresponding remote sensing training image based on the bounding box regression loss, confidence loss, classification loss, and angle loss.

[0157] In summary, the embodiments of the present invention acquire at least two different modalities of remote sensing images through the image acquisition unit, thereby facilitating the remote sensing target detection unit to perform feature enhancement on the remote sensing images of at least two modalities using the remote sensing target detection model. This enhances the semantic features of small targets while effectively aggregating spatial and channel detail information of the targets, thus solving the problem of small target feature loss caused by network deepening or the use of only unidirectional cross-layer fusion. By performing feature fusion on the enhanced features, multimodal feature synergy is achieved while reducing the introduction of redundant information, thereby improving the detection accuracy in complex environments and target scale variation scenarios, while ensuring high inference speed.

[0158] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 81, a communication interface 82, a memory 83, and a communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 communicate with each other through the communication bus 84. The processor 81 can call logical instructions in the memory 83 to execute a multimodal target detection method based on feature enhancement and collaborative interaction. This method includes: acquiring remote sensing images of at least two modalities; inputting the remote sensing images into a remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein the remote sensing target detection model is trained based on the acquired remote sensing training images of at least two modalities and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the enhanced features to perform remote sensing target detection based on the fused features and the enhanced features.

[0159] Furthermore, the logical instructions in the aforementioned memory 83 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal target detection method based on feature enhancement and collaborative interaction provided by the above methods. The method includes: acquiring remote sensing images of at least two modalities; inputting the remote sensing images into a remote sensing target detection model to obtain a remote sensing target detection result output by the remote sensing target detection model; wherein the remote sensing target detection model is trained based on the acquired remote sensing training images of at least two modalities and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the enhanced features, so as to perform remote sensing target detection based on the fused features and the enhanced features.

[0161] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal target detection method based on feature enhancement and cooperative interaction provided by the above methods. The method includes: acquiring remote sensing images of at least two modalities; inputting the remote sensing images into a remote sensing target detection model to obtain a remote sensing target detection result output by the remote sensing target detection model; wherein the remote sensing target detection model is trained based on the acquired remote sensing training images of at least two modalities and merged labels obtained by merging the labels corresponding to each remote sensing training image; the remote sensing target detection model is used to perform feature enhancement on the input remote sensing images of at least two modalities respectively, and to perform feature fusion on the enhanced features, so as to perform remote sensing target detection based on the fused features and the enhanced features.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal target detection method based on feature enhancement and collaborative interaction, characterized in that, include: Acquire remote sensing images in at least two modalities; The remote sensing image is input into the remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein, the remote sensing target detection model is trained based on at least two modalities of remote sensing training images and merged labels obtained by merging the labels corresponding to each of the remote sensing training images; The remote sensing target detection model includes: The feature extraction layer performs feature extraction on the input remote sensing images of at least two modalities to obtain the extracted features corresponding to the remote sensing images; The feature enhancement layer enhances each extracted feature to obtain enhanced features for each of the remote sensing images, including: For the extracted features of a single remote sensing image, a global channel attention mechanism is used to modulate the high-level semantic features of the extracted features from top to bottom to obtain deep features, and a point-by-point channel attention mechanism is used to modulate the low-level detail features of the extracted features from bottom to top to obtain shallow features; the point-by-point channel attention mechanism is used to aggregate the channel feature context information of each spatial location individually. Based on the deep features and the shallow features, the enhanced features corresponding to the remote sensing image are obtained; The feature fusion layer fuses the enhanced features to obtain fused features, including: For each of the aforementioned enhanced features, the corresponding key features are obtained using the multi-scale convolutional attention (MSCA) module; Based on the attention weights of each of the key features and the other enhanced features corresponding to the previously acquired key features, a fused feature is obtained; The target detection layer performs remote sensing target detection on each of the enhanced features and the fused features, and combines non-maximum suppression to obtain the remote sensing target detection result, including: The deep features in each of the enhancement features are upsampled, and based on the first preset splicing order, the upsampled features are spliced ​​with the shallow features in the corresponding enhancement features to obtain the enhancement splicing features corresponding to each enhancement feature. Remote sensing target detection is performed on the enhanced stitched features corresponding to each enhanced feature to obtain the corresponding first target detection result; The deep features in the fusion feature are upsampled, and based on the first preset splicing order, the upsampled features are spliced ​​with the shallow features in the fusion feature to obtain the fusion spliced ​​feature corresponding to the fusion feature; Remote sensing target detection is performed on the fused and stitched features to obtain a second target detection result; Based on the second preset splicing order, all the first target detection results and the second target detection results are spliced ​​together to obtain the splicing result; Based on the stitching results, non-maximum suppression is used to obtain the remote sensing target detection results.

2. The multimodal target detection method based on feature enhancement and collaborative interaction according to claim 1, characterized in that, Training the remote sensing target detection model includes: Obtain remote sensing training images of at least two modalities and the corresponding labels for each of the remote sensing training images; The labels corresponding to each of the remote sensing training images are merged, and duplicate information in the merged labels is removed to obtain merged labels; The remote sensing training images of at least two modalities are input into the model to be trained to obtain the first target training result corresponding to each modal remote sensing training image and the second target training result obtained after fusing the features of the remote sensing training images of at least two modalities. Based on the training results of each first target and the labels of the remote sensing training images corresponding to the training results of the first targets, a first loss function corresponding to each remote sensing training image is constructed. Based on the training results of the second target and the merged labels, a second loss function is constructed; The total loss function is obtained based on the first loss function and the second loss function of all remote sensing training images, and the training ends when the total loss function converges.

3. The multimodal target detection method based on feature enhancement and collaborative interaction according to claim 2, characterized in that, The first target training result includes bounding box training result, confidence training result, classification training result, and angle classification result; the labels of the remote sensing training image include bounding box ground truth, confidence ground truth, classification label, and angle classification label. The step of constructing a first loss function corresponding to each remote sensing training image based on the training results of each first target and the labels of the remote sensing training images corresponding to the training results of the first targets includes: Based on the bounding box training results and the ground truth bounding box values, a bounding box regression loss is constructed using CIoU; Based on the confidence training results and the confidence ground truth, a confidence loss is constructed using smoothed binary cross-entropy. Based on the classification training results and the classification labels, a classification loss is constructed using smooth binary cross-entropy; Based on the angle classification results and the angle classification labels, an angle loss is constructed using CSL circular smooth labels and smooth binary cross-entropy. Based on the bounding box regression loss, the confidence loss, the classification loss, and the angle loss, a first loss function for the corresponding remote sensing training image is obtained.

4. A multimodal target detection device based on feature enhancement and cooperative interaction, used to implement the multimodal target detection method based on feature enhancement and cooperative interaction as described in any one of claims 1 to 3, characterized in that, include: The image acquisition unit acquires remote sensing images in at least two modalities; The remote sensing target detection unit inputs the remote sensing image into the remote sensing target detection model to obtain the remote sensing target detection result output by the remote sensing target detection model; wherein, the remote sensing target detection model is trained based on at least two modalities of remote sensing training images and merged labels obtained by merging the labels corresponding to each of the remote sensing training images; The remote sensing target detection model is used to perform feature enhancement on at least two modalities of input remote sensing images, and to perform feature fusion on the enhanced features, so as to perform remote sensing target detection based on the fused features and the enhanced features.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal target detection method based on feature enhancement and collaborative interaction as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal target detection method based on feature enhancement and cooperative interaction as described in any one of claims 1 to 3.

Images