Target detection and recognition method based on cross-modal feature alignment and cross-fusion

Abstract

The application discloses a target detection and recognition method based on cross-modal feature alignment and cross fusion, and the method comprises the following steps: a double-modal target detection model comprising a backbone network, a neck network and a fusion detection layer is constructed, the backbone network is used for feature extraction to obtain visible light features and infrared features; a cross-modal alignment and enhancement module of the backbone network is used for double-modal feature alignment space correction, cross-modal bidirectional information interaction, and then complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features; the neck network is used for multi-scale feature extraction and input into the fusion detection layer, the fusion detection layer outputs a target detection result by using a fusion feature through a rotating detection head. The application effectively corrects the spatial offset of visible light features and infrared features, fully excavates double-modal complementary features, improves the detection precision and robustness of rotating targets in an unmanned aerial vehicle aerial full-time complex scene, and solves the defect of poor alignment effect of traditional multi-modal detection.

Description

Technical Field

[0001] This invention relates to the field of target detection and recognition in drone aerial imagery, and particularly to a target detection and recognition method based on cross-modal feature alignment and cross-fusion. Background Technology

[0002] With their advantages of high mobility, flexible deployment, and low cost, drones have rapidly expanded from their initial role as reconnaissance tools to numerous civilian applications, including smart city construction, public safety patrols, precision agricultural monitoring, and natural disaster relief. All-weather, robust target detection of key targets such as vehicles and pedestrians in aerial imagery is a crucial technological foundation for intelligent analysis and decision-making in relevant scenarios. While real-time target detection algorithms, such as the YOLO series, have achieved excellent performance in natural scenes thanks to advancements in deep learning technology, drone aerial imagery is characterized by small target scale, dense distribution, severe occlusion, and often rotating orientations. Traditional horizontal bounding boxes are insufficient to accurately describe the target's spatial extent. Furthermore, relying solely on single-modal visible light sensors results in significant image quality degradation at night, in low light conditions, and in adverse weather, failing to guarantee stable all-weather detection performance.

[0003] Deep learning-based target detection technologies are mainly divided into two categories: two-stage and single-stage. Two-stage detectors, represented by the R-CNN series, offer good detection accuracy but suffer from high computational overhead, making them unsuitable for the real-time requirements of UAV scenarios. Single-stage detectors, on the other hand, eliminate the candidate region generation step, achieving outstanding inference speed while maintaining accuracy. The YOLOv8 network combines feature pyramids and path aggregation networks to achieve multi-scale feature fusion and employs an anchor-free decoupled detection head to achieve a good balance between accuracy and efficiency. Its extended framework, YOLOv8-OBB, effectively adapts to the detection needs of rotating targets in UAV aerial photography by adding an angle regression branch. However, these methods are still based on single-modal visible light images and fail to fundamentally solve the detection failure problem under harsh conditions such as low light. In the field of multimodal target detection for UAV aerial photography, existing research can be divided into pixel-level early fusion, feature-level mid-stage fusion, and decision-level late-stage fusion, depending on the fusion stage. Pixel-level fusion directly stitches together images of different modalities at the input, a simple method but struggling to handle semantic differences between modalities. Decision-level fusion merges results after independent detection of each modality, offering greater flexibility but failing to fully exploit the complementary value of underlying features. Feature-level fusion, currently the mainstream approach, often employs a two-stream network architecture to provide independent feature extraction paths for different modalities; however, traditional two-stream network architectures still struggle to fully leverage the semantic information and complementary advantages between multimodalities. Furthermore, visible light and infrared images have fundamentally different imaging mechanisms, and even with image-level registration, a significant spatial misalignment problem remains in the feature space—a weak cross-modal alignment phenomenon—which directly reduces the effectiveness of feature fusion. Currently, offset field sampling can only correct spatial positional deviations and cannot address differences in feature representation between modalities; while global cross-attention can achieve cross-modal information interaction, its high computational complexity makes it unsuitable for resource-constrained UAV equipment. There is an urgent need for a multimodal target detection method that can simultaneously address spatial misalignment and feature representation differences while being computationally efficient. Summary of the Invention

[0004] The purpose of this invention is to address the existing technical pain points in the field of visible light and infrared multimodal target detection in UAV aerial photography, and to provide a target detection and recognition method based on cross-modal feature alignment and cross-fusion. By utilizing cross-modal alignment and enhancement modules, cross-modal spatial alignment, cross-modal information interaction, cross-modal difference enhancement, and modal feature refinement are achieved in sequence, thereby eliminating the spatial misalignment problem between the two modal features and strengthening the complementary features of visible light and infrared modalities.

[0005] The objective of this invention is achieved through the following technical solution: A target detection and recognition method based on cross-modal feature alignment and cross-fusion, the method includes: S1. Construct a dual-modal target detection model with YOLOv8-OBB as the core architecture, including a backbone network, a dual-stream neck network, and a fusion detection layer. Obtain visible light and infrared dual-modal image data and input them into the dual-modal target detection model. The backbone network extracts features from the visible light and infrared dual-modal image data to obtain interrelated visible light features and infrared features. S2. The backbone network includes a cross-modal alignment and enhancement module. The cross-modal alignment and enhancement module first uses visible light features and infrared features to perform bimodal feature alignment spatial correction, then introduces a cross-attention mechanism to perform cross-modal bidirectional information interaction, and then mines the difference information of the two modes and performs complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features. S3. The dual-stream neck network uses enhanced visible and enhanced infrared features to perform multi-scale feature extraction and inputs them into the fusion detection layer. The fusion detection layer performs weighted fusion of visible and infrared features at the same scale to obtain fused features. The fusion detection layer outputs target detection results by rotating the detection head and using the fused features. The target detection results include target detection boxes and target attributes, including target type and target number.

[0006] To better realize the present invention, the dual-modal target detection model uses dual-modal image sample data for target detection and segmentation training. The dual-modal image sample data covers target types including vehicles, ships, site facilities, and pedestrians. The dual-modal image sample data consists of visible light images and infrared thermal imaging images targeting the same location area or the same target, or containing the same target and the same location area.

[0007] Preferably, in method S1, the research area is photographed by a UAV equipped with a visible light and infrared dual-modal sensor. The visible light and infrared dual-modal sensor obtains time-series data of visible light and infrared dual-modal image association. The visible light and infrared dual-modal image data consists of research time frames extracted from the time-series data of visible light and infrared dual-modal image association, and mutually correlated visible light images and infrared thermal images. The time-series data of visible light and infrared dual-modal image association is input into the dual-modal target detection model to obtain the target detection results for each time frame. The same target across time frames is assigned a unique target number.

[0008] Preferably, in method S1, the visible light and infrared dual-modal image data includes mutually correlated visible light images and infrared thermal imaging images. The backbone network has a visible light feature extraction branch and an infrared feature extraction branch. The backbone network uses the two branches of the visible light feature extraction branch and the infrared feature extraction branch to extract features from the visible light and infrared dual-modal image data to obtain mutually correlated visible light features and infrared features.

[0009] Preferably, in method S2, the cross-modal alignment and enhancement module includes a learnable offset alignment unit, a local window cross attention unit, a cross-modal difference enhancement unit, and a feature refinement unit. The learnable offset alignment unit uses visible light features and infrared features to perform bimodal feature alignment spatial correction. The local window cross attention unit introduces a cross attention mechanism to perform cross-modal bidirectional information interaction. The cross-modal difference enhancement unit is used to mine the difference information between the two modes and perform complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features. The feature refinement unit uses channel attention and spatial attention to perform feature refinement processing.

[0010] Preferably, the learnable offset alignment unit acquires visible light features. infrared features absolute difference characteristics between Visible light characteristics infrared features Absolute difference characteristics The features are concatenated along the channel dimension and then input into a three-layer (3×3) stacked convolutional offset prediction network. The offset prediction network then predicts the offset features. ; Apply offset constraints to obtain the limit offset range The expression is as follows: , The tanh function is a hyperbolic tangent constraint. The preset maximum allowable offset; the learnable offset alignment unit is constructed with a basic sampling grid in the GridSample normalized coordinate system. This will limit the offset magnitude. The deformed sampling mesh is obtained by superimposing it in the normalized coordinate space under the normalized coordinate system. Based on the deformed sampling mesh Differentiable bilinear grid sampling method is used for infrared features Resampling is performed to obtain aligned infrared features The expression is as follows: , This indicates that GridSample is a bilinear grid sampling process.

[0011] Preferably, the local window cross-attention unit utilizes a cross-attention mechanism to perform linear projection using visible light features as queries and infrared features as keys, thereby enhancing the visible light features with infrared heat source information. The local window cross-attention unit also utilizes a cross-attention mechanism to perform linear projection using infrared features as queries and visible light features as keys, thereby enhancing the infrared features with visible light texture and color information. The local window cross-attention unit includes a visible light branch and an infrared branch for bidirectional processing of infrared and visible light features, as expressed below: , ,in For the aligned infrared features, It has visible light characteristics. , These are the scaling factor parameters, This represents the LocalCrossAttention function. To enhance infrared signature, To enhance visible light characteristics.

[0012] Preferably, for visible light characteristics infrared features Feature offset alignment processing is performed. The cross-modal difference enhancement unit includes a cross-channel reweighting stage and a directed difference compensation stage. The cross-channel reweighting stage method includes the following: calculating channel attention weights for infrared and visible light features respectively, and performing nonlinear transformation through a reweighting network to obtain infrared weights respectively. and visible light weight The aligned infrared and visible light features are added together to calculate the shared channel weight. Utilizing visible light weighting and shared channel weight The expression for cross-channel modulation of infrared features is as follows: , The infrared features are obtained after cross-channel modulation; infrared features are utilized. and shared channel weight The expression for cross-channel modulation of visible light features is as follows: , The visible light characteristics are those obtained after cross-channel modulation.

[0013] Preferably, the directed difference compensation stage method includes: calculating the directed difference features between the visible light features and the infrared features modes respectively, and obtaining the infrared compensation information by weighting the features through channel attention. and visible light compensation information The expression is as follows: , ,in This refers to the directional difference characteristic of visible light relative to infrared light. This refers to the directional difference characteristics of infrared light relative to visible light. Channel attention is used to weight the differential features; the results of cross-channel reweighting and directed difference compensation are added to the original features through a learnable channel-level scaling factor for feature enhancement, as shown in the following expression: , , ~ These are the learnable channel-level scaling parameters. Enhanced infrared signature or enhanced infrared signature; This refers to the enhanced visible light characteristics or the enhanced visible light characteristics.

[0014] Preferably, the feature refining unit includes two processing stages: channel refining and spatial refining, which enhances infrared features. With enhanced visible light characteristics The input feature refinement unit performs feature refinement processing. Channel refinement uses a channel attention mechanism to generate channel weights, which are multiplied with the input features channel by channel to highlight important channels and suppress redundant channels. Spatial refinement calculates the mean and maximum value of the channel-refined features along the channel dimension and performs feature concatenation. After feature concatenation, spatial attention weights are generated through a convolutional layer and multiplied with the features pixel by pixel to highlight the target region and suppress background noise.

[0015] Preferably, in method S3, the dual-stream neck network has corresponding feature path aggregation networks for enhancing visible features and enhancing infrared features. Each feature path aggregation network includes two paths: top-down and bottom-up. The top-down path upsamples and progressively transmits high semantic features from deep layers to shallow layers and then splices and fuses them. The bottom-up path downsamples and progressively transmits high-resolution features from shallow layers to deep layers and then splices and fuses them with the output of the corresponding layer of the top-down path. The two paths of the feature path aggregation network ultimately output three scales of enhanced features. The dual-stream neck network obtains the three scales of enhanced features corresponding to the enhanced visible features and the three scales of enhanced infrared features.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The backbone network of the dual-modal target detection model of the present invention has a serial structure for cross-modal alignment and enhancement modules, which effectively solves the two core problems of spatial misalignment and feature expression difference between visible light and infrared features. The cross-modal alignment and enhancement modules realize pixel-level spatial correction through learnable offset alignment units, and then gradually complete bidirectional information interaction, complementary feature mining and background noise suppression through local window cross attention units, cross-modal difference enhancement units and feature refinement units. This fundamentally improves the quality of multimodal feature fusion and significantly improves the detection accuracy and robustness of UAV aerial photography for rotating targets in all-weather complex scenarios such as day and night, cloudy, and backlight.

[0017] (2) The local window cross attention unit of the present invention realizes the bidirectional information interaction and mutual enhancement of visible light and infrared features, making the model more adaptable to the resource constraints of computing power and storage of UAV edge devices, and taking into account the requirements of high performance and real-time performance. The cross-modal difference enhancement unit can adaptively mine and enhance the complementary information between the two modalities through cross-channel reweighting and directional difference compensation mechanism. It not only highlights the important channels that are of common concern between the modalities, but also utilizes the directionality of the difference to realize the use of more information-rich modal features to compensate for another modality and maximize the fusion value of multimodal data, effectively solving the problem of information loss of a single modality under specific conditions. Attached Figure Description

[0018] Figure 1 This is a flowchart of the target detection and recognition method of the present invention; Figure 2 This is a schematic diagram illustrating the structural principle of the dual-modal target detection model in the embodiment; Figure 3 This is a schematic diagram illustrating the structural principle of the cross-modal alignment and enhancement module in the embodiment; Figure 4 This is a schematic diagram illustrating the principle of the learnable offset alignment unit in the embodiment; Figure 5 This is a schematic diagram illustrating the principle of the local window cross attention unit in the embodiment; Figure 6 This is a schematic diagram of the principle of the cross-modal difference enhancement unit in the embodiment. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to embodiments: Example like Figure 1 As shown, a target detection and recognition method based on cross-modal feature alignment and cross-fusion is proposed. The method includes: S1, such as Figure 2As shown, a dual-modal target detection model is constructed with YOLOv8-OBB as the core architecture, including a backbone network, a dual-stream neck network, and a fusion detection layer. Visible light and infrared dual-modal image data are acquired and input into the dual-modal target detection model. The backbone network extracts features from the visible light and infrared dual-modal image data to obtain interrelated visible light and infrared features. This invention utilizes a UAV equipped with a visible light and infrared dual-modal sensor to conduct aerial photography of the research area. The visible light and infrared dual-modal sensor obtains temporal data associated with visible light and infrared dual-modal images. The visible light and infrared dual-modal image data consists of research time frames extracted from the temporal data associated with visible light and infrared dual-modal images, along with interrelated visible light and infrared thermal imaging images. The temporal data associated with visible light and infrared dual-modal images is input into the dual-modal target detection model to obtain target detection results for each time frame. Identical targets across time frames are assigned unique target numbers.

[0020] The present invention provides visible light and infrared dual-modal image data, which includes interrelated visible light images and infrared thermal imaging images. The backbone network has visible light feature extraction branches and infrared feature extraction branches. The backbone network uses the two branches of visible light feature extraction branches and infrared feature extraction branches to extract features from the visible light and infrared dual-modal image data to obtain interrelated visible light features and infrared features.

[0021] In some embodiments, such as Figure 3 As shown, the bimodal target detection model uses bimodal image sample data for target detection and segmentation training. The bimodal image sample data covers target types including vehicles, ships, site facilities, and pedestrians. Vehicles include cars, trucks, buses, vans, and lorries, while site facilities include power poles, oil tanks, bridges, and ports. The bimodal image sample data consists of visible light images and infrared thermal images of the same location area, the same target, or the same location area containing the same target. The visible light images and infrared thermal images are image data obtained by capturing and monitoring the same target covered by the same location area. In other words, the visible light images and infrared thermal images are image data of the same target covered by the same location area.

[0022] S2. The backbone network includes a cross-modal alignment and enhancement module. The cross-modal alignment and enhancement module first uses visible light features and infrared features to perform bimodal feature alignment spatial correction, then introduces a cross-attention mechanism to perform cross-modal bidirectional information interaction, and then mines the difference information between the two modes and performs complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features.

[0023] In some embodiments, the cross-modal alignment and enhancement module includes a learnable offset alignment unit, a local window cross-attention unit, a cross-modal difference enhancement unit, and a feature refinement unit, specifically, such as Figure 3As shown, the cross-modal alignment and enhancement module comprises a four-layer serial processing structure. The first layer is a learnable offset alignment unit that uses offset fields and grid sampling to complete the spatial correction of visible light and infrared features. The second layer uses a local window cross-attention unit to achieve bidirectional feature information interaction between the two modalities within a limited window. The third layer sets up a cross-modal difference enhancement unit and relies on cross-channel reweighting and directed difference compensation to mine modal complementary features. The fourth layer is a feature refinement unit that integrates channel attention and spatial attention to filter and optimize the features after the first three layers of processing, suppressing irrelevant background and highlighting target features. The entire cross-modal alignment and enhancement module integrates multiple structures such as spatial alignment, local window cross-attention, difference enhancement, channel attention, and spatial attention to simultaneously solve the problems of cross-modal spatial misalignment and feature expression differences. The learnable offset alignment unit utilizes visible light and infrared features for bimodal feature alignment spatial correction. The local window cross-attention unit introduces a cross-attention mechanism for cross-modal bidirectional information interaction. The cross-modal difference enhancement unit mines the difference information between the two modalities and performs complementary enhancement processing to obtain enhanced visible light and enhanced infrared features. The feature refinement unit uses channel attention and spatial attention for feature refinement processing. The cross-modal alignment and enhancement module progressively completes bimodal feature spatial correction, cross-modal bidirectional information interaction, modal complementary information mining, and feature noise reduction and refinement to eliminate the feature spatial misalignment problem caused by different imaging modalities.

[0024] like Figure 4 As shown, a learnable offset alignment unit (whose goal is to predict a dense spatial offset field from infrared features to visible light features and use this offset field to resample the infrared features into the visible light coordinate space) can be used to obtain visible light features. With infrared features The absolute difference characteristics between Visible light characteristics With infrared features Absolute difference characteristics The features are concatenated along the channel dimension and then input into a three-layer 3×3 convolutional stacked offset prediction network (also known as a lightweight offset prediction network, which includes...). (Activation function), the offset prediction network predicts the predicted offset features. The expression is as follows: ,in for Activation function This represents a 3×3 convolution. This represents feature concatenation processing along the channel dimension. Offset constraints are applied to obtain the limited offset magnitude. The expression is as follows: , The tanh function is a hyperbolic tangent constraint. To preset the maximum allowable offset, the weights and biases of the last convolutional layer in the offset prediction network are initialized to zero. This ensures that the offset is close to zero in the early stages of training, meaning that the infrared features hardly shift, thus avoiding the impact of randomly initialized offsets on training stability. The learnable offset alignment unit is constructed with a basic sampling grid in the GridSample normalized coordinate system. This will limit the offset magnitude. The deformed sampling mesh is obtained by superimposing it in the normalized coordinate space under the normalized coordinate system. (Predicted limit offset magnitude) The offset is converted to a normalized coordinate space and superimposed on the base sampling grid to form a deformed sampling grid. The expression is as follows: , H and W represent the height and width of the feature, respectively. The base sampling grid (also known as the standard uniform sampling grid) covers the range [-1, 1]. To limit the offset amplitude Horizontal x-axis component, This is the normalized x-axis offset. To limit the offset amplitude Horizontal y-axis component, This is the normalized y-axis offset. Based on the deformed sampling mesh. Differentiable bilinear grid sampling method is used for infrared features Resampling is performed to obtain aligned infrared features The expression is as follows: , This indicates that GridSample is a bilinear grid sampling process.

[0025] In some embodiments, such as Figure 5 As shown, the local window cross-attention unit utilizes a cross-attention mechanism to generate and enhance the visible light features by using visible light features as queries and infrared features as keys through linear projection. This enhances the infrared heat source information of the infrared features with the visible light features. Taking the acquisition of cross-modal information from infrared by visible light as an example, the query vector is generated by linear projection of visible light features, and the key and value vectors are generated by linear projection of infrared features. Let the query, key, and value matrices within the window be Q, K, and V, respectively. The multi-head attention calculation is as follows: , ,in, For each dimension of attention head, The number of attention heads; the outputs of multi-head attention are concatenated and linearly projected to serve as cross-modal context information.

[0026] The local window cross-attention unit includes a visible light branch and an infrared branch for bidirectional flow of infrared and visible light features (i.e., it realizes bidirectional interaction of visible light obtaining information from infrared and infrared obtaining information from visible light), as shown in the following expression: , ,in For the aligned infrared features, It has visible light characteristics. , These are the scaling factor parameters, and the scaling factor. and Initializing it to 0.1 makes the contribution of cross attention small in the early stage of training, and gradually increases it as training progresses, which helps to stabilize the training process. This represents the LocalCrossAttention function. To enhance infrared signature, To enhance visible light characteristics.

[0027] In some embodiments, such as Figure 6 As shown, for visible light characteristics With infrared features Perform feature offset alignment processing (i.e., use learnable offset alignment units to perform visible light features) With infrared features Perform feature offset alignment processing; such as Figure 6 As shown, infrared features are first aligned to the visible light coordinate space using learnable offset alignment. Then, a cross-modal difference enhancement unit is used to perform complementary enhancement by utilizing the difference information between the two modes after alignment. The cross-modal difference enhancement unit includes a cross-channel reweighting stage and a directed difference compensation stage. The cross-channel reweighting stage method includes the following: calculating channel attention weights for infrared and visible light features respectively, and performing nonlinear transformation through a reweighting network to obtain infrared weights respectively. and visible light weight Taking infrared mode as an example, infrared weighting The expression is as follows: In this context, ChannelAttention is the channel attention operation, and Reweight is a reweighted network consisting of two convolutional layers and a Sigmoid activation function.

[0028] The shared channel weight is calculated by adding the aligned infrared and visible light features. Calculate the shared channel weights by summing the features of the two modalities. The expression is as follows: ;Utilizing visible light weighting and shared channel weight The expression for cross-channel modulation of infrared features is as follows: , The infrared features are obtained after cross-channel modulation; infrared features are utilized. and shared channel weight The expression for cross-channel modulation of visible light features is as follows: , This represents the visible light features after cross-channel modulation. This cross-channel reweighting mechanism allows the features of each mode to be adaptively adjusted according to the channel importance distribution of the other mode, highlighting channels that are considered important by both modes.

[0029] The directed difference compensation stage method includes: calculating the directed difference features between the visible light and infrared modes separately, and then performing weighted processing through channel attention to obtain infrared compensation information. (After channel attention weighting, multiply with visible light features to obtain difference information used to compensate for infrared features) and visible light compensation information. The expression is as follows: , ,in This refers to the directional difference characteristic of visible light relative to infrared light. This refers to the directional difference characteristics of infrared light relative to visible light. Channel attention is used to weight the differential features. The results of cross-channel reweighting and directed difference compensation are added to the original features through a learnable channel-level scaling factor for feature enhancement (the results of cross-channel reweighting and directed difference compensation are added to the original features through a learnable channel-level scaling factor), as shown in the following expression: , , ~ These are learnable channel-level scaling parameters (which allow the network to adaptively adjust the contribution ratio of cross-channel reweighting and difference compensation during training). This refers to the enhanced infrared signature or the enhanced infrared signature. This is for enhancing visible light features or improving visible light features. The use of directional difference makes the compensation directional; a channel with a positive difference indicates that the mode has richer information in this channel and can be used to compensate for and enhance another mode.

[0030] In some embodiments, such as Figure 3 As shown, the feature refinement unit includes two processing stages: channel refinement and spatial refinement, which enhances infrared features. With enhanced visible light characteristics The input feature refinement unit performs feature refinement processing, with channel refinement utilizing a channel attention mechanism to generate channel weights that are multiplied channel-by-channel with the input features to highlight important channels and suppress redundant channels. Spatial refinement calculates the mean and maximum values ​​of the channel-refined features along the channel dimension and concatenates the features. After concatenation, spatial attention weights are generated through a convolutional layer and multiplied pixel-by-pixel with the features to highlight target regions and suppress background noise. In some embodiments, the features refined by the feature refinement unit are added to the input features via residual connections using a learnable scaling factor.

[0031] S3. The dual-flow neck network uses enhanced visible features and enhanced infrared features to perform multi-scale feature extraction (preferably three-scale feature extraction in this embodiment) and inputs them into the fusion detection layer. The fusion detection layer performs weighted fusion of visible light features and infrared features of the same scale to obtain fusion features. The fusion detection layer outputs target detection results by rotating the detection head and using the fusion features. The target detection results include target detection boxes and target attributes. Target attributes include target type and target number.

[0032] In some embodiments, such as Figure 2 As shown, the dual-flow neck network (i.e. Figure 2 The neck network (in the dual-stream network) has corresponding feature path aggregation networks for enhancing visible and infrared features (i.e., each infrared and visible light data stream has its own complete feature path aggregation network). Each feature path aggregation network includes two paths: top-down and bottom-up. The top-down path upsamples and progressively transfers high-semantic features from deep layers to shallow layers and then merges them. The bottom-up path downsamples and progressively transfers high-resolution features from shallow layers to deep layers and merges them with the outputs of the corresponding layers in the top-down path. The two paths of the feature path aggregation network ultimately output three scales of enhanced features. The dual-stream neck network obtains the three scales of enhanced features corresponding to enhanced visible features and the three scales of enhanced infrared features. The fusion and detection layer fuses the output features of the infrared and visible light data streams at the same scale by element-wise addition, resulting in a three-scale fused feature map. The three-scale fused feature map is then fed into the rotating detection head to output the target detection result.

[0033] The implementation case validation is as follows: Experiments were conducted using the DroneVehicle dataset, a large-scale multimodal UAV aerial vehicle detection benchmark dataset released by Tianjin University, specifically designed for visible light and infrared dual-modal target detection tasks. This dataset contains 56,878 images, or 28,439 visible light-infrared image pairs, totaling over 800,000 labeled instances. The labeled categories cover five main vehicle types: cars, trucks, buses, vans, and trucks. All targets are labeled with rotated bounding boxes, accurately describing the spatial extent and orientation of the vehicles, making it ideal for evaluating the performance of rotated target detection algorithms. The DroneVehicle dataset covers all-weather lighting conditions from daytime, evening to nighttime, as well as various complex scenes such as urban roads, parking lots, and highways. This rich scene diversity allows for a comprehensive evaluation of the robustness of multimodal detection algorithms under different lighting and scene conditions, making it highly suitable for validating the effectiveness of the proposed multimodal fusion method. During training, the input images were scaled to 840 × 712 for all experiments, and performance was reported on the validation dataset for each. The case study implementation used an NVIDIA RTX 4090 as the server. The bimodal object detection model was trained using the AdamW optimizer with an initial learning rate of 1e-2, which decayed to 1e-4 as training progressed. The total number of training epochs was 200, with a batch size of 16. The dataset resolution was 840×712. We used mean average precision (mAP) as the evaluation metric for model accuracy.

[0034] Based on the dual-modal target detection model fusion architecture of this embodiment, the contribution of cross-modal alignment and enhancement modules to multimodal target detection performance and the effectiveness of the progressive module construction strategy are verified. The experimental results are shown in Table 1.

[0035]

[0036] As shown in Table 1, after adding learnable offset alignment to the bimodal target detection model, the mAP50 improved from 78.69% to 79.26%, an increase of 0.57 percentage points. This indicates that explicitly modeling cross-modal spatial offsets can alleviate the local misalignment problem between visible light and infrared features, resulting in better spatial consistency of the fused features. The mAP50 of the local window cross-attention version reached 80.45%, an improvement of 1.76 percentage points compared to the baseline, indicating that cross-modal information interaction within local regions can effectively enhance target-related features, especially helping the model supplement information missing or weakly responded to in the current modality from the other modality. The mAP50 of the joint offset alignment and difference enhancement version was 80.14%, an improvement of 1.45 percentage points compared to the baseline, indicating that explicitly enhancing the difference information between the two modalities helps the model capture complementary cues. The complete cross-modal alignment and enhancement version achieved optimal results, with mAP50 reaching 80.61% and mAP50:95 reaching 61.64%, representing improvements of 1.92 and 1.77 percentage points respectively compared to the dual-stream baseline. Simultaneously, the parameter count was 21.63M, only 0.36M more than the dual-stream baseline. Overall, the complete module significantly improved detection performance with a relatively small parameter increment, indicating that learnable offset alignment, local window cross-attention, cross-modal difference enhancement, and modal feature refinement can form an effective synergy, and also verifying the rationality of the cross-modal alignment and enhancement module construction strategy of this invention.

[0037] In addition, the method of this invention was compared with the research results on multimodal object detection on the DroneVehicle dataset published in authoritative international journals and well-known conferences in the past three years. The comparison results are shown in Table 2, with mAP50 and model parameter size as evaluation indicators. The model parameter size of the dual-modal object detection model of this invention is smaller than that of all other methods, and it has an absolute advantage of 5.91%, 6.41%, 1.21%, and 0.81% in mAP50 compared with MCMF, C²Former, CCLDet, and WaveMamba, respectively.

[0038]

[0039] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target detection and recognition method based on cross-modal feature alignment and cross-fusion, characterized in that: the method include: S1. Construct a dual-modal target detection model with YOLOv8-OBB as the core architecture, including a backbone network, a dual-stream neck network, and a fusion detection layer. Obtain visible light and infrared dual-modal image data and input them into the dual-modal target detection model. The backbone network extracts features from the visible light and infrared dual-modal image data to obtain interrelated visible light features and infrared features. S2. The backbone network includes a cross-modal alignment and enhancement module. The cross-modal alignment and enhancement module first uses visible light features and infrared features to perform bimodal feature alignment spatial correction, then introduces a cross-attention mechanism to perform cross-modal bidirectional information interaction, and then mines the difference information of the two modes and performs complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features. S3. The dual-stream neck network uses enhanced visible and enhanced infrared features to perform multi-scale feature extraction and inputs them into the fusion detection layer. The fusion detection layer performs weighted fusion of visible and infrared features at the same scale to obtain fused features. The fusion detection layer outputs target detection results by rotating the detection head and using the fused features. The target detection results include target detection boxes and target attributes, including target type and target number.

2. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 1, characterized in that: The dual-modal target detection model is trained using dual-modal image sample data for target detection and segmentation. The dual-modal image sample data covers target types including vehicles, ships, site facilities, and pedestrians. The dual-modal image sample data consists of visible light images and infrared thermal images of the same location area, the same target, or the same location area containing the same target. In method S1, the research area is photographed by a UAV equipped with a visible light and infrared dual-modal sensor. The visible light and infrared dual-modal sensor obtains temporal data associated with the visible light and infrared dual-modal images. The visible light and infrared dual-modal image data consists of research time frames extracted from the temporal data associated with the visible light and infrared dual-modal images, and interrelated visible light images and infrared thermal images. The temporal data associated with the visible light and infrared dual-modal images is input into the dual-modal target detection model to obtain the target detection results for each time frame. The same target across time frames is assigned a unique target number.

3. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 1, characterized in that: In method S1, the visible light and infrared dual-modal image data includes interrelated visible light images and infrared thermal imaging images. The backbone network has a visible light feature extraction branch and an infrared feature extraction branch. The backbone network uses the two branches of the visible light feature extraction branch and the infrared feature extraction branch to extract features from the visible light and infrared dual-modal image data to obtain interrelated visible light features and infrared features.

4. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 1, characterized in that: In method S2, the cross-modal alignment and enhancement module includes a learnable offset alignment unit, a local window cross attention unit, a cross-modal difference enhancement unit, and a feature refinement unit. The learnable offset alignment unit uses visible light features and infrared features to perform bimodal feature alignment spatial correction. The local window cross attention unit introduces a cross attention mechanism to perform cross-modal bidirectional information interaction. The cross-modal difference enhancement unit is used to mine the difference information between the two modes and perform complementary enhancement processing to obtain enhanced visible light features and enhanced infrared features. The feature refinement unit uses channel attention and spatial attention to perform feature refinement processing.

5. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 4, characterized in that: The learnable offset alignment unit acquires visible light features. With infrared features The absolute difference characteristics between Visible light characteristics With infrared features Absolute difference characteristics The features are concatenated along the channel dimension and then input into a three-layer (3×3) stacked convolutional offset prediction network. The offset prediction network then predicts the offset features. ; Apply offset constraints to limit the offset magnitude The expression is as follows: , The tanh function is a hyperbolic tangent constraint. The preset maximum allowable offset; the learnable offset alignment unit is constructed with a basic sampling grid in the GridSample normalized coordinate system. This will limit the offset magnitude. The deformed sampling mesh is obtained by superimposing it in the normalized coordinate space under the normalized coordinate system. ; Based on the deformed sampling mesh Differentiable bilinear grid sampling method is used for infrared features Resampling is performed to obtain aligned infrared features The expression is as follows: , This indicates that GridSample is a bilinear grid sampling process.

6. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 5, characterized in that: The local window cross-attention unit utilizes a cross-attention mechanism to perform linear projection using visible light features as queries and infrared features as keys, thereby enhancing the visible light features with infrared heat source information. The local window cross-attention unit also utilizes a cross-attention mechanism to perform linear projection using infrared features as queries and visible light features as keys, thereby enhancing the infrared features with visible light texture and color information. The local window cross-attention unit includes a visible light branch and an infrared branch for bidirectional flow of infrared and visible light features, expressed as follows: , ,in For the aligned infrared features, It has visible light characteristics. , These are the scaling factor parameters, This represents the LocalCrossAttention function. To enhance infrared signature, To enhance visible light characteristics.

7. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 4, characterized in that: Visible light characteristics With infrared features Feature offset alignment processing is performed. The cross-modal difference enhancement unit includes a cross-channel reweighting stage and a directed difference compensation stage. The cross-channel reweighting stage method includes the following: calculating channel attention weights for infrared and visible light features respectively, and performing nonlinear transformation through a reweighting network to obtain infrared weights respectively. and visible light weight The aligned infrared and visible light features are added together to calculate the shared channel weight. Utilizing visible light weighting and shared channel weight The expression for cross-channel modulation of infrared features is as follows: , For the infrared features modulated across channels; using Infrared features and shared channel weight The expression for cross-channel modulation of visible light features is as follows: , The visible light characteristics are those obtained after cross-channel modulation.

8. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 7, characterized in that: The directed difference compensation stage method includes: calculating the directed difference features between the visible light features and infrared features modes respectively, and obtaining infrared compensation information by weighting the features through channel attention. and visible light compensation information The expression is as follows: , ,in This refers to the directional difference characteristic of visible light relative to infrared light. This refers to the directional difference characteristics of infrared light relative to visible light. Channel attention is used to weight the differential features; the results of cross-channel reweighting and directed difference compensation are added to the original features through a learnable channel-level scaling factor for feature enhancement, as shown in the following expression: , , ~ These are the learnable channel-level scaling parameters. Enhanced infrared signature or enhanced infrared signature; This refers to the enhanced visible light characteristics or the enhanced visible light characteristics.

9. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 4, characterized in that: The feature refining unit includes two processing stages: channel refining and spatial refining, which enhances infrared features. With enhanced visible light characteristics The input feature refinement unit performs feature refinement processing. Channel refinement uses a channel attention mechanism to generate channel weights, which are multiplied with the input features channel by channel to highlight important channels and suppress redundant channels. Spatial refinement calculates the mean and maximum values ​​of the channel-refined features along the channel dimension and performs feature concatenation. After feature concatenation, spatial attention weights are generated through a convolutional layer and multiplied with the features pixel by pixel to highlight the target region and suppress background noise.

10. The target detection and recognition method based on cross-modal feature alignment and cross-fusion according to claim 1, characterized in that: In method S3, the dual-stream neck network has corresponding feature path aggregation networks for enhancing visible features and enhancing infrared features. Each feature path aggregation network includes two paths: top-down and bottom-up. The top-down path upsamples and progressively transmits high semantic features from deep layers to shallow layers and then splices and fuses them. The bottom-up path downsamples and progressively transmits high-resolution features from shallow layers to deep layers and then splices and fuses them with the output of the corresponding layer of the top-down path. The two paths of the feature path aggregation network ultimately output three scales of enhanced features. The dual-stream neck network obtains the three scales of enhanced features corresponding to the enhanced visible features and the three scales of enhanced infrared features.

Images