Pedestrian target detection method based on multiple modes

Abstract

The invention provides a pedestrian detection method based on deep learning and in combination with cross-modal adaptive feature fusion and alignment. According to the method, features are extracted through a dual-backbone network, a Transform-based feature fusion module TFM is designed, and a long-distance context dependency relationship is modeled through a cross-modal attention mechanism. In the input stage, features of visible light and infrared images are extracted through a double-backbone network, the obtained features are fused through a TFM module and then sent to a neck and head network for reasoning, and a detection result is output. The TFM module is mainly composed of a cross-modal feature fusion module CFFM and a feature alignment module FAM. The CFFM integrates multi-modal features through a feature aggregation pooling unit FAPM, and captures cross-modal association by using a Transform mechanism in combination with a plurality of CPSABlocks. And the FAM realizes alignment of local features and global semantics by means of deformable convolution. According to the invention, through a dual-backbone structure and introduction of the TFM feature fusion module, the precision, stability and cross-scene generalization ability of multi-modal pedestrian target detection are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and relates to a detection network based on deep learning and combining cross-modal adaptive feature fusion and alignment, which is used for improving cross-modal pedestrian target detection precision and can be widely applied to intelligent systems requiring robust perception. BACKGROUND

[0002] Although some existing models extract multi-modal features through double backbone networks, design different network structures and feature fusion modules, and optimize the integration effect of multi-modal features to a certain extent, there are two challenges in the fusion stage: on the one hand, in the fusion process, the phenomenon of strong modal feature suppressing weak modal information is prone to occur, for example, in the night scene, the significant heat signal features in the infrared image cover the weak texture and structure information in the visible light image, resulting in the loss of important details in the fusion stage; on the other hand, due to the fusion strategy, the features lack local or global information, or due to the different field of view ranges of different architectures in the fusion stage, feature mapping conflicts, i.e. the "spatial misplacement" problem, are prone to occur.

[0003] In view of the above problems, the present application provides an integrated solution capable of realizing inter-modal dynamic adaptive fusion and cross-modal feature space alignment, so as to obtain information balanced, spatial consistent high-quality fusion features, and improve the stability and precision of the multi-modal detection model under various illumination conditions. SUMMARY

[0004] In order to solve the problems of strong modal feature suppressing weak modal information and spatial misplacement caused by feature mapping conflict in multi-modal target detection, the present application provides a multi-modal detection method based on cross-modal adaptive feature fusion and alignment. The method uses two independent backbone networks to extract differentiated features in visible light and infrared images, and designs a feature fusion module TFM to realize feature fusion between different modalities.

[0005] In the first aspect, the present application provides a detection network based on deep learning and combining cross-modal adaptive feature fusion and alignment, which is used for multi-modal pedestrian target detection. The improvement lies in using a double backbone structure to extract features of visible light and infrared images respectively, and introducing a TFM module for cross-modal feature fusion and alignment.

[0006] Secondly, this invention provides a detection network based on deep learning and combined with cross-modal adaptive feature fusion and alignment for multimodal pedestrian target detection. The TFM module mainly includes a cross-modal feature fusion module (CFFM) and a feature alignment module (FAM). The CFFM module is used to achieve effective fusion of visible light and infrared image features; while the FAM module focuses on coordinating the local features extracted by CNN and the global context modeled by Transformer, thereby alleviating the misalignment between features.

[0007] Thirdly, this invention provides a detection network based on deep learning and combined with cross-modal adaptive feature fusion and alignment for multimodal pedestrian target detection. The CFFM module makes two key improvements to the C2PSA module. First, a Feature Aggregation Pooling (FAPM) module is introduced, extending its single-modal input structure to a form capable of simultaneously receiving and processing multimodal data. This module combines average pooling and max pooling operations, preserving the texture details and contour information of features without introducing additional parameters, and compressing the spatial dimension of the feature map, thereby efficiently aggregating infrared and visible light features and reducing information loss. Second, the PSABlock in the original structure is replaced with CPSABlock. CPSABlock achieves cross-modal feature interaction through its internal Transformer fusion mechanism and is designed with a learnable weight module, using trainable parameters to dynamically adjust the contribution of each modality, thereby avoiding the strong modality features from masking or interfering with the auxiliary information provided by another modality, thus achieving more balanced and effective feature fusion.

[0008] Fourthly, this invention provides a detection network based on deep learning and combined with cross-modal adaptive feature fusion and alignment for multimodal pedestrian target detection. The FAM module employs deformable convolution, achieving dynamic adjustment and deformation of the feature structure through adaptive learning of sampling offsets at various positions in the feature map. This module can collaboratively integrate local details captured by CNNs with global contextual information modeled by Transformers, thereby effectively alleviating the misalignment problem between features and enhancing the network's ability to model the correlation between feature regions at different levels.

[0009] The beneficial effects of this invention are: This invention proposes a network model that integrates cross-modal feature adaptive fusion and feature alignment mechanisms for multimodal pedestrian target detection. Specifically, a dual-backbone network architecture is introduced to simultaneously extract features from visible light and infrared images. The extracted features are processed by the cross-modal adaptive feature fusion module (CFFM) and the feature alignment module (FAM) within the TFM module to achieve effective multimodal feature adaptive fusion and spatial alignment. Experimental results show that this method improves both average accuracy and detection speed compared to multi-class detection models. Attached Figure Description

[0010] Referring to the accompanying drawings, further objects, functions, and advantages of the present invention will be clarified through the following description of embodiments of the invention, wherein: Figure 1 The TFM-YOLO backbone network structure provided in this embodiment of the invention; Figure 2 The TFM structure provided in the embodiments of the present invention; Figure 3 This refers to the C2PSA module network structure in YOLOv11 supported by existing technology. Figure 4 The CFFM module network structure provided in this embodiment of the invention; Figure 5 The FAM module network structure provided in this embodiment of the invention; Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0012] The core of this invention lies in providing a multimodal pedestrian target detection method. For example... Figure 1 The improved dual-backbone feature extraction network (TFM-YOLO) shown below has the following complete workflow: First, dual-modal data input and feature extraction: The registered and aligned visible light RGB image and infrared IR image are used as the system input, which can be denoted as follows: and These two images are input in parallel into a dual-backbone feature extraction module. This module contains two structurally identical independent backbone network branches, one dedicated to processing visible light images and the other to processing infrared images. Each backbone network adopts a YOLOv11-based backbone structure, sequentially passing through five layers from C1 to C5. Subsequently, multi-scale feature maps are extracted from the last three layers (C3, C4, and C5) of each backbone network.

[0013] For the visible light branch, the set of output feature maps is denoted as . For the infrared branch, the set of output feature maps is denoted as... These feature maps from different levels form the basis for subsequent fusion processing.

[0014] The next step is feature fusion and alignment: Next, features of the same scale from both branches are fed in pairs into the Feature Fusion and Alignment (TFM) module. For example... Figure 2 As shown, the TFM module consists of a cross-modal feature fusion module (CFFM) and a feature alignment module (FAM) connected sequentially.

[0015] The CFFM module is responsible for achieving deep fusion of dual-modal features, and its structure is as follows: Figure 4 As shown, the paired visible light and infrared feature maps first enter the Feature Aggregation Pooling (FAPM) module. This module performs both max pooling and average pooling operations on each input feature map simultaneously, then concatenates the two pooling results and processes them through a lightweight multilayer perceptron (MLP). This operation effectively aggregates the texture details and contour information of the features while downsampling the feature map spatial size by a factor of S, resulting in a preliminary aggregated feature representation. Features after aggregation The data is fed into a fusion unit consisting of multiple CPSABlocks. Within each CPSABlock, cross-modal attention interaction is first achieved through a Transformer-based fusion block, the TransformerFusion Block. Specifically, this mechanism converts features from one modality into keys and values, and features from another modality into queries. The information in the values ​​is weighted and aggregated by calculating the similarity between the queries and keys, thus achieving information complementarity between modalities. Subsequently, a dynamic weight parameter is generated through a learnable weight module, LearnWeight. and according to the formula The features are adaptively weighted and fused. This design allows the network to automatically adjust the contribution ratio of the two modalities based on the specific content of the input image, preventing features from one modality from becoming absolutely dominant, thus outputting more balanced and effective preliminary fused features. .

[0016] Features obtained by CFFM fusion It is then fed into the Feature Alignment Module (FAM), whose structure is as follows: Figure 5 As shown. The core of the FAM module is to use deformable convolutions to adjust the spatial structure of features. The module first passes the input features through an additional convolutional layer. The content predicts the 2D offset of each sampling point in the spatial location. Then, the deformable convolutional layer uses these predicted offsets to adaptively sample the feature map using an irregular grid. This process allows the receptive field of the convolutional operation to flexibly adapt to the actual shape and edges of the target, thereby effectively aligning the local detail features extracted by the CNN with the global context information modeled by the preceding Transformer module, reducing the representational differences between them, and finally outputting refined and aligned fused features. .

[0017] Finally, multi-scale prediction and result generation are performed: aligned multi-scale fused features obtained by the TFM module. The features are fed into the neck and detection layers of the network. The neck network further upsamples, downsamples, and fuses these multi-scale features to enhance the model's ability to detect targets of different sizes. The detection head then performs convolutional operations on the feature maps at each scale, directly predicting the bounding box coordinates, confidence scores, and class probabilities of the targets, ultimately outputting a complete list of pedestrian detection results.

[0018] To verify the effectiveness of the improved YOLOv11 network multimodal pedestrian target detection method provided by this invention, the present invention also provides the following experiments.

[0019] (I) Experimental Data and Experimental Environment The datasets used in this invention to verify the effectiveness of pedestrian target detection include the KAIST dataset (HWANG S, PARK J, KIM N. Multispectralpedestrian detection: Benchmark dataset and baseline [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1037-1045.), captured by a vehicle-mounted camera and applied to pedestrian detection in low-light conditions; and the LLVIP dataset (Jia X, Zhu C, Li M. LLVIP: Avisible-infrared paired dataset for low-light vision [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 3496-3504.), a multimodal pedestrian target detection dataset for low-light vision in low-light environments. The experimental platform uses Windows 11 Professional as the operating system, AMD Ryzen 7 7840H as the CPU, NVIDIA RTX 4060 (8GB VRAM) as the GPU, and Python 3.9, CUDA 11.6, and PyTorch 1.13 as the software.

[0020] (II) Experimental Setup and Evaluation Indicators The TFM-YOLO network structure designed in this invention was used to train the data. The batch size was set to 8 training samples, the number of training epochs was set to 300, and the learning rate was 0.01.

[0021] Deep learning-based object detection and recognition typically evaluates the detection performance of a single class of objects by calculating the model's average precision (AP) on the test set. This invention uses two metrics, mAP@0.5 and mAP@0.5:0.95, for this purpose. mAP@0.5 represents the average AP calculated for all images in each class when the IOU threshold is set to 0.5. The AP value refers to the area under the P (Precision) and R (Recall) curves. mAP@0.5:0.95 represents the mAP when the IOU threshold is between 0.5 and 0.95 (with a step size of 0.05, taking values ​​of 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, and 0.95).

[0022] (III) Experimental Results and Analysis The TFM-YOLO algorithm and other different model algorithms were trained and tested on the LLVIP dataset and the KAIST dataset, respectively. The results are shown in Table 1 and Table 2.

[0023] Table 1 Comparison of different models on the LLVIP dataset

[0024] Table 2 Comparison of different models on the KAIST dataset

[0025] Experimental results show that, compared with the latest YOLO algorithm in the same series and other mainstream pedestrian detection algorithms, the TFM-YOLO proposed in this invention exhibits superior overall performance across multiple evaluation metrics, achieving higher detection accuracy and effectively addressing the key issues of feature fusion and alignment in multimodal pedestrian detection. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for all-weather, all-time pedestrian target detection based on deep learning and adaptive feature fusion using dual-modal data sources, characterized in that: S1: Dual-backbone feature extraction goes through five different stages from C1 to C5 to achieve feature dimensionality reduction. The large-scale feature map is downsampled using a convolutional structure. The resulting feature map contains shallow detail information, deep semantic information, and structural information in sequence. S2: The features extracted from the last three levels of the dual backbone are processed through a 3-layer feature fusion module TFM based on the Transformer structure. The self-attention mechanism is used to mine the potential correlation between different modalities to obtain the weight matrix, thereby realizing the adaptive fusion of visible light and infrared features. S3: Output module, which integrates the above-mentioned fused features into the neck and head networks to obtain the detection label and confidence score of the pedestrian target.

2. The cross-modal pedestrian detection network according to claim 1, characterized in that, The feature fusion module CFFM and the feature alignment module FAM are integrated to construct the core of TFM. The core function of CFFM is to establish robust transmission and integration channels between different data streams, thereby enabling collaborative processing of heterogeneous modal features. FAM focuses on coupling local and global features of multiple modalities; its mechanism is to perform misalignment recalibration and spatial remapping of features through a dynamic convolutional structure. The main functions of the feature fusion module CFFM are jointly implemented by the feature aggregation and pooling module FAPM and the CPSABlock module.

3. The feature aggregation and pooling module simultaneously performs average pooling and max pooling operations on the input bimodal features, introduces a feature mapping factor S, and makes the size of the aggregated feature map C×H / S×W / S to realize multimodal feature aggregation, and inputs the result into the CPSABlock module; The FAPM module reduces the dimensionality of the feature space while maximizing the aggregation and condensation of infrared and visible light features, thereby minimizing information loss.

4. The cross-modal pedestrian detection network according to claim 3, characterized in that, The CPSABlock includes a TransformerFusionBlock for cross-modal feature fusion, and subsequently connected learnable weight modules.

5. The cross-modal pedestrian detection network according to claim 4, characterized in that, The TransformerFusionBlock employs an eight-parallel multi-head cross-modal attention mechanism, enabling the model to understand the correlation between infrared and visible light features from different perspectives.

6. The cross-modal pedestrian detection network according to claim 4, characterized in that, To enhance the texture of infrared thermal targets at night to compensate for the lack of visible light, and to optimize the blurred infrared contours by utilizing visible light details during the day, a learnable weight module is proposed. The learnable weight module introduces learnable parameters and adaptively adjusts learnable coefficients λ ranging from 0 to 1, enabling the network to adaptively allocate the fusion ratio of different modal features according to the actual situation of the input image pairs, thereby gradually optimizing the cooperation efficiency between modalities during continuous training. Meanwhile, to ensure that the auxiliary features of another modality are not obscured by the dominant modality, the module also sets a weight lower bound mechanism to avoid any modality being completely ignored during the fusion process, thus ensuring the stability and robustness of multimodal information complementarity.

7. The cross-modal pedestrian detection network according to claim 1, characterized in that, The Feature Alignment Module (FAM) based on dynamic convolution is mainly used to adaptively align the feature differences between visible light and infrared modes. Its core mechanism is to use deformable receptive fields to adapt to the spatial distribution and semantic context of different modal features, thereby achieving consistent expression of cross-modal features. The FAM module receives feature maps from two modes: After combining them, the feature maps are applied through two layers of convolution and a 2D offset is learned. Then, the concatenated features and attention weights att are used as input to the offset convolution, and the deformable convolution is guided to complete the structure-aware feature alignment. The FAM module is deployed in two key locations: aligning the visible light and infrared features input into the TFM module to obtain feature F, and aligning the features after feature fusion by the CFFM module with feature F.

Images