A Multispectral Pedestrian Detection Method Based on Cross-Modal Feature Calibration and Dynamic Fusion

Abstract

This invention relates to a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion, comprising: step S1, constructing a dual-stream feature extraction network to extract features from visible light and infrared images; step S2, constructing a cross-modal feature calibration sub-network; step S3, constructing a multimodal dynamic fusion sub-network; step S4, constructing a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; step S5, training the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; and step S6, using the multispectral pedestrian detection network trained in step S5 to perform pedestrian detection. This invention can eliminate modal redundancy, highlight the complementarity between modalities, facilitate better multimodal feature fusion, and thus improve multispectral pedestrian detection capabilities.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and pedestrian detection technology, and in particular to a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion. Background Technology

[0002] Pedestrian detection technology automatically detects pedestrian targets in images or videos using computer algorithms, and it is an important research direction in the field of computer vision. With the rapid development of artificial intelligence and deep learning technologies, pedestrian detection technology has broad application prospects in traffic management, intelligent monitoring, autonomous driving, and other fields.

[0003] Traditional pedestrian detection methods rely solely on single-modal information, resulting in poor detection outcomes. Optical images provide the most intuitive representation of human movement but are easily affected by lighting conditions, while infrared images can still be used for detection in low-light conditions but lack texture information. In recent years, deep learning-based multispectral pedestrian detection has become a cutting-edge research direction in the field and has achieved significant breakthroughs. This is mainly due to the powerful feature extraction capabilities of deep neural networks, which can effectively extract rich features of pedestrians from different modalities. However, deep learning-based multispectral pedestrian detection still faces the following challenges:

[0004] 1) Multimodal complementary information is difficult to extract. Different modal information is complementary, but it also has modal redundancy. Current mainstream multispectral pedestrian detection technologies still cannot effectively eliminate the interference of modal-related noise and cannot fully extract the complementary information between different modalities.

[0005] 2) The multimodal complementary information fusion method is too simplistic. Existing multispectral pedestrian detection methods simply add / cascade multimodal features, ignoring the complex and diverse complementary characteristics between different modalities. This makes the existing static fusion methods inadequate for pedestrian detection in complex scenes.

[0006] Due to the complex and diverse complementarity and redundancy between visible light and infrared images, current multispectral pedestrian detection methods are still unable to fully explore and utilize the effective information between different modal images for the interpretation task. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention aims to provide a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion, which can fully utilize complex and diverse complementary information to achieve better detection performance.

[0008] To achieve the above-mentioned objectives, this invention provides a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion, comprising the following steps:

[0009] Step S1: Construct a dual-stream feature extraction network to extract features from visible light and infrared images;

[0010] Step S2: Construct a cross-modal feature calibration sub-network;

[0011] Step S3: Construct a multimodal dynamic fusion subnetwork;

[0012] Step S4: Construct a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion;

[0013] Step S5: Train a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion;

[0014] Step S6: Perform pedestrian detection using the multispectral pedestrian detection network trained in step S5 based on cross-modal feature calibration and dynamic fusion.

[0015] According to the technical solution of the present invention, step S1 specifically includes:

[0016] Acquire pedestrian detection data from visible light and infrared images, and construct a paired dataset of visible light and infrared images;

[0017] The dataset is divided into training and testing sets according to a preset ratio;

[0018] The CSPDarknet53 backbone network is used as the feature extraction network for visible light and infrared images. The training set containing visible light and infrared images is input into the feature extraction network to form multi-scale visible light and infrared feature maps, and output visible light and infrared image feature pairs with multiple feature extraction levels.

[0019] According to the technical solution of the present invention, in step S2, at the multiple feature extraction levels output in step S1, the two modes of visible light and infrared mutually utilize the global and local information of the other mode to calibrate the current mode features from both channel and spatial dimensions, eliminate mode-related noise, and obtain complementary visible light calibration feature maps and infrared calibration feature maps.

[0020] According to the technical solution of the present invention, in step S3, the visible light calibration feature map and the infrared calibration feature map are aggregated into a visible light and infrared fused feature map by a dynamic fusion method.

[0021] According to the technical solution of the present invention, in step S4, a feature pyramid structure is used to fuse a high-level feature map with rich semantic information and a low-level feature map with localization information to obtain three-level feature maps with multi-scale target detection capability. Then, a detection head is used to output the confidence score and coordinates of each pixel on the feature map as a pedestrian.

[0022] According to the technical solution of the present invention, in step S5, the training set in step S1 is used to train the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion. Based on the stochastic gradient descent optimization algorithm, a learning rate decay strategy is set to update the parameters of the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion constructed in step S4 until convergence.

[0023] According to the technical solution of the present invention, step S5 further includes: using the test set in step S1 to verify the trained multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion.

[0024] According to the technical solution of the present invention, in step S2, the current modal characteristics are calibrated from two dimensions: channel and space, specifically including:

[0025] For channel calibration, V and T are embedded along the spatial dimension into a visible light channel attention vector. and infrared channel attention vector To capture the correlation between channels and eliminate redundancy, where, Representing visible light image features, To represent the features of an infrared image, then:

[0026]

[0027] Where || represents the channel concatenation operation, δ represents the sigmoid activation function, and F split This represents the channel splitting function, which is the inverse operation of ||.

[0028] Using two embedding methods: global context and local context Local context embedding F l C (X) If pointwise convolution is used to emphasize the channel response at each spatial location, then:

[0029] F l C (X)=PWConv l2 (relu(PWConv l1 (X)))

[0030] Where relu is the Relu activation function, PWConv l2 and PWConv l1 It is a pointwise convolution, with kernel sizes of 2C / r*2C*1*1 and 2C*2C / r*1*1, where r is the channel compression ratio and global context is used. Embedding uses a spatial dimension-based global average pooling method:

[0031]

[0032] Among them, F swap It is spatial dimension average pooling, PWConv g2 and PWConv g1 The kernel size and PWConv l2 and PWConv l1 Keep them consistent;

[0033] Finally, the visible light image features V after channel calibration cr With infrared image features V cr Represented as:

[0034]

[0035]

[0036] For space calibration, V cr and T cr Embedded along the channel dimension into a visible light spatial attention vector and infrared spatial attention vector To capture spatial correlation and eliminate redundancy, we have:

[0037]

[0038] Channel embedding includes two embedding methods: global context and local context.

[0039]

[0040] For global context Channel average pooling F cwap Each modal feature is compressed and concatenated along the channel dimension to obtain the spatial embedding:

[0041]

[0042] Where Conv g2 and Conv g1 These are convolution operations, with kernel sizes of 1*2*3*3 and 2*1*3*3 respectively;

[0043] For the local context F l S (V cr ,T cr ), employing group convolution and interleaving splicing operations To obtain spatial embedding:

[0044]

[0045] Among them, group convolution GConv l2 and GConv l1 The kernel sizes are C*2*3*3 and 2C*1*3*3, respectively;

[0046] Finally, the spatially calibrated visible light image features V sr With infrared image features V sr for:

[0047]

[0048]

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] This invention proposes a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion. The method employs a cross-modal feature calibration approach, which uses an attention mechanism to allow multimodal features to mutually utilize global and local information from another modality in both spatial and channel dimensions to calibrate the features of the current modality. This eliminates modal redundancy, highlights the complementarity between modalities, and facilitates better multimodal feature fusion, thereby improving the multispectral pedestrian detection capability.

[0051] Furthermore, this invention employs a multimodal feature dynamic fusion method, which can adaptively adjust the weights of various static convolution kernels according to multimodal features, better capture the complex complementary information between multimodal features in the convolution kernel space, thereby improving the fusion capability of multimodal features and enhancing the detection performance of multispectral pedestrian detection. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0053] Figure 1 The illustration shows a schematic diagram of a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion according to an embodiment of the present invention;

[0054] Figure 2 This diagram illustrates a cross-modal feature calibration subnetwork according to one embodiment of the present invention.

[0055] Figure 3 This diagram illustrates a multimodal feature dynamic fusion subnetwork according to an embodiment of the present invention.

[0056] Figure 4 This illustration shows the pedestrian detection visualization results on the LLVIP dataset using a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion.

[0057] Figure 5 The flowchart illustrates a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion provided in one embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] like Figures 1 to 5 As shown, this invention provides a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion. The following uses the LLVIP dataset as an example to illustrate the usage process of the visible-infrared pedestrian detection method, including the following steps:

[0060] Step S1: Construct a dual-stream feature extraction network to extract features from visible light and infrared images.

[0061] The LLVIP dataset is a paired dataset of visible light and infrared images, suitable for computer vision tasks such as image fusion, object detection, image translation, and generation in low-light scenes. The LLVIP dataset contains 33,672 images (16,836 image pairs), each 512×640 pixels in size. Most of the images were taken in very dark scenes, and all images are strictly aligned in both time and space.

[0062] In the visible-infrared pedestrian detection task, the LLVIP dataset was randomly divided into training and testing sets in a 7:3 ratio. For each data pair in the dataset, the CSPDarknet53 backbone network was used as the feature extraction network for visible and infrared images, outputting five levels of visible and infrared image feature pairs.

[0063] Step S2: Construct a cross-modal feature calibration sub-network.

[0064] like Figure 2 As shown, starting from the second level, both channel calibration and spatial calibration are used to calibrate the visible light and infrared image feature pairs and then input them into the feature extraction network of the next level. Representing visible light image features, Indicates infrared image features.

[0065] 1) Channel calibration

[0066] V and T are embedded along the spatial dimension into a visible light channel attention vector. and infrared channel attention vector To capture the correlation between channels and eliminate redundancy, where, Representing visible light image features, To represent the features of an infrared image, then:

[0067]

[0068] Where || represents the channel concatenation operation, δ represents the sigmoid activation function, and F split This represents the channel splitting function, which is the inverse operation of ||.

[0069] Using two embedding methods: global context and local context Local context embedding F l C (X) If pointwise convolution is used to emphasize the channel response at each spatial location, then:

[0070] F l C (X)=PWConv l2 (relu(PWConv l1 (X)))

[0071] Where relu is the Relu activation function, PWConv l2 and PWConv l1 It is a pointwise convolution, with kernel sizes of 2C / r*2C*1*1 and 2C*2C / r*1*1, where r is the channel compression ratio, which defaults to 4, and the global context is used. Embedding uses a spatial dimension-based global average pooling method:

[0072]

[0073] Among them, F swap It is spatial dimension average pooling, PWConv g2 and PWConv g1 The kernel size and PWConv l2 and PWConv l1 Keep them consistent;

[0074] Finally, the visible light image features V after channel calibration cr With infrared image features V cr Represented as:

[0075]

[0076]

[0077] 2) Spatial calibration

[0078] Similar to channel calibration, V cr and T cr Embedded along the channel dimension into a visible light spatial attention vector and infrared spatial attention vector To capture spatial correlation and eliminate redundancy, we have:

[0079]

[0080] Channel embedding includes two embedding methods: global context and local context.

[0081]

[0082] For global context Channel average pooling F cwap Each modal feature is compressed and concatenated along the channel dimension to obtain the spatial embedding:

[0083]

[0084] Where Conv g2 and Conv g1 These are convolution operations, with kernel sizes of 1*2*3*3 and 2*1*3*3 respectively;

[0085] For the local context F l S (V cr ,T cr ), employing group convolution and interleaving splicing operations To obtain spatial embedding:

[0086] Among them, group convolution GConv l2 and GConv l1 The kernel sizes are C*2*3*3 and 2C*1*3*3, respectively;

[0087] Finally, the spatially calibrated visible light image features V sr With infrared image features V sr for:

[0088]

[0089]

[0090] Step S3: Construct a multimodal dynamic fusion subnetwork.

[0091] like Figure 3 As shown, after obtaining the multimodal calibration features (visible light calibration feature map and infrared calibration feature map), a dynamic fusion method is used to aggregate them into a single feature map, which is beneficial for subsequent detection. Specifically, a set of linear weights of static convolutional kernels is used as dynamic convolution to fuse the multimodal features:

[0092]

[0093] in, and These represent the i-th static convolution kernel, This represents a convolution operation, where K is the number of convolution kernels. i It is a linear weighting factor, usually learned by the network, to achieve the ability to dynamically capture complex complementary information between multimodal features:

[0094] a=δ(PWConv d2 (relu(PWConv d1 (F swap (V+T)))));

[0095] Among them, PWConv d2 and PWConv d1 The convolution kernel sizes are 2C / r*2C*1*1 and K*2C*1*1, respectively.

[0096] Step 4: Construct a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion.

[0097] After obtaining the fused feature maps of visible light and infrared light, the PANet feature pyramid structure is adopted. By further fusing high-level feature maps with rich semantic information and low-level feature maps with localization information, three-level feature maps with multi-scale target detection capabilities are obtained. Then, a detection head is used to output the confidence score and coordinates of each pixel on the feature map as a pedestrian.

[0098] Step S5: Train a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion, and perform numerical verification.

[0099] A multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion was trained on the LLVIP dataset training set. The dual-stream backbone network was initialized using CSPDarknet-53 pre-trained on the COCO dataset to accelerate convergence. Stochastic gradient descent was used for iterative training for 20 epochs with a batch size of 16. The initial learning rate was set to 0.005, and the weight decay was set to 0.0005. In the post-processing algorithm, the intersection-union ratio (IUU) threshold between bounding boxes was set to 0.5 to filter out redundant boxes.

[0100] Step S6: Perform pedestrian detection using the multispectral pedestrian detection network trained in step S5 based on cross-modal feature calibration and dynamic fusion.

[0101] Table 1 below compares the mean average accuracy (mAP) on the LLVIP dataset test set between the basic detection method (excluding the cross-modal feature calibration and multimodal feature dynamic fusion sub-network) and the detection method based on cross-modal feature calibration and dynamic fusion. Compared with the basic detection method, the detection method based on cross-modal feature calibration and dynamic fusion achieves performance improvements of 1%, 2.5%, and 2% on the three metrics of mAP50, mAP75, and mAP50:95, respectively, and ultimately achieves a detection result of 97.3% mAP50. These improvements demonstrate that our proposed cross-modal feature calibration and multimodal feature dynamic fusion method can utilize adaptive calibration of multimodal features, eliminate modal redundancy noise, and thus better fuse multimodal features to leverage the complementarity between multimodal information, significantly improving the detection performance of the multispectral pedestrian detection network (partial visualization results are shown in [link]). Figure 4 ).

[0102] Basic Methods 96.3％ 72.2％ 63.0％ Improvement methods 97.3％ 74.7％ 65.0％

[0103] Table 1

[0104] This invention discloses a multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion, comprising: step S1, constructing a dual-stream feature extraction network to extract features from visible light and infrared images; step S2, constructing a cross-modal feature calibration sub-network; step S3, constructing a multimodal dynamic fusion sub-network; step S4, constructing a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; step S5, training the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; and step S6, using the multispectral pedestrian detection network trained in step S5 to perform pedestrian detection. The cross-modal feature calibration method utilizes an attention mechanism to allow multimodal features to mutually utilize global and local information from another modality in both spatial and channel dimensions to calibrate the features of the current modality, thereby eliminating modal redundancy, highlighting the complementarity between modalities, facilitating better multimodal feature fusion, and thus improving multispectral pedestrian detection capabilities.

[0105] Furthermore, this invention employs a multimodal feature dynamic fusion method, which can adaptively adjust the weights of various static convolution kernels according to multimodal features, better capture the complex complementary information between multimodal features in the convolution kernel space, thereby improving the fusion capability of multimodal features and enhancing the detection performance of multispectral pedestrian detection.

[0106] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A multi-spectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion, characterized in that, Includes the following steps: Step S1: Construct a dual-stream feature extraction network to extract features from visible light and infrared images; Step S2: Construct a cross-modal feature calibration sub-network; In step S2, at the multiple feature extraction levels output in step S1, the two modes of visible light and infrared mutually utilize the global and local information of the other mode to calibrate the current mode features from both channel and spatial dimensions, eliminating mode-related noise and obtaining complementary visible light calibration feature maps and infrared calibration feature maps; wherein, calibrating the current mode features from both channel and spatial dimensions specifically includes: For channel calibration, V and T are embedded along the spatial dimension into a visible light channel attention vector. and infrared channel attention vector To capture the correlation between channels and eliminate redundancy, where, Representing visible light image features, To represent the features of an infrared image, then: in, This indicates a channel splicing operation. This represents the sigmoid activation function. The channel splitting function is... The reverse operation; Using two embedding methods: global context and local context Local context embedding By employing pointwise convolution to emphasize the channel response at each spatial location, we have: Where relu is the relu activation function. and It is pointwise convolution, and the kernel sizes are respectively and r is the channel compression ratio, global context Embedding uses a spatial dimension-based global average pooling method: ； in, It is spatial dimension average pooling. and kernel size and and Keep them consistent; Finally, the visible light image features after channel calibration Features of infrared images Represented as: ； For space calibration, and Embedded along the channel dimension into a visible light spatial attention vector and infrared spatial attention vector To capture spatial correlation and eliminate redundancy, we have: Channel embedding includes two embedding methods: global context and local context. ； For global context Channel average pooling is used Each modal feature is compressed and concatenated along the channel dimension to obtain the spatial embedding: ； in and It is a convolution operation, and its convolution kernel size is respectively and ; For local context Using group convolution and interleaving splicing operations To obtain spatial embedding: ； Among them, group convolution and The kernel sizes are respectively and ; Finally, spatially calibrated visible light image features Features of infrared images for: ； Step S3: Construct a multimodal dynamic fusion subnetwork; Step S4: Construct a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; through the feature pyramid structure, fuse high-level feature maps with rich semantic information and low-level feature maps with localization information to obtain three-level feature maps with multi-scale target detection capabilities; then use a detection head to output the confidence score and coordinates of each pixel on the feature map as a pedestrian. Step S5: Train a multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion; Step S6: Perform pedestrian detection using the multispectral pedestrian detection network trained in step S5 based on cross-modal feature calibration and dynamic fusion.

2. The multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion according to claim 1, characterized in that, Step S1 specifically includes: Acquire pedestrian detection data from visible light and infrared images, and construct a paired dataset of visible light and infrared images; The dataset is divided into training and testing sets according to a preset ratio; The CSPDarknet53 backbone network is used as the feature extraction network for visible light and infrared images. The training set containing visible light and infrared images is input into the feature extraction network to form multi-scale visible light and infrared feature maps, and output visible light and infrared image feature pairs with multiple feature extraction levels.

3. The multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion according to claim 2, characterized in that, In step S3, the visible light calibration feature map and the infrared calibration feature map are combined into a visible light and infrared fused feature map using a dynamic fusion method.

4. The multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion according to claim 1, characterized in that, In step S5, the training set from step S1 is used to train the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion. Based on the stochastic gradient descent optimization algorithm, a learning rate decay strategy is set to update the parameters of the multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion constructed in step S4 until convergence.

5. The multispectral pedestrian detection method based on cross-modal feature calibration and dynamic fusion according to claim 4, characterized in that, Step S5 further includes: using the test set from step S1 to validate the trained multispectral pedestrian detection network based on cross-modal feature calibration and dynamic fusion.

Images