Low-light environment multispectral pedestrian detection method based on cross-modal attention fusion network

Abstract

A low-light environment multispectral pedestrian detection method based on a cross-modal attention fusion network comprises the following steps: constructing a pedestrian detection network comprising a double-flow feature extraction backbone network (CSPDarknet53), a multi-model aggregation Transformer network (MFT), a double-model feature aggregation network (DMFF) and a detection head at an offline stage, training by adopting a multispectral data set, and then, carrying out multi-spectral detection on the pedestrian detection network; and performing real-time image processing through the trained pedestrian detection network in the online stage. According to the method, better pixel-level attention can be generated for small objects, and feature representation under different scales is enhanced.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a multi-spectral pedestrian detection method in a low-light environment based on a cross-modal attention fusion network. Background Art

[0002] Existing technologies have several significant defects in multispectral pedestrian detection. Traditional detection methods are not adaptable enough to complex environments and low-light conditions, resulting in a significant drop in pedestrian recognition rates. Secondly, many existing models mainly rely on information from a single modality and fail to effectively integrate the advantages of multiple modal data, resulting in insufficient information utilization. Existing algorithms often lack flexibility in the feature extraction and fusion stages, and have difficulty coping with diverse scene changes. Some methods have high computational complexity and insufficient real-time performance, and cannot meet the rapid response requirements in practical applications. Many technologies perform poorly when dealing with occlusion and background interference, which further reduces the accuracy and robustness of detection. These defects limit the widespread application of multispectral pedestrian detection technology, and new solutions are urgently needed. Summary of the invention

[0003] In view of the fact that the convolutional layers of existing convolutional neural networks usually use smaller kernels and have limited receptive fields, making it difficult to capture the long-distance spatial dependencies of intra-spectral and inter-spectral images, as well as the problems of insufficient feature fusion of existing multimodal target following, poor object feature representation, and low detection accuracy due to occlusion and light changes, the present invention proposes a multispectral pedestrian detection method in low-light environments based on a cross-modal attention fusion network, which can generate better pixel-level attention to small objects and enhance feature representation at different scales.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multispectral pedestrian detection method in a low-light environment based on a cross-modal attention fusion network. A pedestrian detection network including a dual-stream feature extraction backbone network (CSPDarknet53), a multi-model aggregation Transformer network (MFT), a dual-model feature aggregation network (DMFF) and a detection head is constructed in the offline stage. After training with a multispectral dataset, real-time image processing is performed through the trained pedestrian detection network in the online stage.

[0006] The pedestrian detection network includes: a dual-stream feature extraction backbone network (CSPDarknet53), a multi-model aggregation Transformer network (MFT), a dual-model feature aggregation network (DMFF) and a detection head, wherein: CSPDarknet53 extracts bimodal image features from infrared images and visible light images; MFT fuses contextual information of different scales in CSPDarknet53 according to the bimodal image features and generates multi-scale features to improve the pixel-level attention distribution; DMFF deeply fuses multi-scale features to maximize the correlation between multi-scale features and greatly enhances the ability of features to express semantic information and geometric details; the detection head performs pedestrian recognition and positioning of fused features on the deeply fused features. Technical Effects

[0007] The present invention embeds a multi-model aggregation Transformer network into a dual-stream feature extraction backbone to respectively process the relationship between input features and input channels of different modules to effectively integrate the complementary information between multi-spectral modalities. At the same time, the DMFF module is introduced to perform deep fusion of multi-scale features, thereby maximizing the correlation between multi-scale features. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart of the method of the present invention;

[0009] Figure 2 Schematic diagram of the backbone network for dual-stream feature extraction;

[0010] Figure 3 Schematic diagram of the multi-model aggregation Transformer network;

[0011] Figure 4 Schematic diagram of the dual-model feature aggregation network. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, this embodiment relates to a multi-spectral pedestrian detection method in a low-light environment based on a cross-modal attention fusion network, including:

[0013] Step 1: construct a two-stream feature extraction backbone network to extract bimodal image features of infrared images and visible light images.

[0014] like Figure 2 As shown, the dual-stream feature extraction backbone network is implemented using a pair of parallel YOLOv5s, wherein the RGB path and IR path YOLOv5s respectively focus on color and texture and extract RGB features, and focus on temperature changes and extract thermal radiation features in the environment based on the input RGB image and IR image information.

[0015] In the second step, a multi-model aggregated Transformer network (MFT) is used to fuse contextual information of different scales to produce better pixel-level attention.

[0016] like Figure 3 As shown, the multi-model aggregation Transformer network includes: a self-attention module (SAB) unit, a self-aggregation module (SAM) unit, and a multi-layer perceptron (MLP) unit, wherein: the SAB unit converts the image into a sequence according to the feature information of the input image, and strengthens the representation and learning ability of the feature through local-global attention interaction, thereby realizing multi-layer learning of the sequence model, thereby extracting the semantic information of the position, context and dependency in the sequence; the SAM unit performs normalization and context feature extraction processing according to the input feature sequence information to obtain a weighted feature representation; the MLP unit performs linear transformation and nonlinear activation processing according to the input feature information to obtain a processed feature representation.

[0017] The fusion of context information of different scales is specifically: the fused features in the i-th layer in: H, W, and C represent the height, width, and number of channels of the feature map. SAB and SAM are the self-attention feature fusion function and self-aggregation feature fusion function with parameters, respectively. MLP is a multi-layer perceptron, and LN is LayerNorm. is the RGB feature map, For IR feature maps, they can process the spatial information of multi-scale input feature maps, effectively establish the relationship between input features of different modules and input channels, and reduce the interference of background noise.

[0018] The SAB unit uses Transformer to extract cross-modal semantic information: the input sequence is projected onto three weight matrices after passing through LayerNorm to calculate a set of queries, keys, and values ​​(Q, K, and V), specifically: Q = W Q φ x , K=W K φ x , V=W V φ x , where: W Q , W K and W V is the weight matrix. After the self-attention layer uses the scaled dot product between Q and K to calculate the attention weights, it multiplies these values ​​to infer in: is a scaling factor used to prevent the softmax function from converging to the region with the smallest gradient when the dot product becomes larger. Then it passes through a multi-layer perceptron MLP and finally obtains the output sequence

[0019] The SAM unit receives the output sequence After the Layer Norm is used to normalize each feature dimension of each sample, the mean of each feature is 0 and the variance is 1. After passing through a multi-layer perceptron MLP consisting of two linear layers and a Gelu activation function, the channel attention weight φ is generated through the sigmoid activation function. W ∈R C×1×1 , and get the output feature map φ ο , specifically: φ O =Wφ W +φ Z ,φ W =Sig mod(MLP(LN(φ Z )).

[0020] Step three, the feature map after feature extraction after MFT fusion is input into the dual-model feature aggregation network, and after dimensionality reduction and adaptive aggregation pooling based on convolution operation, cross-modal multi-scale feature deep fusion is performed to maximize the correlation between multi-scale features.

[0021] like Figure 4 As shown, the dual-model feature aggregation network includes: a spatial feature shrinkage (SFS) unit and two cross-modal feature enhancement (CFE) units, wherein: the SFS unit is based on the visible light and infrared feature maps F after fusion from the MFT R and F T ∈R H×W×C , perform dimensionality reduction processing based on convolution operation, reconstruct the dimension of feature map, convert the spatial information of feature into channel dimension, and then use 1×1 convolution operation to compress the channel dimension to obtain feature map T R ,T T ∈R HW×C .

[0022] The dimensionality reduction based on convolution operation refers to: F conv =conv 1×1 (Re shape (F)), where: F is the input feature map, F conv is a compressed feature map after 1×1 convolution.

[0023] The adaptive aggregation pooling means that in each pooling area, not only the average value is calculated, but also the maximum value is considered, thereby taking into account both background information and texture features, specifically: α =AvgPooling(F,S), F m =MaxPooling(F,S), F O =λ·Fα +(1-λ)·F m , where: F is the input feature map. S is the scale factor of the feature map, F α and F m They represent the feature maps compressed by AvgPooling and MaxPooling respectively. λ represents the weight between 0 and 1 and is a learnable parameter.

[0024] The cross-modal multi-scale feature deep fusion means that the two CFE units perform interactive feature processing according to the information of the RGB and IR feature maps, and obtain complementary information to enhance the RGB texture features and IR image features respectively.

[0025] The IR image features Where: T R and T T are the RGB and IR feature maps of the input CFE unit, Γ CFE-T (·) represents the IR branch CFE unit.

[0026] The interactive feature processing specifically includes:

[0027] Step a) Transform the IR modality T T The feature maps are projected into two independent matrices V T ,K T ∈R HW×C Then, the RGB mode T R The feature map is projected into another independent matrix Q T ∈R HW×C And a set of queries are generated, specifically: V T =T T W V ,K T =T T W K ,Q T =T R W Q , where: W V ,W K ,W Q ∈R C×C is the weight matrix.

[0028] Step b) After building the matrix through dot product operation, the softmax function is used to normalize the correlation scores. After expressing the similarity between different features of RGB and IR modalities, the vector is obtained by multiplying the matrix with the vector, and the RGB features are refined using the similarity between the modalities. In addition, a multi-head attention mechanism is also used to enable the model to better understand the correlation between RGB and IR features, specifically: T T′=α·Z T W O +β·V T , Γ CFR-T ({T R ,T T})=γ·T′ T +δ·FFN(T′ T ).

[0029] Step c) Transform the vector Z T It is reprojected back to the original space through nonlinear transformation and added to the input sequence through residual connection, where: W O ∈R C×C Represents the output weight matrix before the FFN layer.

[0030] Step d) uses a two-layer fully connected feedforward network (FFN) in the standard Transformer to further refine the global information. Improve the robustness and accuracy of the model and output enhanced IR image features

[0031] Step 4: The cross-transformed feature map obtained in step 3 is input into the detection head for pedestrian recognition and positioning of the final fused features.

[0032] After specific practical experiments, the results were implemented on a Linux server using the PyTorch 1.10.1 framework with 7 NVIDIA GeForce GTX1080 Ti GPUs. The training phase requires 300 rounds with a batch size of 21. The initial learning rate of the SGD optimizer is 1.0×10-2. In addition, the input size of the images used for training and testing is 640×640. The loss function uses the detectors of YOLOv5 and FCOS for detection. In the ablation study, the extended YOLOv5 framework is used as the default baseline for comparison. In the LLVIP dataset, the mAP50 indicator reached 97.1% and the mAP indicator reached 65.1%. The average average precision (mAP) reached 77.3% in the FLIR dataset.

[0033] Compared with the prior art, the present invention uses the Transformer model to perform global modeling of cross-modal features in the image fusion stage, and effectively captures the complex relationship between RGB and IR features through the self-attention mechanism. Secondly, not only is fusion performed in the middle stage of feature extraction, but further feature integration is also performed before entering the detection head in the later stage to ensure that the features of each stage can be fully utilized. This dual fusion strategy enables the model to utilize the rich features extracted in the early stage and optimize the results in the later stage, thereby improving the accuracy of recognition and positioning and having better adaptability to complex environments.

[0034] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multispectral pedestrian detection method in low-light environment based on a cross-modal attention fusion network, characterized in that: In the offline stage, a pedestrian detection network consisting of a two-stream feature extraction backbone network (CSPDarknet53), a multi-model aggregation Transformer network (MFT), a dual-model feature aggregation network (DMFF) and a detection head is constructed. After training with a multispectral dataset, real-time image processing is performed through the trained pedestrian detection network in the online stage.

2. The multispectral pedestrian detection method in low-light environment based on cross-modal attention fusion network according to claim 1 is characterized in that: The pedestrian detection network includes: a dual-stream feature extraction backbone network (CSPDarknet53), a multi-model aggregation Transformer network (MFT), a dual-model feature aggregation network (DMFF) and a detection head, wherein: CSPDarknet53 extracts bimodal image features from infrared images and visible light images; MFT fuses contextual information of different scales in CSPDarknet53 according to the bimodal image features and generates multi-scale features to improve the pixel-level attention distribution; DMFF deeply fuses multi-scale features to maximize the correlation between multi-scale features and greatly enhances the ability of features to express semantic information and geometric details; the detection head performs pedestrian recognition and positioning of fused features on the deeply fused features.

3. The multispectral pedestrian detection method in low light environment based on cross-modal attention fusion network according to claim 1 or 2 is characterized in that: include: Step 1: construct a dual-stream feature extraction backbone network to extract bimodal image features of infrared images and visible light images; Step 2: Use the Multi-Model Aggregation Transformer Network (MFT) to fuse contextual information of different scales to produce better pixel-level attention; Step 3: After MFT fusion and feature extraction, the feature map is input into the dual-model feature aggregation network, and after dimensionality reduction and adaptive aggregation pooling based on convolution operation, cross-modal multi-scale feature deep fusion is performed to maximize the correlation between multi-scale features; Step 4: The cross-transformed feature map obtained in step 3 is input into the detection head for pedestrian recognition and positioning of the final fused features.

4. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 2 or 3, characterized in that: The dual-stream feature extraction backbone network is implemented using a pair of parallel YOLOv5s, wherein the RGB path and IR path YOLOv5s respectively focus on color and texture and extract RGB features, and focus on temperature changes and extract thermal radiation features in the environment based on the input RGB image and IR image information.

5. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 2 or 3, characterized in that: The multi-model aggregation Transformer network includes: a self-attention module (SAB) unit, a self-aggregation module (SAM) unit, and a multi-layer perceptron (MLP) unit, wherein: the SAB unit converts the image into a sequence according to the feature information of the input image, strengthens the representation and learning ability of the feature through local-global attention interaction, and realizes multi-layer learning of the sequence model, thereby extracting the semantic information of the position, context and dependency in the sequence; the SAM unit performs normalization and context feature extraction processing according to the input feature sequence information to obtain a weighted feature representation; the MLP unit performs linear transformation and nonlinear activation processing according to the input feature information to obtain a processed feature representation; The fusion of context information of different scales is specifically: the fused features in the i-th layer in: H, W, and C represent the height, width, and number of channels of the feature map. SAB and SAM are the self-attention feature fusion function and self-aggregation feature fusion function with parameters, respectively. MLP is a multi-layer perceptron, and LN is LayerNorm. is the RGB feature map, For IR feature maps, they can process the spatial information of multi-scale input feature maps, effectively establish the relationship between input features of different modules and input channels, and reduce the interference of background noise.

6. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 5 is characterized in that: The SAB unit uses Transformer to extract cross-modal semantic information: the input sequence is projected onto three weight matrices after passing through LayerNorm to calculate a set of queries, keys, and values ​​(Q, K, and V), specifically: Q = W Q φ x , K=W K φ x , V=W V φ x , where: W Q , W K and W V is the weight matrix. After the self-attention layer uses the scaled dot product between Q and K to calculate the attention weights, it multiplies these values ​​to infer in: It is a scaling factor used to prevent the softmax function from converging to the area with the smallest gradient when the dot product becomes larger. Then it passes through a multi-layer perceptron MLP to get the output sequence. The SAM unit receives the output sequence After the Layer Norm is used to normalize each feature dimension of each sample, the mean of each feature is 0 and the variance is 1. After passing through a multi-layer perceptron MLP consisting of two linear layers and a Gelu activation function, the channel attention weight φ is generated through the sigmoid activation function. W ∈R C×1×1 , and get the output feature map φ ο , specifically: φ O =Wφ W +φ Z ,φ W =Sigmod(MLP(LN(φ Z )).

7. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 2 or 3, characterized in that: The dual-model feature aggregation network includes: a spatial feature shrinkage (SFS) unit and two cross-modal feature enhancement (CFE) units, wherein: the SFS unit is based on the visible light and infrared feature maps F after fusion from the MFT R and F T ∈R H ×W×C , perform dimensionality reduction processing based on convolution operation, reconstruct the dimension of feature map, convert the spatial information of feature into channel dimension, and then use 1×1 convolution operation to compress the channel dimension to obtain feature map T R ,T T ∈R HW×C .

8. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 7 is characterized in that: The dimensionality reduction based on convolution operation refers to: F conv =conv 1×1 (Reshape(F)), where: F is the input feature map, F conv is the compressed feature map after 1×1 convolution; The adaptive aggregation pooling means that in each pooling area, not only the average value is calculated, but also the maximum value is considered, thereby taking into account both background information and texture features, specifically: α =AvgPooling(F,S),F m =MaxPooling(F,S), F O =λ·F α +(1-λ)·F m , where: F is the input feature map, S is the scale factor of the feature map, F α and F m They represent the feature maps compressed by AvgPooling and MaxPooling respectively, and λ represents the weight between 0 and 1, which is a learnable parameter.

9. The method for multispectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 7 is characterized in that: The cross-modal multi-scale feature deep fusion means that the two CFE units perform interactive feature processing according to the information of the RGB and IR feature maps, and obtain complementary information to enhance the RGB texture features and IR image features respectively.

10. The method for multi-spectral pedestrian detection in low-light environment based on cross-modal attention fusion network according to claim 9 is characterized in that: The IR image features Where: T R and T T are the RGB and IR feature maps of the input CFE unit, Γ CFE-T (·) indicates IR branch CFE unit; The interactive feature processing specifically includes: Step a) Transform the IR modality T T The feature maps are projected into two independent matrices V T ,K T ∈R HW×C After a set of values ​​and keys are generated, the RGB mode T R The feature map is projected into another independent matrix Q T ∈R HW×C And a set of queries are generated, specifically: V T =T T W V ,K T =T T W K ,Q T =T R W Q , where: W V ,W K ,W Q ∈R C×C is the weight matrix; Step b) After building the matrix through dot product operation, the softmax function is used to normalize the correlation scores. After expressing the similarity between different features of RGB and IR modalities, the vector is obtained by multiplying the matrix with the vector. The similarity between the modalities is used to refine the RGB features and a multi-head attention mechanism is used to understand the correlation between RGB and IR features. Specifically: T′ T =α·Z T W O +β·V T , Γ CFR-T ({T R ,T T })=γ·T′ T +δ·FFN(T′ T ); Step c) Transform the vector Z T It is reprojected back to the original space through nonlinear transformation and added to the input sequence through residual connection, where: W O ∈R C×C Represents the output weight matrix before the FFN layer; Step d) uses a two-layer fully connected feedforward network (FFN) in the standard Transformer to further refine the global information to improve the robustness and accuracy of the model and output enhanced IR image features.

Images