A multispectral pedestrian detection method based on cross-modal feature decomposition

Abstract

The application discloses a multispectral pedestrian detection method based on cross-modal feature decomposition, comprising the following steps: (1) obtaining a multispectral pedestrian dataset composed of paired visible light and infrared pictures; (2) constructing a double-flow network by adopting a backbone network of two-way YOLOv8, and respectively extracting visible light modal features and infrared modal features; (3) obtaining visible light-infrared fusion features of different scales containing low-frequency information and high-frequency information; (4) fusing the obtained low-frequency global information features and high-frequency local information features; (5) performing multiscale fusion on the aggregated high-low frequency fusion features; (6) outputting a prediction frame and a corresponding confidence score; the application can extract high-frequency and low-frequency information between different modes by using a Transformer and a carefully designed CNN structure, and effectively fuse the high-frequency and low-frequency information; and the application further improves the accuracy of pedestrian detection in a low-light environment by adopting a neighboring attention mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer digital image processing, and particularly relates to a multispectral pedestrian detection method based on cross-modal feature decomposition. BACKGROUND

[0002] Pedestrian detection is an important technology in the fields of autonomous driving, automatic video monitoring and robots, etc. With the development of deep convolutional neural networks (CNN) in the field of target detection, considerable progress has been made in multispectral pedestrian detection. However, due to the sensitivity of visible light cameras to lighting conditions, even excellent detectors cannot accurately detect pedestrians in low-light environmental conditions, which limits the practical application of many technical solutions. In order to solve this problem, thermal infrared cameras have been introduced to capture the infrared radiation emitted by objects, which are not sensitive to light and weather changes, and thus can provide environmental supplementary information under adverse lighting conditions.

[0003] At present, some multispectral pedestrian detection methods based on deep learning have been proposed, such as methods based on convolutional neural networks (CNN), methods based on autoencoders (AE), methods based on generative adversarial networks (GAN) and methods based on Transformers. However, deep learning-based methods can produce observable results in most low-light scenes, but still have some shortcomings. First, convolutional neural network-based methods only extract local information in a relatively small receptive field, making it difficult to extract global information to generate high-quality image features. Second, Transformer-based methods use self-attention to model the global to effectively handle low-frequency global features, but use a relatively simple method to handle high-frequency local features, failing to effectively extract high-frequency local features in cross-modal. SUMMARY

[0004] The purpose of the present application is to provide a multispectral pedestrian detection method based on cross-modal feature decomposition that can fully extract and decompose high-frequency local features and low-frequency global features between multispectral features, and effectively fuse features of different frequencies to obtain high-quality cross-modal fusion features.

[0005] Technical solution: The multispectral pedestrian detection method based on cross-modal feature decomposition provided by the present application comprises the following steps:

[0006] (1) Obtain a multispectral pedestrian dataset composed of paired visible light and infrared pictures, divide it into a training set and a test set according to a ratio of M:N, and each visible light and infrared picture pair has the same number of picture channels

[0007] (2) Use a two-way YOLOv8 backbone network to build a dual-stream network to extract visible light modal features and infrared modal features respectively;

[0008] (3) Input the layered visible light output features and infrared output features from step (2) into the CTF module for feature extraction, decomposition and fusion to obtain visible light-infrared fusion features containing low-frequency and high-frequency information at different scales.

[0009] (4) Input the low-frequency global information features and high-frequency local information features obtained in step (3) into the high-low frequency information feature fusion module and fuse them according to the low-frequency and high-frequency features;

[0010] (5) Input the aggregated high and low frequency fusion features from step (4) into the Neck layer of the YOLOv8 network to perform multi-scale fusion;

[0011] (6) Input the multi-scale features obtained in step (5) into the detection head of the YOLOv8 network and output the prediction box and the corresponding confidence score.

[0012] Furthermore, the dataset obtained in step (1) is: Visible-Infrared Pairing Dataset LLVIP.

[0013] Further, step (2) is as follows: the visible light and infrared images are fed into a two-stream network to extract features; the two-stream network uses stacked convolutional layers to extract multimodal features; and three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the visible light branch of the two-stream network are used as visible light output features. Three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the infrared branch were used as infrared output features.

[0014] Furthermore, step (3) is specifically as follows: The visible light output characteristics at i = 1, 2, and 3 are... and infrared output characteristics The data is input into three CTF modules according to different layers; the CTF module consists of two parts: a feature decomposition module (FD) and a high-low frequency information feature fusion module.

[0015] Furthermore, the Feature Decomposition (FD) module is implemented as follows: The visible light output features from step 2 are... and infrared output characteristics The data are fed into two feature decomposition modules (FD) for processing. Each FD module consists of two parts: a low-frequency global information extraction module (Global) based on Transformer and a high-frequency local information extraction module (Local) based on CNN.

[0016] Furthermore, the low-frequency global information extraction process is as follows:

[0017] First, input features After passing through a LayerNorm normalization layer, the data is fed into the low-frequency global feature extraction module (Global) and the high-frequency local feature extraction module (Local) to extract the global feature. First, the query matrix Q is obtained directly through a fully connected layer, and then... The input is downsampled by a depthwise separable convolutional layer (DWConv); then, it is mapped through a fully connected layer to obtain the keyword matrix K and the value matrix V; the query matrix, keyword matrix, and value matrix are then fed as input to a dilated neighborhood attention module for long-range dependency modeling and global information extraction, followed by residual connections to correct the features; finally, it passes through a layer normalization layer (LayerNorm) and a multilayer perceptron (MLP) to obtain low-frequency global information features. The formula is as follows:

[0018]

[0019]

[0020]

[0021] Attn = Attention(Q,K',V')

[0022]

[0023] Z = MLP(Norm(Y)) + Y

[0024] Furthermore, the process of extracting local high-frequency information based on CNN is as follows:

[0025] First, input features The query matrix Q, keyword matrix K, and value matrix V are obtained through linear transformation. Then, Q, K, and V are input into a simple deep convolution with globally shared weights for local feature enhancement, resulting in Q', K', and V' with aggregated local information. Next, the Hadamard product of Q' and K' is calculated, and then passed through a multilayer perceptron (MLP) to obtain Attn. h The Multilayer Perceptron (MLP) consists of two linear layers and a Swish activation function; Attn h After applying the Softmax activation function, the Hadamard product with V' is calculated to obtain high-frequency local information features with local context awareness. The formula is as follows:

[0026]

[0027] Q',K',V'=DWConv(Q,K,V)

[0028] Attn h =FC(Swish(FC(Q'⊙K')))

[0029]

[0030]

[0031] Furthermore, step (4) is as follows:

[0032] For the visible light mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features For the infrared mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features Low-frequency local information features of different modalities and By performing addition and fusion, low-frequency global fusion features are obtained. High-frequency local information features of different modes and By performing additive fusion, high-frequency local fusion features are obtained. Will and Connecting them along the channel dimension, and then applying a fully connected layer for fusion, yields the aggregated high- and low-frequency fusion feature Z. i The formula is as follows:

[0033]

[0034]

[0035]

[0036] Furthermore, step (6) is as follows: calculate the loss value between the predicted result and the real target box according to the loss function, and update the learnable parameters in the network using stochastic gradient descent; repeat steps (1)-(6) until the number of repetitions reaches the preset number of training times to complete the training process and obtain the detection model.

[0037] The multispectral pedestrian detection system based on cross-modal feature decomposition described in this invention includes the following modules:

[0038] Data acquisition module: used to acquire a multispectral pedestrian dataset consisting of paired visible light and infrared images, divided into training and test sets according to the M:N ratio, with each pair of visible light and infrared images having the same number of image channels;

[0039] Network construction module: used to construct a two-stream network using a two-way YOLOv8 backbone network to extract visible light mode features and infrared mode features respectively;

[0040] Visible-infrared fusion feature module: used to input layered visible light output features and infrared output features into the CTF module for feature extraction, decomposition and fusion, to obtain visible-infrared fusion features at different scales containing low-frequency and high-frequency information;

[0041] Low-frequency and high-frequency feature fusion module: This module inputs the obtained low-frequency global information features and high-frequency local information features into the high-low frequency information feature fusion module, and fuses them according to the low-frequency and high-frequency features.

[0042] Multi-scale fusion module: used to input the aggregated high and low frequency fusion features from the previous steps into the Neck layer of the YOLOv8 network for multi-scale fusion;

[0043] Prediction module: This module takes the obtained multi-scale features and inputs them into the detection head of the YOLOv8 network, and outputs the predicted bounding box and the corresponding confidence score.

[0044] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it can extract high-frequency and low-frequency information between different modalities using Transformer and a carefully designed CNN structure, and perform effective fusion; it adopts a proximity attention mechanism to further improve the accuracy of pedestrian detection in low-light environments. Attached Figure Description

[0045] Fig. 1 This is a network structure diagram of the present invention;

[0046] Fig. 2 This is a CTF structure diagram of the present invention;

[0047] Fig. 3 This is a structural diagram of the FD of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0049] like Figs. 1-3 As shown, this embodiment of the invention provides a multispectral pedestrian detection method based on cross-modal feature decomposition, comprising the following steps:

[0050] (1) Obtain a pair of visible light and infrared images to form a multispectral pedestrian dataset, and divide it into a training set and a test set in an 8:2 ratio. Each pair of visible light and infrared images has the same number of image channels. The obtained dataset is: Visible Light-Infrared Paired Dataset LLVIP.

[0051] (2) A two-stream network was constructed using a two-path YOLOv8 backbone network to extract visible light modal features and infrared modal features, respectively. Specifically, visible light and infrared images were fed into the two-stream network to extract features. The two-stream network used stacked convolutional layers to extract multimodal features. Three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the visible light branch of the two-stream network were used as visible light output features. Three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the infrared branch were used as infrared output features.

[0052] (3) Input the layered visible light output features and infrared output features from step (2) into the CTF module for feature extraction, decomposition, and fusion to obtain visible light-infrared fusion features containing low-frequency and high-frequency information at different scales; specifically as follows: input the visible light output features at i = 1, 2, and 3 and infrared output characteristics The data is input into three CTF modules for processing according to different layers. Each CTF module consists of two parts: a Feature Decomposition (FD) module and a high-low frequency information feature fusion module. The FD module is implemented as follows: the visible light output features from step 2 are processed... and infrared output characteristics The data are fed into two feature decomposition (FD) modules for processing. Each FD module consists of two parts: a Transformer-based low-frequency global information extraction (Global) module and a CNN-based high-frequency local information extraction (Local) module. The implementation process for low-frequency global information extraction (Global) is as follows:

[0053] First, input features After passing through a LayerNorm normalization layer, the data is fed into the low-frequency global feature extraction module (Global) and the high-frequency local feature extraction module (Local) to extract the global feature. First, the query matrix Q is obtained directly through a fully connected layer, and then... The input is downsampled by a depthwise separable convolutional layer (DWConv); then, it is mapped through a fully connected layer to obtain the keyword matrix K and the value matrix V; the query matrix, keyword matrix, and value matrix are then fed as input to a dilated neighborhood attention module for long-range dependency modeling and global information extraction, followed by residual connections to correct the features; finally, it passes through a layer normalization layer (LayerNorm) and a multilayer perceptron (MLP) to obtain low-frequency global information features. The formula is as follows:

[0054]

[0055]

[0056]

[0057] Attn = Attention(Q,K',V')

[0058]

[0059] Z = MLP(Norm(Y)) + Y

[0060] The process of extracting high-frequency local information (Local) based on CNN is as follows:

[0061] First, input features The query matrix Q, keyword matrix K, and value matrix V are obtained through linear transformation. Then, Q, K, and V are input into a simple deep convolution with globally shared weights for local feature enhancement, resulting in Q', K', and V' with aggregated local information. Next, the Hadamard product of Q' and K' is calculated, and then passed through a multilayer perceptron (MLP) to obtain Attn. h The Multilayer Perceptron (MLP) consists of two linear layers and a Swish activation function; Attn h After applying the Softmax activation function, the Hadamard product with V' is calculated to obtain high-frequency local information features with local context awareness. The formula is as follows:

[0062]

[0063] Q',K',V'=DWConv(Q,K,V)

[0064] Attn h =FC(Swish(FC(Q'⊙K')))

[0065]

[0066]

[0067] (4) Input the low-frequency global information features and high-frequency local information features obtained in step (3) into the high-low frequency information feature fusion module, and fuse them according to the low-frequency and high-frequency features; as follows:

[0068] For the visible light mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features For the infrared mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features Low-frequency local information features of different modalities and By performing addition and fusion, low-frequency global fusion features are obtained. High-frequency local information features of different modes and By performing additive fusion, high-frequency local fusion features are obtained. Will and Connecting them along the channel dimension, and then applying a fully connected layer for fusion, yields the aggregated high- and low-frequency fusion feature Z. i The formula is as follows:

[0069]

[0070]

[0071]

[0072] (5) Input the aggregated high and low frequency fusion features from step (4) into the Neck layer of the YOLOv8 network to perform multi-scale fusion;

[0073] (6) Input the multi-scale features obtained in step (5) into the detection head of the YOLOv8 network and output the predicted bounding box and the corresponding confidence score. Specifically, calculate the loss value between the predicted result and the real target box according to the loss function, and update the learnable parameters in the network using stochastic gradient descent; repeat steps (1)-(6) until the number of repetitions reaches the preset number of training times to complete the training process and obtain the detection model.

[0074] This invention also provides a multispectral pedestrian detection system based on cross-modal feature decomposition, comprising the following modules:

[0075] Data acquisition module: used to acquire a multispectral pedestrian dataset consisting of paired visible light and infrared images, divided into training and test sets according to the M:N ratio, with each pair of visible light and infrared images having the same number of image channels;

[0076] Network construction module: used to construct a two-stream network using a two-way YOLOv8 backbone network to extract visible light mode features and infrared mode features respectively;

[0077] Visible-infrared fusion feature module: used to input layered visible light output features and infrared output features into the CTF module for feature extraction, decomposition and fusion, to obtain visible-infrared fusion features at different scales containing low-frequency and high-frequency information;

[0078] Low-frequency and high-frequency feature fusion module: This module inputs the obtained low-frequency global information features and high-frequency local information features into the high-low frequency information feature fusion module, and fuses them according to the low-frequency and high-frequency features.

[0079] Multi-scale fusion module: used to input the aggregated high and low frequency fusion features from the previous steps into the Neck layer of the YOLOv8 network for multi-scale fusion;

[0080] Prediction module: This module takes the obtained multi-scale features and inputs them into the detection head of the YOLOv8 network, and outputs the predicted bounding box and the corresponding confidence score.

Claims

1. A multispectral pedestrian detection method based on cross-modal feature decomposition, characterized in that, Includes the following steps: (1) Obtain visible light and infrared images to form a paired multispectral pedestrian dataset, and divide it into training set and test set according to the ratio of M:N. Each pair of visible light and infrared images has the same number of image channels. (2) A dual-stream network was constructed using a two-way YOLOv8 backbone network to extract visible light mode features and infrared mode features respectively; (3) Input the layered visible light output features and infrared output features from step (2) into the CTF module for feature extraction, decomposition and fusion to obtain visible light-infrared fusion features containing low-frequency and high-frequency information at different scales. (4) Input the low-frequency global information features and high-frequency local information features obtained in step (3) into the high-low frequency information feature fusion module and fuse them according to the low-frequency and high-frequency features; (5) Input the aggregated high and low frequency fusion features from step (4) into the Neck layer of the YOLOv8 network to perform multi-scale fusion; (6) Input the multi-scale features obtained in step (5) into the detection head of the YOLOv8 network and output the prediction box and the corresponding confidence score.

2. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 1, characterized in that, The dataset obtained in step (1) is: Visible-Infrared Pairing Dataset LLVIP.

3. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 1, characterized in that, Step (2) is as follows: Visible light and infrared images are fed into a dual-stream network to extract features; the dual-stream network uses stacked convolutional layers to extract multimodal features; three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the visible light branch of the dual-stream network are used as visible light output features. Three feature maps with output feature map sizes of 80×80×256, 40×40×512, and 20×20×1024 in the infrared branch were used as infrared output features. .

4. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 1, characterized in that, The specific steps (3) are as follows: The visible light output characteristics at i = 1, 2, and 3 are... and infrared output characteristics The data is input into three CTF modules according to different layers; the CTF module consists of two parts: a feature decomposition module (FD) and a high-low frequency information feature fusion module.

5. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 4, characterized in that, The Feature Decomposition (FD) module is implemented as follows: The visible light output features from step 2 are... and infrared output characteristics The data are fed into two feature decomposition modules (FD) for processing. Each FD module consists of two parts: a low-frequency global information extraction module (Global) based on Transformer and a high-frequency local information extraction module (Local) based on CNN.

6. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 5, characterized in that, The process of extracting low-frequency global information (Global) is as follows: First, input features After passing through a LayerNorm normalization layer, the data is fed into the low-frequency global feature extraction module (Global) and the high-frequency local feature extraction module (Local) to extract the global feature. First, the query matrix Q is obtained directly through a fully connected layer, and then... The input is downsampled by a depthwise separable convolutional layer (DWConv); then, it is mapped through a fully connected layer to obtain the keyword matrix K and the value matrix V. The query matrix, keyword matrix, and value matrix are fed as input to the Hollow Neighbor Attention module for long-range interdependence modeling and global information extraction. Residual connections are then used to correct the features. Finally, a LayerNorm normalization layer and a Multilayer Perceptron (MLP) are applied to obtain low-frequency global information features. The formula is as follows: Attn = Attention(Q,K',V') Z = MLP(Norm(Y)) + Y.

7. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 5, characterized in that, The process of extracting high-frequency local information (Local) based on CNN is as follows: First, input features The query matrix Q, keyword matrix K, and value matrix V are obtained through linear transformation. Then, Q, K, and V are input into a simple deep convolution with globally shared weights for local feature enhancement, resulting in Q', K', and V' with aggregated local information. Next, the Hadamard product of Q' and K' is calculated, and then passed through a multilayer perceptron (MLP) to obtain Attn. h The Multilayer Perceptron (MLP) consists of two linear layers and a Swish activation function; Attn h After applying the Softmax activation function, the Hadamard product with V' is calculated to obtain high-frequency local information features with local context awareness. The formula is as follows: Q',K',V'=DWConv(Q,K,V) Attn h =FC(Swish(FC(Q'WK'))) 。 8. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 1, characterized in that, Step (4) is as follows: For the visible light mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features For the infrared mode, the low-frequency global information features are obtained from step (3). and high-frequency local information features Low-frequency local information features of different modalities and By performing addition and fusion, low-frequency global fusion features are obtained. High-frequency local information features of different modes and By performing additive fusion, high-frequency local fusion features are obtained. Will and Connecting them along the channel dimension, and then applying a fully connected layer for fusion, yields the aggregated high- and low-frequency fusion feature Z. i The formula is as follows: 。 9. The multispectral pedestrian detection method based on cross-modal feature decomposition according to claim 1, characterized in that, The specific steps (6) are as follows: calculate the loss value between the predicted result and the real target box according to the loss function, and update the learnable parameters in the network using stochastic gradient descent; repeat steps (1)-(6) until the number of repetitions reaches the preset number of training times, and the training process is completed to obtain the detection model.

10. A multispectral pedestrian detection system based on cross-modal eigenvalue decomposition, characterized in that, Includes the following modules: Data acquisition module: used to acquire a multispectral pedestrian dataset consisting of paired visible light and infrared images, divided into training and test sets according to the M:N ratio, with each pair of visible light and infrared images having the same number of image channels; Network construction module: used to construct a two-stream network using a two-way YOLOv8 backbone network to extract visible light mode features and infrared mode features respectively; Visible-infrared fusion feature module: used to input layered visible light output features and infrared output features into the CTF module for feature extraction, decomposition and fusion, to obtain visible-infrared fusion features at different scales containing low-frequency and high-frequency information; Low-frequency and high-frequency feature fusion module: This module inputs the obtained low-frequency global information features and high-frequency local information features into the high-low frequency information feature fusion module, and fuses them according to the low-frequency and high-frequency features. Multi-scale fusion module: used to input the aggregated high- and low-frequency fusion features from the previous steps into the Neck layer of the YOLOv8 network for multi-scale fusion; Prediction module: This module takes the obtained multi-scale features and inputs them into the detection head of the YOLOv8 network, and outputs the predicted bounding box and the corresponding confidence score.

Images