Multi-spectral pedestrian detection method based on cross-modal progressive fusion

Abstract

The invention discloses a multispectral pedestrian detection method based on cross-modal progressive fusion, and belongs to the technical field of target detection. Constructing a pedestrian detection network, wherein the whole multispectral pedestrian detection network comprises a double-branch feature extraction module, a modal feature enhancement module, a cross-modal feature fusion module and a detection module; according to the method, a double-branch multispectral pedestrian detection network is adopted, and paired visible light and infrared images are used as input; multi-level features of the image are extracted through a multi-scale feature aggregation block, and the expression ability of each modal feature is improved by using a modal quality perception enhancement module; stage progressive fusion and reinforcement are carried out on visible light and infrared features through a cross-modal progressive feature fusion module, so that the network can make full use of complementary advantages of two modals; the method shows excellent performance in a multispectral pedestrian detection task, and each index is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a multispectral pedestrian detection method based on cross-modal progressive fusion. Background Art

[0002] As a core task in the field of computer vision, pedestrian detection is widely used in scenarios such as autonomous driving, intelligent monitoring, and robot interaction. Its main goal is to accurately identify and locate pedestrians from images or videos to ensure that the system can respond promptly in complex and dynamic environments. At present, single-sensor pedestrian detection based on visible light has made significant progress, but there are still some limitations. For example, visible light images can provide clear details under good lighting conditions, but their performance is poor in rainy, foggy or low-light environments. In contrast, infrared images rely on the thermal radiation of the target itself and can still effectively identify pedestrians in low-light or bad weather environments, but they are sensitive to temperature and are easily disturbed in strong light or high temperature environments, resulting in reduced recognition ability. Therefore, by combining the complementary characteristics of visible light and infrared images, the adaptability of pedestrian detection systems to various environments can be enhanced, and the accuracy and reliability of detection can be improved.

[0003] The Chinese patent publication number is "CN115410147A", and its name is "All-weather cross-modal adaptive fusion pedestrian target detection system and method". This method first passes paired visible light and infrared images through two single multi-frame detectors respectively to extract visible light and infrared features at different levels; then the differential feature mapping of the two modes is used to obtain enhanced features; then the enhanced features are weighted according to the lighting conditions, and the fusion features are weighted to obtain the fusion features; finally, the final detection result is obtained through the fully connected layer; the feature enhancement module of this method directly uses the differential feature mapping of the two modes to enhance the vector, ignoring the uniqueness and quality differences of the features of different modalities, and it is difficult to effectively improve the feature expression ability; the fusion module cannot dynamically adjust the strength of local features, and detail information is easily lost during the fusion process, resulting in low final detection accuracy. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a multispectral pedestrian detection method based on cross-modal progressive fusion, which solves the problems raised in the above-mentioned background technology.

[0005] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A multispectral pedestrian detection method based on cross-modal progressive fusion includes the following steps: Step 1: Build a pedestrian detection network: The entire multispectral pedestrian detection network consists of four parts: dual-branch feature extraction, modal feature enhancement, cross-modal feature fusion and detection module; Step 2, data set preparation and preprocessing: This method uses the KAIST multispectral pedestrian detection data set, and performs data cleaning and frame downsampling operations on the data set to improve data quality, and then performs image augmentation processing on the data set to increase sample diversity; Step 3: Train the pedestrian detection network: Train the multispectral pedestrian detection network. Input the visible light-infrared pedestrian dataset preprocessed in step 2 into the network model built in step 1 for training. Step 4: Select the best loss function and the best evaluation index of this method: Select a suitable loss function to minimize the loss between the output detection result and the true detection label value, set the training loss threshold, and continuously iterate and optimize the model until the set number of training times is reached or the loss function is within the predetermined range. The model parameters can be considered to have been pre-trained and saved. Select the best evaluation index to evaluate the output results of the test image to comprehensively evaluate the performance of the detection network. Step 5, save the best model: solidify the network model parameters and determine the final pedestrian detection model. If a pedestrian detection task is performed, the visible light-infrared image pair can be directly input into the trained pedestrian detection network model to obtain the final pedestrian target detection result.

[0006] Furthermore, the dual-branch feature extraction part in step 1 is composed of visible light feature extraction and infrared feature extraction, and each branch is uniformly composed of a multi-scale feature aggregation block and a downsampling layer in series to extract five levels of features; the input of the dual-branch feature extraction part is a pair of visible light and infrared images; the multi-scale feature aggregation block is composed of a convolutional layer, an upsampling layer, a downsampling layer, an L-type function, multi-head self-attention, channel attention and several convolutional blocks; the upsampling layer is a sub-pixel convolution, and the downsampling layer is uniformly a maximum pooling.

[0007] Furthermore, the modal feature enhancement part in step 1 is composed of five modal quality perception enhancement modules; the module is composed of cross attention, a feature quality perception module and several convolution blocks; the feature quality perception module is composed of three linear layers and a multi-layer perceptron; the input visible light and infrared features are passed through the feature quality perception module, and the excellent features in the two modalities are integrated to obtain high-quality features, and the high-quality features are used as query inputs of the two modal features, so that the two modal features have richer expression capabilities.

[0008] Furthermore, the cross-modal feature fusion part in the step 1 is composed of five cross-modal progressive feature fusion modules; the cross-modal progressive feature fusion module is composed of a linear layer, a convolutional layer, a convolutional block, a sampling layer one, a sampling layer two and an adaptive fusion module; the adaptive fusion module associates the two modalities, applies different weights to each modal feature, and performs more refined local feature fusion, thereby improving the effect of modal feature fusion; the two sampling layers sample compensatory features in the original features based on the generated sampling reference points and sampling offsets to make up for the detail information lost in the primary fusion process, and further enhance the detail expression of the fused features.

[0009] Furthermore, the detection module in step 1 is a YOLO detection module.

[0010] Furthermore, in the step 2, the KAIST multispectral pedestrian detection dataset first removes images that do not contain pedestrians and images that are incorrectly marked in the dataset, and then downsamples continuous images in the same scene, retaining one out of every three images in the training set and one out of every nineteen images in the test set; the dataset is then augmented to obtain the final dataset through random scaling, cropping and splicing.

[0011] Furthermore, the loss of the entire pedestrian detection network training process in step 4 includes three parts, namely classification loss, box regression loss, and confidence loss. The present invention designs a dual-branch multispectral pedestrian detection network, which extracts multi-level features of images through a multi-scale feature aggregation block, and combines the modal quality perception enhancement module and the cross-modal progressive feature fusion module to effectively enhance and integrate the information of the two modalities; the present method can dynamically capture the advantageous features of the two modalities, and through stage-by-stage progressive adaptive fusion, deeply mine the modal correlation at different scales and detail levels, and make full use of the complementary advantages of the two modalities; compared with the existing methods, the average accuracy of the present method is improved by nearly 4.15%.

[0012] The cross-modal progressive feature fusion module proposed in the present invention first performs a primary fusion of the features of the two modalities, then resamples the compensation information from the original features and further fuses them; during the fusion process, channels and spatial weights are independently assigned to each modality, so that the network can dynamically adjust the fusion weight according to the local feature intensity, thereby achieving efficient fusion and information retention of visible light and infrared features, providing rich and comprehensive feature information for subsequent detection. The modality quality perception enhancement module proposed in the present invention captures the advantageous information of the two modalities through self-screening and global screening, and uses the comprehensive advantageous information to guide the two modalities to suppress the interference of noise and low-quality features, thereby strengthening the sparse features in the weak modality and improving the expressiveness and complementary efficiency of the two features; compared with the existing methods, this method improves the recall rate by nearly 2.23%.

[0013] The multi-scale feature aggregation block proposed in the present invention utilizes feature downsampling and reconstruction mechanism to effectively restore detail features, while combining multi-head self-attention to capture global context dependencies; and fuses local spatial features from different receptive fields, thereby enhancing the adaptability to targets of different scales, enabling the network to extract rich features with both local details and global semantics, and improving its feature extraction capability in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of a multi-spectral pedestrian detection method based on progressive fusion; Figure 2 This is a network structure diagram of a multi-spectral pedestrian detection method based on progressive fusion; Figure 3 It is the feature extraction branch structure diagram of the present invention; Figure 4 This is a diagram of the multi-scale feature aggregation block structure of the present invention; Figure 5 This is a structural diagram of the modal quality perception enhancement module of the present invention; Figure 6 This is a structural diagram of the feature quality perception module of the present invention; Figure 7 This is a structural diagram of the cross-modal progressive feature fusion module of the present invention; Figure 8 This is a structural diagram of the adaptive fusion module of the present invention; Fig. 9 This is a structural diagram of the convolution block described in the present invention; Fig.10 This is a structural diagram of the depth-separable convolutional block described in the present invention; Fig.11 A comparison chart of related indicators between the prior art described in the present invention and the method proposed in the present invention; DETAILED DESCRIPTION The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Example 1 The present invention provides a flowchart of a multi-spectral pedestrian detection method based on progressive fusion, which specifically includes the following steps: Step 1, build a pedestrian detection network; the entire multispectral pedestrian detection network consists of four parts: dual-branch feature extraction, modal feature enhancement, cross-modal feature fusion and detection module; the dual-branch feature extraction part is composed of visible light and infrared feature extraction, each feature extraction branch contains five multi-scale feature aggregation blocks and downsampling layers to extract five layers of image features; the multi-scale feature aggregation block is composed of upsampling layer, downsampling layer, multi-head self-attention, channel attention, L-type function, convolution layer and ten convolution blocks, among which convolution block eight, convolution block nine, convolution block The convolution kernel sizes of convolution block 10 and convolution layer are 5×5, 7×7, 1×1, and 1×1 respectively. This module captures local details and global context information of features at different scales through two branches. The modal feature enhancement part consists of five modal quality perception enhancement modules, which are composed of cross attention, feature quality perception module and five convolution layers. It extracts advantageous features from the two modalities, guides the enhancement of the two modalities, and improves the discriminability and complementarity of the two modal features. Among them, the convolution kernel size of convolution block 1 and convolution block 2 is 1×1. The feature quality perception module is composed of a multi-layer perceptron and three linear layers. The input features are self-screened and globally screened to extract high-quality features with the advantages of both modalities as common query inputs. The cross-modal feature fusion part is composed of five cross-modal progressive feature fusion modules, which are composed of a linear layer, a convolutional layer, a convolutional block, two sampling layers and an adaptive fusion module. The convolutional block uses a 1×1 convolution kernel. The cross-modal progressive feature fusion module first performs primary fusion of features through an adaptive fusion module, and then samples the original features to obtain compensation features to strengthen the primary fusion features and realize progressive feature fusion. The adaptive fusion module is composed of two maximum pooling layers, two average pooling layers, two L2 pooling layers, two multi-layer perceptrons, three depth-separable convolutional blocks and two 1×1 convolutional layers. The adaptive fusion module can assign spatial weights to visible light and infrared features respectively, and perform more refined and efficient feature fusion. The detection module uses the YOLO detection module. The convolution kernel size of the remaining convolutions in the network model is unified as 3×3 and the step size is 1.

[0016] Step 2: Dataset preparation and preprocessing: Use the KAIST multispectral pedestrian detection dataset as the training, testing, and validation dataset for the pedestrian detection network; perform data cleaning and downsampling on the dataset to improve the quality of the dataset; and then perform data augmentation on the data to increase the diversity of training samples.

[0017] Step 3, training the pedestrian detection network; training the multispectral pedestrian detection network, inputting the preprocessed data set in step 2 into the network model constructed in step 1 for training.

[0018] Step 4, select the best loss function and the best evaluation index of this method; select a suitable loss function to minimize the loss between the detection result output by the network and the real detection label value, set the training loss threshold, and continuously iterate and optimize the model until the number of training times reaches the set threshold or the value of the loss function reaches the set range. The model parameters can be considered to have been pre-trained and the model parameters can be saved; the loss function of the pedestrian detection network is jointly constructed by the detection result output by the network and the real label, including classification loss, box regression loss and confidence loss; select the best evaluation index to evaluate the output result of the test image to comprehensively evaluate the performance of the detection network; during the training process, use loss rate, accuracy and recall rate as evaluation indicators of the pedestrian detection model.

[0019] Step 5, save the best model; solidify the network model parameters and determine the final pedestrian detection model. If a pedestrian detection task is performed, the visible light-infrared image pair can be directly input into the trained pedestrian detection network model to obtain the final pedestrian target detection result.

[0020] Example 2 The pedestrian detection network structure in step 1 is as follows: Figure 2 As shown in the figure, the entire multispectral pedestrian detection network consists of four parts: dual-branch feature extraction, modal feature enhancement, cross-modal feature fusion and detection module.

[0021] The dual-branch feature extraction part consists of visible light and infrared feature extraction branches, and the two branches have the same structure; Figure 3 As shown, each branch consists of a multi-scale feature aggregation block 1, a downsampling layer 1, a multi-scale feature aggregation block 2, a downsampling layer 2, a multi-scale feature aggregation block 3, a downsampling layer 3, a multi-scale feature aggregation block 4, a downsampling layer 4, a multi-scale feature aggregation block 5, and a downsampling layer 5. The five-level features of the input image are extracted layer by layer. The structure of the convolution block is shown in Fig. 9 as shown; Figure 4As shown in the figure, the input features of the multi-scale feature aggregation block pass through the convolution block one and are then input into three branches respectively; the first branch consists of the convolution block six and the convolution block seven; the second branch consists of the convolution block two, the downsampling layer, the convolution block three, the upsampling layer, the convolution block four, the multi-head self-attention and the convolution block five. The output of the first branch is fused with the output of the convolution block four to obtain delicate local information, and then input into the multi-head self-attention to extract global information; the third branch consists of the convolution block eight, the convolution block nine, the channel attention and the convolution block ten. The convolution kernel sizes of the convolution blocks eight and nine are 5×5 and 7×7 respectively, and the padding is 2 and 3 respectively, with a larger sense. The field of view captures a wider range of local contextual information and enhances the responsiveness to targets of different scales; the outputs of the second branch and the third branch are fused and further integrated through the convolution layer, and then the original features are fused and the output is obtained through the L-type function; the remaining convolutions in the multi-scale feature aggregation block uniformly use a 3×3 convolution kernel, a step size of 1, and a padding of 1; the downsampling layer of the feature extraction branch uses maximum pooling, and the generated feature map sizes are 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 32 of the original image; paired visible light and infrared images are input into the dual-branch feature extraction part to obtain paired five-level feature maps.

[0022] The modal feature enhancement part consists of five modal quality perception enhancement modules, which enhance the features of five levels layer by layer. The structure of the modal quality perception enhancement module is as follows: Figure 5 As shown in Figure 1, the visible light and infrared features are respectively passed through convolution blocks 1 and 2, and then the cross attention is input to capture the complementarity between the two modalities and enhance the consistency in the feature space; the processed visible light features are obtained through convolution blocks 3 and 4. and Feature matrix, infrared features are obtained through convolution blocks six and seven and Feature matrix; the features of the two modalities are passed through the feature quality perception module to obtain high-quality features that integrate the advantages of the two modalities, and then passed through the convolution block 5 to obtain the common Feature matrix; and The feature matrix is ​​multiplied, and then limited to [0, 1] by softmax, and then Perform dot multiplication on the feature matrix, and add the dot multiplication result to the original feature to obtain the enhanced visible light feature; , and The feature matrix is ​​enhanced by the same operation to obtain infrared features. The convolution layer in the modal quality perception enhancement module uses a convolution kernel of size 3×3, a step size of 1, and a padding of 1. The structure of the feature quality perception module is as follows: Figure 6As shown in the figure, after the visible light and infrared features are flattened, they are self-screened through linear layer one and linear layer two respectively to remove interference such as noise background, and then the splicing operation is performed. The screening number and feature score are obtained through linear layer three and multi-layer perceptron respectively, and then the screening number and feature score are used to screen features, and the screened features are reshaped to obtain high-quality features; the feature quality perception module is defined as follows: , in, and represents a self-screening function, represents the scoring function, Represents the visible light characteristics, Indicates infrared characteristics, To filter the number, select the one with the highest score Features.

[0023] The cross-modal feature fusion part consists of five cross-modal progressive feature fusion modules, such as Figure 7 As shown in the figure, each cross-modal progressive feature fusion module consists of a linear layer, a convolutional layer, a convolutional block, a sampling layer 1, a sampling layer 2, and two adaptive fusion modules; the input of this module is the visible light and infrared enhancement features of the same level. First, the visible light and infrared features are primarily fused through the adaptive fusion module to obtain the primary fusion features; The structure diagram of the adaptive fusion module is as follows: Figure 8 As shown in the figure, it includes two parts: channel adaptation and spatial adaptation; visible light and infrared features are processed by spatial L2 pooling, average pooling and maximum pooling respectively, and then the L2 pooling result is point-multiplied with the maximum pooling through the S-type function, and the point-multiplication results of the two modalities are respectively merged with the average pooling, and further processed by two multi-layer perceptrons; the part corresponding to each modality in the multi-layer perceptron output is split and added, and then point-multiplied with the original feature through the S-type function to obtain channel adaptation information; the channel adaptation information is added to the original feature to obtain the channel adaptation feature; the channel adaptation features of the two modalities are spliced ​​in the channel dimension, and then two spatial weight maps are obtained through three depth-separable convolution blocks and two parallel 1×1 convolutions. Each spatial weight map is multiplied with the corresponding channel average pooling result, and then processed by the S-type function and point-multiplied with the original feature to obtain the spatial adaptation information of the two modalities. The specific structure of the depth-separable convolution block is as follows Fig.10 As shown in the figure, the spatial adaptive information is added to the channel adaptive feature to obtain the adaptive feature; finally, the two adaptive features are added to obtain the final fusion feature; the definition of the adaptive fusion module is as follows: , in, Indicates channel adaptation information, Represents spatial adaptation information.

[0024] After obtaining the primary fusion features, the visible light features, infrared features and primary fusion features are concatenated, compressed and information integrated through convolution blocks, and then the sampling reference points and sampling offsets of the two modal features are obtained respectively through parallel linear layers and convolution layers; the sampling reference points and sampling offsets of the visible light and infrared modalities are input into sampling layer one and sampling layer two respectively to obtain the corresponding compensation features; by combining the primary fusion features with the original features, the model can distinguish the missing and weak information in the primary fusion, resample the features from the original features, and use the new features to further strengthen the primary fusion features; the feature sampling process is defined as follows: , in, represents the characteristic mode, represents the characteristic scale, represents the number of sampling points, represents the sampled features, represents the sampling reference point, represents the scaling factor of the sampling reference point, Indicates the sampling offset.

[0025] After sampling to obtain the compensation features, the compensation features are fused through the adaptive fusion module and integrated with the primary fusion features to obtain the final fusion features. , thereby achieving progressive fusion; the fusion feature is defined as follows: , The detection module uses the YOLO detection module, which passes the fused five-level features through the detection module to perform detection at different scales and comprehensively obtain the final detection result.

[0026] The pedestrian dataset in step 2 uses the KAIST multispectral pedestrian detection dataset, which is released by the Korea Advanced Institute of Science and Technology. The dataset contains 95,328 pairs of synchronously collected visible light images and infrared images, covering various traffic scenes such as campuses, streets and villages, captured during the day and at night, and has rich environmental change characteristics; since the original dataset contains a large number of images without pedestrians and incorrectly labeled images, and the similarity of consecutive frames is high, this method cleans and downsamples the dataset; first, images that do not contain pedestrians and incorrectly labeled images are removed, and then the consecutive frame images are downsampled, one for every three training sets and one for every nineteen test sets, and finally 7,601 training set images and 2,252 test set images are retained; finally, the dataset images are subjected to data augmentation processing such as random cropping, scaling and splicing, and the augmented dataset is used as the input of the pedestrian detection network.

[0027] In step 3, a pedestrian detection network is trained by inputting the KAIST data set prepared in step 2 into the pedestrian detection network constructed in step 1 for training.

[0028] The loss function in step 4 is designed to quantify the difference between the model prediction result and the real detection label. By optimizing the loss function, the model parameters are continuously and dynamically adjusted to improve the detection performance. The loss function in the training process By classification loss , positioning loss and confidence loss It consists of three parts: the classification loss uses cross entropy loss, which regards the output category as the probability distribution of the category; the positioning loss uses GIOU loss, which not only focuses on the overlapping area between the predicted box and the true box, but also focuses on other non-overlapping areas, which can better reflect the relationship between the two boxes; the confidence loss uses mean square error loss, which directly quantifies the difference between the confidence of the predicted value and the confidence of the true value. The loss function is defined as: , , , , in, represents the number of predicted image grids, represents the number of prediction boxes in each grid, Indicates The first Is the predicted box a positive sample? and They represent the probability and true probability of the network predicting that the sample is a pedestrian, respectively. represents the prediction box, represents the real frame, Indicates the minimum enclosed area of ​​the predicted box and the true box, and Represents the true value and the network prediction value of the confidence.

[0029] In step 4, the accuracy (Precision, P), recall (Recall, R) and average precision (AveragePrecision, AP) are selected as indicators for evaluating the quality of the pedestrian detection network; the accuracy, recall and average precision are defined as follows: , , , in, It represents the number of pedestrian samples correctly predicted by the model. Indicates the number of samples that are actually pedestrians but are not correctly identified. The number of non-pedestrian samples that are incorrectly predicted as pedestrians, represents the total number of samples containing pedestrians, represents the total number of samples predicted to be pedestrians, Represents the total number of detection thresholds.

[0030] Optimize the network model, set the number of training rounds to 300, use the Adam optimizer, set the initial learning rate to 0.001, and set the learning rate to be reduced to 10% of the original after every 10 rounds of training. Set the loss threshold to 0.0003, and iterate continuously during the training process. When the training loss is infinitely close to the set loss threshold, the network is considered to have been basically trained.

[0031] After the network training is completed in step 5, it is necessary to solidify all the trained parameters in the network and determine the final pedestrian detection model. If a pedestrian detection task is performed, the visible light-infrared image pair can be directly input into the trained pedestrian detection network model to obtain the final pedestrian target detection result.

[0032] Among them, the implementation of convolutional layers, average pooling, cross attention and multi-layer perceptron are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0033] The present invention designs a multispectral pedestrian method based on cross-modal progressive fusion and applies it to pedestrian target detection tasks. It uses the complementary characteristics of visible light and infrared images to overcome the limitations of a single modality and significantly improves the performance of the pedestrian detection system. Fig.11 As shown; under the same conditions, the various indicators of this method perform better, which further verifies the feasibility and superiority of this method.

[0034] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multispectral pedestrian detection method based on cross-modal progressive fusion, characterized by: The steps include: Step 1: Build a pedestrian detection network: Step 2, data set preparation and preprocessing: This method uses the KAIST multispectral pedestrian detection data set, and performs data cleaning and frame downsampling operations on the data set to improve data quality, and then performs image augmentation processing on the data set to increase sample diversity; Step 3: Train the pedestrian detection network: Train the multispectral pedestrian detection network. Input the visible light-infrared pedestrian dataset preprocessed in step 2 into the network model built in step 1 for training. Step 4: Select the best loss function and the best evaluation index of this method: Select a suitable loss function to minimize the loss between the output detection result and the true detection label value, set the training loss threshold, and continuously iterate and optimize the model until the set number of training times is reached or the loss function is within the predetermined range, and save the model parameters; select the best evaluation index to evaluate the output results of the test image to comprehensively evaluate the performance of the detection network; Step 5, save the best model: solidify the network model parameters and determine the final pedestrian detection model. If a pedestrian detection task is performed, the visible light-infrared image pair can be directly input into the trained pedestrian detection network model to obtain the final pedestrian target detection result.

2. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 1 is characterized in that: Building a pedestrian detection network includes four parts: dual-branch feature extraction, modal feature enhancement, cross-modal feature fusion and detection module.

3. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 2, characterized in that: The dual-branch feature extraction part in step 1 is composed of visible light feature extraction and infrared feature extraction. Each branch is uniformly composed of a multi-scale feature aggregation block and a downsampling layer connected in series to extract five levels of features.

4. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 3 is characterized in that: The input of the dual-branch feature extraction part is a pair of visible and infrared images; The multi-scale feature aggregation block consists of a convolutional layer, an upsampling layer, a downsampling layer, an L-type function, multi-head self-attention, channel attention, and several convolutional blocks; the upsampling layer is sub-pixel convolution, and the downsampling layer is unified as maximum pooling.

5. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 4 is characterized in that: The modal feature enhancement part consists of five modal quality perception enhancement modules; the modal quality perception enhancement module consists of a cross attention, a feature quality perception module and several convolution blocks.

6. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 5 is characterized in that the feature The quality perception module consists of three linear layers and a multi-layer perceptron. The input visible light and infrared features pass through the feature quality perception module, and the excellent features of the two modalities are integrated to obtain high-quality features. The high-quality features are used as the query input of the two modal features, so that the two modal features have richer expression capabilities.

7. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 6, characterized in that: The cross-modal feature fusion part is composed of five cross-modal progressive feature fusion modules; The cross-modal progressive feature fusion module consists of a linear layer, a convolutional layer, a convolutional block, a sampling layer one, a sampling layer two, and an adaptive fusion module.

8. The multispectral pedestrian detection method based on cross-modal progressive fusion according to claim 7, characterized in that: The adaptive fusion module associates the two modalities and applies different weights to each modal feature to perform more refined local feature fusion, thereby improving the effect of modal feature fusion; the two sampling layers sample compensatory features in the original features based on the generated sampling reference points and sampling offsets to make up for the detail information lost in the primary fusion process, further enhancing the detail expression of the fused features.

Images