Pedestrian detection method based on small target enhancement and bimodal multi-scale attention fusion

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a pedestrian detection method based on small target enhancement and bimodal multi-scale attention fusion. Comprising the following steps: step 1, preparing data sets: preparing two types of infrared and visible light bimodal public data sets, dividing the data sets into a training set, a verification set and a test set according to a preset proportion, and synchronously introducing a data enhancement strategy to ensure sample distribution balance; step 2, constructing a small target enhanced feature extraction module which is composed of a small target enhancement unit and a multi-scale extraction unit which are respectively used for enhancing small-size pedestrian feature response and capturing multi-scale pedestrian context information; and step 3, constructing a multi-scale attention progressive fusion module: forming a four-stage progressive architecture by a self-adaptive modal weight unit, a channel attention unit, a space attention unit and a cross attention unit, and realizing dual-modal feature redundancy elimination, strong correlation and deep complementation.

Description

Technical Field

[0001] This invention relates to a pedestrian detection method based on small target enhancement and bimodal multi-scale attention fusion, belonging to the field of computer vision technology. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, pedestrian detection, as one of the core technologies in the field of intelligent perception, plays a crucial role in ensuring public safety, improving traffic efficiency, and promoting the industrialization of autonomous driving. Among them, pedestrian detection algorithms that fuse infrared and visible light modes, leveraging the complementary characteristics of the two modes, effectively overcome the performance limitations of single modes in complex environments. This has become a mainstream research direction in the field and is widely used in multiple key scenarios such as autonomous driving collision avoidance systems, urban video surveillance networks, and intelligent traffic management platforms.

[0003] Visible light images possess rich details such as texture and color, enabling precise depiction of pedestrian appearance features and making them suitable for well-lit daytime or indoor scenes. Infrared images, generated based on the difference in thermal radiation between the target and the environment, clearly present pedestrian contour information and are unaffected by factors such as light intensity and weather conditions (e.g., fog, snow, rain), offering unique advantages in harsh environments such as nighttime, backlighting, and low visibility. The fusion of these two technologies enables "all-weather, all-scene" pedestrian perception coverage, significantly improving the detection system's adaptability to complex environments.

[0004] In the field of autonomous driving, dual-modal pedestrian detection technology provides vehicles with real-time and accurate pedestrian location information, serving as a core support for collision avoidance warning and active safety control. In urban video surveillance, this technology enables 24 / 7 uninterrupted pedestrian monitoring in key areas, aiding in public security prevention and emergency response. In the field of intelligent transportation, it can accurately identify pedestrian dynamics at intersections and crosswalks, providing data support for signal timing optimization and pedestrian traffic safety. In recent years, with the improvement of chip computing power and the iteration of deep learning algorithms, infrared and visible light fusion pedestrian detection technology has been continuously upgraded, its application scenarios have continued to expand, and market demand has been growing. Summary of the Invention

[0005] Existing infrared and visible light fusion schemes still face problems in practical applications, such as disordered complementary logic of dual-modal features, insufficient fusion efficiency, and weak characterization ability of single-modal features for small pedestrians. Features of small pedestrian targets are easily submerged by background noise, making it difficult to form an effective feature response. Dual-modal features lack targeted redundancy removal and correlation enhancement mechanisms, resulting in low utilization of fused features. Especially in complex scenarios such as pedestrian occlusion and low illumination at night, it is difficult to balance the capture of detailed information and anti-interference capabilities, ultimately leading to insufficient detection accuracy and a high false negative rate.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution: A pedestrian detection method based on small target enhancement and bimodal multi-scale attention fusion includes the following steps: Step 1, Prepare the dataset: Prepare two types of publicly available bimodal datasets, specifically the LLVIP dataset and the KAIST dataset; divide each dataset into training, validation and test sets according to a preset ratio to ensure that the datasets are evenly distributed and meet the needs of model training and evaluation.

[0007] Step 2, construct a small target enhanced feature extraction module: extract low-level features from infrared and visible light images respectively using "convolutional block + downsampling module". The convolutional block is composed of convolutional layer, regularization layer and activation function connected in series. The size of the convolutional kernel and the stride are dynamically set according to the downsampling requirements. To address the issue that small pedestrian features are easily obscured by the background, the extracted low-level features are input into a small target enhancement structure: a lightweight deconstruction of "group convolution - channel reduction - depthwise convolution - pointwise convolution" is adopted. First, group convolution preserves the local texture and edge features of the small target. Then, 1×1 convolution reduces the number of channels to 1 / 2 of the input (but not less than 4, to avoid loss of small target features) to reduce the computational load. Subsequently, spatial features are extracted through depthwise convolution and the number of channels is restored through pointwise convolution. After fusion, nonlinearity is introduced through regularization and activation functions to enhance the response of small pedestrian features. Based on the enhanced features of small targets, the input multi-scale extraction structure is adopted: three parallel convolutional branches with shared convolutional kernels are used. Through the dilation rate design of 1, 2, and 3, the contextual information of pedestrians at small, medium, and large scales are captured respectively. Each branch shares the core convolutional kernel to reduce parameter redundancy. The output features are fused with the enhanced features of small targets through residual connections to fill the information gaps of single-scale features and improve the adaptability of features to complex backgrounds and targets of different sizes.

[0008] Step 3: Construct a multi-scale attention-based progressive feature fusion module: Build a four-level progressive fusion unit to address the issues of redundancy and weak correlation in dual-modal feature fusion layer by layer. The bimodal adaptive weight unit performs global average pooling on the multi-scale features of both modalities, converting the two-dimensional feature map into a one-dimensional feature vector. It learns modal adaptive weights through a double fully connected layer (channel reduction ratio of 4). The weight coefficients are multiplied element-wise with the original features to dynamically balance the feature contributions of the two modalities under different scenarios and remove redundant information. The convolutional layer of this unit uses the ReLU activation function and a Dropout layer (dropout probability set to 0.5) to avoid overfitting of the weight learning. Channel attention unit: Through an improved version of the squeeze-excitation mechanism, global average pooling and global max pooling are performed on the adaptively weighted two-modal features respectively. The concatenated vector learns the channel importance weights through the "dimensionality reduction-activation-dimensionality increase" process. The convolution kernel size is adaptively calculated according to the number of channels, which strengthens the feature channels related to pedestrian targets and suppresses the influence of redundant channels such as background noise and infrared thermal interference. Spatial Attention Unit: Performs global average pooling and global max pooling on the enhanced features of the channel, merges the two pooling results and generates spatial weighted coefficients through double convolution operation, highlighting the features of pedestrian regions (such as head, shoulders and legs) and weakening the interference of background regions; Cross-attention unit: Calculates the spatial correlation weight between bimodal features by multiplying the transpose of the feature matrix. After weighted fusion with another modal feature, it performs residual connection with the original modal feature to establish a deep cross-modal feature correlation, retains the initial feature information, improves the integrity of the fused features, and finally outputs a high-level fused feature that directly connects to the subsequent output head.

[0009] Step 4, construct the network model: with the small target enhanced feature extraction module as the front end, the multi-scale attention progressive feature fusion module as the core, and the detection output head as the back end, a complete dual-modal pedestrian detection network is constructed, and the feature transfer path and interface adaptation rules between each module are clarified.

[0010] Step 5, train the network model: input the preprocessed training set into the network and perform iterative training with the optimization objective of minimizing the combined loss function; the combined loss function includes the original detection loss and the splicing information loss, and the weight ratio of classification loss to regression loss is set to 1:3, focusing on optimizing the bounding box regression accuracy; during the training process, a mixed precision training strategy, AdamW optimizer and cosine annealing learning rate scheduling are adopted to improve training speed, numerical stability and anti-overfitting ability.

[0011] Step 6, fine-tune the network model: Select a public infrared pedestrian detection dataset as the fine-tuning dataset. Perform the same preprocessing operations as the training set to ensure data distribution consistency. Adopt a hierarchical fine-tuning strategy, fix the parameters of the bottom convolutional blocks of the feature extraction module, and only adjust the parameters of the upper structure and fusion module. Set a low learning rate (1 / 10 of the initial learning rate in the training phase, i.e., 0.0001), set the number of iterations to 100 epochs, and keep the triggering conditions of the early stopping strategy consistent with those in the training phase to optimize the model's generalization ability in a single infrared modality scene.

[0012] Step 7, Save the model: Monitor the model performance in real time on the validation set using evaluation metrics such as precision, recall, and mAP. Select the snapshot of the model parameters with the best overall evaluation metrics and save it to generate a detection model file that can be directly deployed.

[0013] In step 1, the dataset partitioning ratio can be flexibly adjusted according to the actual amount of data. When the amount of data is small, a partitioning ratio of 6:2:2 can be used. At the same time, data augmentation strategies are introduced, such as random flipping, translation, scaling, color gamut dithering, and random occlusion of infrared and visible light modes, to expand the number of training samples and improve the model's generalization ability.

[0014] In step 2, the output features of the small target enhancement structure are fused element by element and then nonlinearity is introduced through the ReLU activation function; the output features of each branch of the multi-scale extraction structure are integrated by channel splicing and then the number of channels is adjusted by a 1×1 convolutional layer to ensure the consistency of feature dimensions.

[0015] In step 3, the bimodal adaptive weight module uses ReLU activation and Dropout to remove redundancy, laying the foundation for accurate fusion; then the channel attention unit adaptively learns the channel weights to strengthen the target channel and suppress interference; then the spatial attention unit focuses on the core area of ​​the pedestrian and the weak background; finally, the cross attention unit achieves deep complementary fusion of the bimodal modes through feature interaction and residual connection, solving the problem of weak correlation.

[0016] In step 5, the weight ratio of classification loss to regression loss in the total loss function is set to 1:3, focusing on optimizing the bounding box regression accuracy and improving pedestrian positioning accuracy; a hybrid precision training strategy is adopted during training to improve training speed and numerical stability.

[0017] In step 6, the fine-tuning dataset needs to undergo the same preprocessing operations as the training set to ensure data distribution consistency; the number of fine-tuning iterations is set to 100 epochs, and the triggering conditions for the early stopping strategy are consistent with those in the training phase.

[0018] Compared with existing infrared and visible light dual-modal pedestrian detection technologies, this invention has significant technical advantages, effectively improving detection performance and scene adaptability. Specific beneficial effects are as follows: This invention proposes a synergistic design of lightweight small target enhancement and multi-scale extraction with shared convolutional kernels, significantly improving small target detection capabilities. Addressing the pain point that small pedestrian features are easily obscured by the background, this invention employs a lightweight deconstruction of "group convolution - channel reduction - depthwise convolution - pointwise convolution" to enhance the texture and edge features of small targets while reducing the number of parameters. Combined with a multi-scale extraction structure using shared convolutional kernels, parameter redundancy is reduced, ensuring the correlation of pedestrian features at different scales. The synergistic effect of these two approaches significantly improves small target detection accuracy, effectively reducing the false negative rate in scenarios with low illumination, strong occlusion, and long distances. It is particularly suitable for applications with a large number of small target samples, and improves feature utilization compared to traditional separate designs.

[0019] This invention designs a multi-scale attention-based progressive fusion scheme, overcoming the limitations of traditional fusion methods. Unlike the simple superposition of independent attention in existing technologies, this invention constructs a four-level progressive unit: "dynamic weight allocation - channel attention - spatial attention - cross attention." First, it eliminates bimodal redundancy, then strengthens key channels and target regions, and finally establishes cross-modal feature associations. This design enables the fused features to simultaneously possess the anti-interference advantages of the infrared mode and the detail advantages of the visible light mode, significantly improving the average accuracy in complex scenes. It effectively solves the problems of poor modal information adaptability and feature redundancy, and improves the efficiency of cross-modal feature complementarity.

[0020] This invention combines hierarchical fine-tuning with a combined loss function adaptation strategy, balancing model generalization and adaptability to extreme scenarios. It employs a combined loss of "original detection loss + stitched information loss" to optimize pedestrian localization accuracy. After training, a hierarchical fine-tuning strategy is used to specifically improve detection performance in extreme infrared scenarios (complete darkness, heavy fog, and strong occlusion). Experiments demonstrate that the fine-tuned model significantly improves recall and mAP in extreme scenarios, exhibiting stronger anti-interference capabilities and environmental adaptability. Compared to existing single fine-tuning strategies, it accelerates adaptation to extreme scenarios and can meet the high reliability requirements of pedestrian detection in scenarios such as autonomous driving and intelligent monitoring, showing broad application prospects. Attached Figure Description

[0021] Figure 1 This is a flowchart of the pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion of the present invention; Figure 2 This is a schematic diagram of the network structure described in this invention; Figure 3 This is a schematic diagram of the feature fusion module described in this invention; Figure 4 This is a schematic diagram of the small target enhancement module structure described in this invention; Figure 5 This is a schematic diagram of the multi-scale enhancement module structure described in this invention; Figure 6 This is a schematic diagram of the dual-modal dynamic weight structure described in this invention; Figure 7 This is a schematic diagram of the attention structure described in this invention; Figure 8 This is a schematic diagram of the feature fusion process described in this invention being input into the detection structure. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 like Figure 1 The flowchart shown is a pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion proposed in one embodiment of the present invention. The method specifically includes the following steps: Step 1: Prepare the dataset: Select two publicly available infrared and visible light dual-modal pedestrian detection benchmark datasets: the LLVIP dataset and the KAIST dataset. These datasets cover diverse scenarios such as nighttime, low-light conditions, urban roads, and campuses, catering to both extreme and common environments and providing comprehensive sample support for model training. The datasets are then divided into training, validation, and test sets according to a preset ratio. A 6:2:2 ratio is used when the data volume is small. Data augmentation strategies such as random flipping, translation, scaling, color gamut dithering, and random occlusion of the infrared and visible light modalities are introduced to expand the number of training samples, ensuring uniform sample distribution and meeting the needs of model training, validation, and performance testing.

[0024] Step 2: Construct a small target enhanced feature extraction module: A dual-branch structure is used to extract low-level features from both visible and infrared images. Subsequently, a small target enhancement unit and a multi-scale extraction unit are concatenated to form a complete feature extraction module. The small target enhancement unit uses multi-branch convolution parallel processing to enhance the feature response of small pedestrian targets, preventing small target features from being overwhelmed by background information. Its specific structure is as follows: Figure 4 As shown, the multi-scale extraction unit extracts contextual information of pedestrians of different sizes through multi-receptive-field convolutional branches, adapting to the feature capture needs of targets at different scales. Its architecture is as follows: Figure 5 As shown, the final output is the enhanced features of the two modalities, providing high-quality basic features for the subsequent fusion module.

[0025] Step 3, Construct a multi-scale attention-based progressive feature fusion module: A four-level progressive structure is used to construct the feature fusion module, which sequentially includes an adaptive modality weighting unit, a channel attention unit, a spatial attention unit, and a cross-attention unit. The overall module structure is as follows: Figure 3 As shown. The adaptive modal weighting unit dynamically adjusts the contribution ratio of the two modal features to achieve balanced fusion between modalities. Its implementation logic is as follows: Figure 6As shown; the channel attention unit enhances the feature channels related to the pedestrian target and suppresses redundant information; the spatial attention unit highlights the foreground pedestrian region and weakens background interference; the cross attention unit realizes deep interaction and complementarity of two modal features, making up for the shortcomings of single modal information. The collaborative structure of the three types of attention units is as follows: Figure 7 As shown, the final output is a high-level fusion feature, which is directly input into the detection head for subsequent prediction.

[0026] Step 4, Construct the network model: Build an end-to-end bimodal pedestrian detection network, which consists of a small target enhanced feature extraction module, a multi-scale attention-based progressive feature fusion module, and a detection module (output head). The overall network architecture is as follows: Figure 2 As shown, the input layer receives preprocessed infrared and visible light dual-modal images, which are then enhanced and multi-scale captured by the feature extraction module. The feature fusion module then achieves cross-modal feature complementarity, and finally, the detection module outputs pedestrian category, bounding box, and confidence score, realizing fully automated processing from image input to detection result output.

[0027] Step 5, Train the Network Model: Perform end-to-end training on the constructed network model, with the optimization objective of minimizing the combined loss function. This loss function includes the original detection loss and the concatenation information loss, with the classification loss and regression loss weighted at a ratio of 1:3, which can balance target localization, class determination, and modality fusion effects. During training, a mixed precision training strategy, the AdamW optimizer, and cosine annealing learning rate scheduling are used to improve the model's adaptability to dual-modal data, enhance gradient stability, and avoid overfitting.

[0028] Step 6, Fine-tuning the network model: To improve the model's detection performance in extreme infrared scenarios (such as complete darkness, heavy fog, and strong occlusion), a dedicated infrared pedestrian detection dataset was selected to fine-tune the trained model. A transfer learning strategy was adopted, loading the pre-trained model parameters and setting the learning rate to 0.0001 to avoid disrupting existing parameters. The parameters of the lower-level convolutional blocks of the fixed feature extraction module were fine-tuned hierarchically, optimizing only the upper-level structure and fusion module parameters. The focus was on improving the model's ability to extract and fuse features in extreme scenarios, further enhancing the model's scene adaptability and robustness. The fine-tuning iterations were set to 100 epochs, and the triggering conditions for the early stopping strategy remained consistent with the training phase.

[0029] Step 7, Save the model: After the model training and fine-tuning are completed, select the model parameters with the best performance on the validation set and save them. In practical pedestrian detection applications, the infrared and visible light images to be detected are directly input into the saved model, and accurate detection results can be quickly output through network inference, meeting the application needs of real-world scenarios.

[0030] Example 2 like Figure 1 The flowchart shown is a pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion proposed in one embodiment of the present invention. The method specifically includes the following steps: Step 1: Prepare the dataset: Select two publicly available infrared and visible light bimodal pedestrian detection benchmark datasets: the LLVIP dataset and the KAIST dataset. The LLVIP dataset contains over 10,000 pairs of infrared and visible light images, covering adverse weather scenarios such as nighttime, low light, and fog / snow, with small pedestrian targets accounting for 35%, making it suitable for validating the model's small target detection capabilities. The KAIST dataset contains over 95,000 pairs of images, covering diverse scenarios such as urban roads, campuses, and commercial areas, with a wide range of lighting variations, effectively improving the model's scene adaptability. Using both datasets together can accommodate both extreme and normal environments, ensuring comprehensive model training.

[0031] The dataset was divided into training, validation, and test sets in a 6:2:2 ratio. Data augmentation strategies such as random flipping, translation, scaling, color gamut dithering, and random occlusion of infrared and visible light modes were introduced to expand the number of training samples and improve the model's generalization ability.

[0032] Step 2: Construct a small target enhanced feature extraction module: A dual-branch parallel architecture is adopted to perform low-level feature extraction and enhancement on visible light and infrared images respectively. Through small target enhancement units and multi-scale extraction units, the module strengthens the representation of small target features and the ability to capture multi-scale features, providing high-quality basic features for subsequent fusion modules. The overall network architecture is as follows: Figure 2 As shown.

[0033] Small Target Enhancement Unit: Designed for small pedestrian targets, such as the head and shoulders of a distant pedestrian who occupy only 5% of the image, where feature information is easily obscured by the background, this unit enhances the responsiveness of small target features through multi-branch convolutional operations. Its specific structure is as follows: Figure 4 As shown.

[0034] This unit employs a dual-branch parallel structure: Branch 1 is a small convolutional branch, using a 1×1 convolutional kernel to perform channel dimensionality reduction on low-level features, decreasing the number of channels from 512 to 256, reducing computational cost while preserving detailed features of small objects; Branch 2 is a combination of pointwise convolution and depthwise separable convolution. It first uses 1×1 pointwise convolution to reorganize channels, then uses 3×3 depthwise separable convolution to extract spatial features. Depthwise separable convolution separates the channel convolution from the spatial convolution of conventional convolution, reducing computational cost by 75% while maintaining feature extraction quality. The output features of the two branches are fused element-wise and then nonlinearity is introduced through the ReLU activation function to obtain visible light features. With infrared features The specific calculation formula is as follows: , Multi-scale convolutional unit: To capture contextual information of pedestrian targets of different sizes and adapt to the feature extraction needs of large-sized full-body pedestrians at close range and small-sized pedestrians at a distance, a multi-scale convolutional unit is designed, the architecture of which is shown in Figure 5. This unit constructs three convolutional branches with different receptive fields, using 3×3 dilated convolutions with dilation rates of 1, 3, and 5 respectively. Increasing the dilation rate can effectively expand the receptive field without increasing the size of the convolutional kernel: Branch 1 (dilation rate 1) extracts local features of an 11×11 receptive field, adapting to small-sized pedestrians; Branch 2 (dilation rate 3) extracts mid-scale features of a 19×19 receptive field, adapting to medium-sized pedestrians; Branch 3 (dilation rate 5) extracts global features of a 27×27 receptive field, adapting to large-sized pedestrians. Each branch adopts a "dilated convolution-BN-ReLU" structure. After channel concatenation, the output features are adjusted to 512 channels through a 1×1 convolutional layer to maintain consistency with the input feature dimension. Finally, a residual connection method is used to add the multi-scale convolutional features to the input small target enhancement features element-wise, achieving multi-scale feature fusion and gradient propagation optimization. The specific calculation formula is as follows: , Step 3: Construct a multi-scale attention-based progressive feature fusion module: The feature fusion module is constructed using a four-level progressive structure, which includes an adaptive modality weight unit, a channel attention unit, a spatial attention unit, and a cross attention unit. The overall structure of the module is shown in Figure 3, which solves the core problems of redundancy and weak correlation in dual-modality feature fusion layer by layer.

[0035] Adaptive Modal Weighting Unit: To address the discrepancy in the effectiveness of infrared and visible light modal information across different scenarios, an adaptive modal weighting unit is designed to dynamically adjust the contribution ratio of the two modal features, achieving balanced fusion between the modalities. Its implementation logic is as follows: Figure 6 The specific operating procedure is shown below: First, global average pooling is performed on the two modal features after multi-scale enhancement to convert the two-dimensional feature map into a one-dimensional feature vector (dimension 512), capturing global feature information. Then, the one-dimensional feature vector is input into a weight generation network consisting of two fully connected layers. The first fully connected layer reduces the feature dimension from 512 to 128 by introducing non-linearity using the ReLU activation function. The second fully connected layer reduces the dimension from 128 to 1 by using the Sigmoid activation function to output weight coefficients. The magnitude of the weight coefficients directly reflects the effectiveness of the corresponding modal features. Finally, the weight coefficients are multiplied element-wise with the original modal features to achieve dynamic weighting of the modal features. The specific calculation formula is as follows: , Channel Attention Unit: The channel attention unit aims to enhance feature channels relevant to pedestrian targets by modeling the dependencies between feature channels, suppressing the influence of redundant channels such as background noise and infrared thermal interference, and improving the discriminative ability of feature channel dimensions. It employs an improved version of the "squeeze-excitation" mechanism, with the following specific structure: Figure 7 As shown, the specific steps are as follows: Global average pooling and global max pooling are applied to the adaptively weighted two-modal features to obtain two sets of one-dimensional channel statistical vectors (both with a dimension of 512). These two vectors are concatenated and input into a fully connected layer. Channel importance weights are learned through a "dimensionality reduction-activation-dimensionality increase" process. The first layer of the fully connected layer uses a 1×1 convolution to reduce the dimension (from 512 to 64), and the ReLU activation function introduces non-linearity. The second layer uses a 1×1 convolution to increase the dimension (from 64 to 512), and the Sigmoid activation function outputs channel attention weights (with a dimension of 512). Finally, the channel attention weights are multiplied element-wise with the adaptively weighted features to obtain the channel-enhanced features. The specific calculation formula is as follows: , Spatial Attention Unit: After channel weighting, the features are further weighted in a refined spatial dimension. The core is to accurately locate the pedestrian target region and suppress background interference by modeling the dependencies in the spatial distribution of features, thereby improving the foreground-background separation capability of the feature map. This forms a complementary enhancement with the channel attention unit in both channel and spatial dimensions. The specific structure is as follows: Figure 7 As shown, the specific steps are as follows: Spatial Feature Statistics: Two-Modal Features After Channel Enhancement , Global average pooling and global max pooling operations are performed separately. Global average pooling captures the overall feature response of the spatial region, while global max pooling focuses on the local peak features of the spatial region, ultimately yielding two sets of features whose dimensions are the same as the input features. Figure 1 Two-dimensional spatial statistical feature map; Feature fusion and weight learning: Two sets of spatial statistical feature maps are concatenated along the channel dimension to obtain a fused feature map with 2 channels. This fused feature map is then input into a weight generation network consisting of two 3×3 convolutional layers. The first convolutional layer captures spatial neighborhood association information through local receptive fields and introduces non-linearity via a ReLU activation function. The second convolutional layer compresses the feature dimension to 1 channel and then passes it through a Sigmoid activation function to output a spatial attention weight map Ms. The importance coefficient of each pixel value in the weight map corresponding to the same position in the input feature map ranges from 0 to 1. Spatial feature enhancement: The spatial attention weight map Ms is multiplied element-wise with the channel-enhanced feature map to enhance the features of pedestrian target areas such as head, shoulders, and legs, with the weight coefficient approaching 1. The features of background redundant areas are suppressed, with the weight coefficient approaching 0, and finally the spatially enhanced features are obtained.

[0036] The specific expression is: , Ms is generated by convolution after channel average pooling and max pooling.

[0037] Cross-attention unit: Based on spatial attention features, it achieves deep complementarity of bimodal features by modeling cross-modal spatial dependencies. The core is to allow RGB and IR features to mutually borrow advantageous information, addressing the information limitations of a single modality in extreme scenarios. Together with the aforementioned channel and spatial attention units, it forms a three-dimensional enhancement system of "channel-space-cross-modality." Specific linkages and cross-attention structures are as follows: Figure 7 As shown. The specific steps are as follows: Cross-modal correlation modeling: spatially augmented two-modal features ( , Dimensional reconstruction is performed, transforming the feature map dimensions from (C×H×W) to (C×HW), where C is the number of channels, H is the height, and W is the width, facilitating spatial correlation calculation; subsequently, the feature matrices are multiplied by their transposes, resulting in the expression: , , Generate a cross-modal spatial correlation weight matrix (dimension HW×HW). The larger the weight value, the stronger the cross-modal feature correlation of the corresponding spatial location. Cross-modal feature weighting: This involves performing matrix multiplication between the associated weight matrix and the spatial enhancement features of another modality to achieve spatial weighted compensation of the source modality features by the target modality features. For example, strong contour features of the infrared modality can be used to compensate for weak texture features of the visible light modality in nighttime scenes. Specifically: , , The compensated feature dimension is then reconstructed back to (C×H×W) to maintain consistency with the original spatial augmented feature dimension. Residual fusion and output: A residual connection strategy is adopted, adding cross-modal compensation features and original spatial enhancement features element-wise. This retains the core feature information of the modality itself while incorporating the complementary advantages of the other modality, avoiding feature distortion. The final output is a high-level fusion feature after cross-modal enhancement, which can be directly input into the detector head for pedestrian category determination and bounding box regression. The specific calculation formula is as follows: , Output and To provide the final high-level features for the fusion module, the detection head can be directly input to generate pedestrian categories, bounding boxes, and confidence scores.

[0038] Step 4, Construct the network model: The entire detection network adopts an end-to-end architecture, consisting of three parts: the small target enhanced feature extraction module from Step 2, the multi-scale attention progressive fusion module from Step 3, and the detection module (output head). These modules work collaboratively, and the overall network architecture is as follows: Figure 2 As shown, the specific process of feature fusion input detection head is as follows: Figure 8 As shown, the entire process from dual-modal image input to pedestrian detection result output is automated. The specific architecture design is as follows: The network architecture is as follows: The input layer receives preprocessed infrared and visible light dual-modal image pairs with an image size of 512×512. It adopts a dual-branch parallel input to maintain modal independence. The feature extraction module adopts a dual-branch structure, performing low-level feature extraction, small target enhancement, and multi-scale residual enhancement on the two modal images respectively, and outputs enhanced features with a dimension of 512×64×64 (the input size of 512×512 is downsampled three times to obtain a 64×64 feature map). The feature fusion module performs a four-level progressive fusion of the enhanced features of the two modalities, and outputs dual-output high-level features with a dimension of 512×64×64. The detection module integrates and predicts the dual-output features, and outputs pedestrian category, bounding box coordinates, and confidence score.

[0039] Detection module (output head) design: An anchor-based detection architecture is adopted to adapt to pedestrian target detection of different sizes. Features are concatenated along the channel dimension to obtain a fused feature map with dimensions of 1024×64×64. Subsequently, two 3×3 convolutional layers further enhance the concatenated features, while maintaining the number of channels at 1024. A multi-scale detection strategy is employed, outputting three sets of feature maps at different downsampling stages of the feature extraction module (64×64, 32×32, 16×16), corresponding to small, medium, and large-sized pedestrian targets, respectively. Each set of feature maps is processed by the feature fusion module before being input into the detection head. Three sets of anchors of different sizes are designed for each set of feature maps. Based on pedestrian size statistics in the dataset, the small-scale feature map (64×64) is adapted to the anchor size. The size of the anchor is (30×60, 45×90, 60×120) for the medium-scale feature map (10×20, 15×30, 20×40), and the size of the anchor is (30×60, 45×90, 60×120) for the large-scale feature map (16×16). The detection head includes a classification branch and a regression branch. The classification branch uses a 1×1 convolutional layer to output the pedestrian category probability (2 categories: pedestrian or background). The regression branch uses a 1×1 convolutional layer to output the bounding box coordinate offset (4 values: offset of x, y, w, h). Both branches use the Sigmoid activation function to output the results.

[0040] Step 5, Training the Network Model: Perform end-to-end training. To fully utilize the infrared and visible light dual-modal features and improve the detection capability for small targets, the training process employs the following strategies: Batch and Image Size Optimization: A batch size of 8 is used to enhance gradient stability by leveraging the diversity of dual-modal data. The input image size is set to 512×512 to increase resolution and capture small targets at a distance (such as pedestrians or vehicles in the nighttime LLVIP dataset). Training Evolutions and Mixed Precision: The number of training epochs is 100 to increase the network's adaptability to modal differences. Mixed precision training with amp=False is used to ensure the stability of gradient accuracy when processing dual-modal information.

[0041] The optimizer and regularization use the AdamW optimizer, which is better suited for handling complex loss surfaces with bimodal features. The initial learning rate is lr_0 = 0.001, and this is lowered to avoid instability caused by modality conflicts. The momentum parameter is set to a slightly higher value of momentum = 0.937, which helps to overcome local optima. The weight decay is set to weight_decay = 5 × 10⁻⁶. -4Enhanced regularization mitigates bimodal overfitting. Learning rate scheduling is implemented. Cosine annealing with a learning rate of cos_lr=True is used to smoothly decay the learning rate, better adapting to differences between modalities. The final learning rate coefficient lrf=0.1 enhances convergence. The warmup epochs are extended to 3, allowing the model to adapt to the bimodal data distribution first.

[0042] The data augmentation strategy employs mosaic augmentation in the early stages of training to improve the model's robustness to scene complexity and target occlusion. Later, mosaic augmentation is disabled (close_mosaic=10) to allow the model to better adapt to the original bimodal data distribution. Through this training strategy, the network can fully learn the complementary information from infrared and visible light, enhancing overall detection accuracy and stability while maintaining sensitivity to small targets.

[0043] Step 6, Fine-tuning the network model: To further improve the model's detection performance and robustness in extreme infrared scenarios (such as complete darkness, heavy fog, and strong occlusion), the trained model is fine-tuned using a publicly available infrared pedestrian detection dataset to achieve scene-specific optimization. The specific fine-tuning process is as follows: A nighttime subset of the LLVIP dataset and an extreme weather subset of the FLIR ADAS dataset were selected as fine-tuning datasets. Both datasets primarily consist of infrared images and contain a large number of pedestrian samples with small targets, strong occlusion, and low signal-to-noise ratio, which can specifically improve the model's infrared scene adaptability. The fine-tuning dataset is approximately 20% the size of the training set, ensuring that the model learns the feature patterns of extreme scenes during fine-tuning without forgetting the generalization knowledge from the original training set.

[0044] Fine-tuning parameter settings: A transfer learning strategy is adopted, loading the model parameters trained in step 5 as pre-trained weights into the network. The initial learning rate during the fine-tuning phase is set to 0.0001, which is 10% of the original initial learning rate, to avoid the pre-trained parameters being corrupted due to an excessively high learning rate; the batch size is set to 4 to reduce the batch size and improve the model's sensitivity to extreme samples; the number of training epochs is set to 100, and an early stopping strategy is adopted (fine-tuning stops if there is no improvement in mAP on the validation set for 5 consecutive epochs); the optimizer remains AdamW, and the weight decay is set to 1×10. -4 The value is slightly lower than that during the training phase, reducing the impact of regularization constraints on learning from extreme samples.

[0045] Fine-tuning strategy design: A layered fine-tuning strategy is adopted, fixing the parameters of the bottom-level convolutional blocks (the first two basic convolutional blocks) of the feature extraction module, and only adjusting the parameters of the upper-level convolutional blocks, small target enhancement units, multi-scale convolutional residual units, as well as all parameters of the feature fusion module and detection module. This strategy can retain the basic visual features learned by the model in the original training set, focusing on optimizing the model's ability to extract and fuse features in extreme infrared scenes. In addition, during the fine-tuning process, only infrared images are data augmented, while visible light images remain in their original preprocessed state, simulating the situation where infrared modalities dominate in real-world scenes.

[0046] Validation of Fine-Tuning Results: After fine-tuning, the model performance was evaluated on the validation set of the fine-tuned dataset, focusing on the detection accuracy of small targets, strong occlusion scenes, and low signal-to-noise ratio scenes. Experimental results show that the fine-tuned model significantly improves mAP, small target detection accuracy, and recall rate in strong occlusion scenes under extreme infrared scenarios, effectively enhancing the model's adaptability to extreme scenarios.

[0047] Step 7, Save the model: Save the trained model parameters. In actual application, simply input the infrared and visible light images to be detected, and the detection results can be output through network inference.

[0048] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pedestrian detection method based on small target enhancement and bimodal multi-scale attention fusion, comprising the following steps: Step 1, Prepare the dataset: Obtain the public datasets for infrared and visible light dual-modality tests, including the LLVIP dataset and the KAIST dataset; divide each dataset into training set, validation set and test set according to a preset ratio; Step 2, construct a small target enhancement feature extraction module: perform convolution and downsampling operations on infrared and visible light images respectively to extract low-level features of each modality; to address the problem that small pedestrian features are easily submerged by the background, input the low-level features into the small target enhancement structure, and enhance the response of small pedestrian features through lightweight deconstruction of "group convolution - channel reduction - depthwise convolution - pointwise convolution"; Based on the enhanced features, a multi-scale extraction structure is input, and multi-branch design with shared convolutional kernels is used to capture multi-scale contextual information, thereby achieving comprehensive extraction of pedestrian features of different sizes. Step 3: Construct a multi-scale attention-based progressive feature fusion module: Input the features output from the multi-scale extraction structure into a dual-modal adaptive weight unit, and eliminate redundant features in the dual-modal model through dynamic weight allocation; then pass through a channel attention unit and a spatial attention unit in sequence to strengthen the key channel features and target region features respectively; finally, input the features into a cross-attention unit, and establish feature association by calculating cross-modal spatial association weights to generate the final high-level fusion features; Step 4, Construct the network model: Construct a dual-modal pedestrian detection network consisting of a small target enhanced feature extraction module, a multi-scale attention progressive feature fusion module, and a detection output module. The detection output module receives high-level fused features and outputs the detection results. Step 5, train the network model: input the training set into the detection network and train it with the goal of minimizing the combined loss function, which includes the original detection loss and the splicing information loss; Step 6, fine-tune the network model: use a publicly available infrared pedestrian detection dataset to fine-tune the trained detection network and optimize the model's detection performance; Step 7, Save the model: Select the optimal model parameters based on the evaluation metrics on the validation set and save them. The optimal model can directly input the infrared and visible light images to be detected and output pedestrian detection results.

2. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, First, local features of small targets are preserved through grouped convolution. Then, the number of channels is reduced to half of the input channels through 1×1 convolution. Subsequently, spatial features are extracted through depthwise convolution and the number of channels is restored through pointwise convolution. Finally, the output is regularized and activated. The number of parameters is significantly reduced compared with the traditional structure, while suppressing the interference of background noise on the features of small targets.

3. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, The multi-scale extraction structure contains three parallel convolutional branches, sharing 1×1 and 3×3 convolutional kernels, with dilation rates set to 1, 2, and 3, respectively; the output features of each branch are fused with residuals through element-wise addition.

4. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, The bimodal adaptive weight unit generates modal adaptive weights through global average pooling, dual fully connected layers, and activation functions, with a channel reduction ratio set to 4; the channel attention unit adaptively learns channel weights through one-dimensional convolution, without the need to manually set the convolution kernel size.

5. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, The channel attention unit adopts an improved squeeze-excitation mechanism, which performs global average pooling and global max pooling on the bimodal features respectively. The concatenated feature vector learns the channel importance weights through a dimensionality reduction-activation-dimensionality increase process.

6. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, The spatial attention unit performs global average pooling and global max pooling on the channel-enhanced features. After fusing the two pooling results, a double convolution operation is performed to generate spatial weighted coefficients, highlighting the features of target regions such as the pedestrian's head, shoulders, and legs.

7. The pedestrian detection method based on small target enhancement and dual-modal multi-scale attention fusion according to claim 1, characterized in that, The cross-attention unit calculates the spatial correlation weight between infrared and visible light features by multiplying the transposed feature matrix, and retains the original feature information by combining residual connections, thereby achieving deep complementarity of dual-modal features.

Images