Pedestrian detection method based on traffic scene

Abstract

The invention discloses a pedestrian detection method based on a traffic scene, and relates to the technical field of target detection, and the method comprises the steps: constructing a pedestrian detection data set based on the traffic scene, the pedestrian detection data set comprises visible light modal data and infrared spectrum modal data, and carrying out the time alignment and space alignment of different modal data, performing two-dimensional detection frame labeling on a pedestrian target in each image in the pedestrian detection data set; constructing and training a pedestrian detection model based on the marked pedestrian detection data set; and performing pedestrian detection of the traffic scene according to the converged pedestrian detection model. The technical problem that a traditional pedestrian detection mode is influenced by environmental condition changes and pedestrian feature differences in a traffic scene and is difficult to comprehensively and accurately capture pedestrian related information is solved, and the technical effects of accurate pedestrian detection, subsequent safety control and efficiency improvement are achieved.

Description

A pedestrian detection method based on traffic scenarios Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a pedestrian detection method based on traffic scenarios. Background Technology

[0002] Pedestrian detection in traffic scenarios is crucial for traffic safety, autonomous driving operation, and traffic management efficiency. Its detection accuracy directly impacts travel safety and the reliability of the traffic system. Existing technologies mostly employ single-modal devices combined with conventional detection models to achieve pedestrian detection, playing a certain role in simple scenarios with stable lighting. However, traffic scenarios have variable lighting conditions, such as nighttime, low light in tunnels, and strong backlighting. Furthermore, pedestrians vary greatly in size and viewing angle. Traditional technologies are easily affected by lighting interference, and single-modal data information is incomplete. They cannot effectively integrate multi-source features and separate pedestrians from background information, resulting in incomplete and inaccurate detection data, which fails to meet the needs of accurate pedestrian detection and safety management in traffic scenarios. Summary of the Invention

[0003] This application solves the technical problem that traditional pedestrian detection methods are difficult to comprehensively and accurately capture pedestrian-related information in traffic scenarios due to changes in environmental conditions and differences in pedestrian characteristics.

[0004] To address the aforementioned technical problems, this application proposes a pedestrian detection method based on traffic scenes. The method includes: constructing a pedestrian detection dataset based on traffic scenes, the pedestrian detection dataset including visible light modal data and infrared spectral modal data, and performing temporal and spatial alignment on the different modal data; and annotating pedestrian targets within each image in the pedestrian detection dataset with two-dimensional detection boxes; based on the annotated pedestrian detection dataset, constructing and training a pedestrian detection model, the construction and training of the pedestrian detection model including the following steps: Step 1: Constructing a dual-branch extraction layer of the pedestrian detection model, the dual-branch extraction layer based on a backbone network to separately extract visible light modal data and infrared spectral modal data. Step 1: Extract features from the light image and infrared spectral image, and output multi-scale features mapped by two branches respectively; Step 2: Construct a multi-scale spectral feature perception module to perform weighted fusion based on the lighting environment and the multi-scale features output by the two branches to construct multi-modal fusion features; Step 3: Construct an illumination robust feature decoupling module to decouple pedestrian features from background features of the multi-modal fusion features; Step 4: Input the decoupled pedestrian-related features into the pedestrian detection head to complete the category prediction and bounding box regression of pedestrian targets; Step 5: Perform convergence training of the pedestrian detection model using a loss function; Perform pedestrian detection in traffic scenes based on the converged pedestrian detection model.

[0005] This application proposes one or more technical solutions, which have at least the following technical effects:

[0006] This application constructs a pedestrian detection dataset containing two modalities in a traffic scenario, performs temporal and spatial alignment and annotation, extracts features from different modalities, fuses multi-scale features, separates pedestrian and background-related features, and achieves pedestrian category and location prediction and model training optimization to obtain a converged pedestrian detection model. Using this model, pedestrian detection in traffic scenarios is carried out, thereby accurately identifying pedestrian targets in traffic scenarios, making pedestrian detection results in traffic scenarios more accurate and reliable, meeting the needs of pedestrian detection and control in this scenario, and achieving the technical effect of accurate pedestrian detection and subsequent safety control and efficiency improvement. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 is a flowchart illustrating a pedestrian detection method based on a traffic scenario provided in an embodiment of this application.

[0009] Figure 2 is a schematic diagram of the decoupling and separation of pedestrian features and background features in a pedestrian detection method based on a traffic scene provided in an embodiment of this application. Detailed Implementation

[0010] This application provides a pedestrian detection method based on traffic scenarios, which solves the technical problem that traditional pedestrian detection methods are difficult to fully and accurately capture pedestrian-related information in traffic scenarios due to changes in environmental conditions and differences in pedestrian characteristics.

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0012] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0013] As shown in Figure 1, a pedestrian detection method based on a traffic scene includes:

[0014] A pedestrian detection dataset based on traffic scenarios is constructed. The pedestrian detection dataset includes visible light modal data and infrared spectral modal data. The different modal data are aligned in time and space. Two-dimensional detection boxes are labeled for pedestrian targets in each image in the pedestrian detection dataset.

[0015] Specifically, firstly, visible light cameras and infrared thermal imaging cameras are deployed in traffic scenarios such as urban intersections and road sections to simultaneously collect image data at different times to obtain self-collected visible light modal data and infrared spectral modal data; at the same time, publicly available pedestrian-related image data in traffic scenarios are acquired, and the self-collected data and open source data are integrated to obtain an initial dual-modal data set, namely the pedestrian detection dataset.

[0016] Next, timestamp information corresponding to the acquisition time is added to each frame of the integrated visible light modal data and infrared spectral modal data. Then, the timestamps of the two types of data are traversed, and visible light images and infrared images with completely consistent timestamps are paired to achieve synchronization of the two types of modal data in the time dimension, resulting in a time-aligned dual-modal image group.

[0017] Then, fixed landmarks such as streetlights are selected in the traffic scene. In each set of visible light and infrared images after time alignment, the pixel coordinates of the landmark in the two images are determined. By calculating the transformation matrix of the pixel coordinates, the pixel position of the infrared image is adjusted so that the pixel position of the landmark in the two types of images completely overlaps, thus completing the spatial alignment of the two types of modal data.

[0018] Then, for each image in the temporally and spatially aligned bimodal image group, the image annotation tool is used to identify pedestrian targets in the image one by one. For each pedestrian target, a two-dimensional rectangle covering the pedestrian is drawn in the image, and the pixel coordinates of the upper left corner and the lower right corner of the rectangle are recorded to complete the two-dimensional detection box annotation of all pedestrian targets in each image.

[0019] By collecting and integrating dual-modal data from multiple devices, synchronizing timestamps and calibrating markers to achieve data alignment, and using manual annotation tools to complete pedestrian bounding box annotations, the goal of constructing a well-structured and complete pedestrian detection dataset for traffic scenes was achieved.

[0020] Based on the labeled pedestrian detection dataset, a pedestrian detection model is constructed and trained. The construction and training of the pedestrian detection model includes the following steps:

[0021] Step 1: Construct a dual-branch extraction layer for the pedestrian detection model. This layer extracts features from visible light and infrared spectral images based on a backbone network, outputting multi-scale features mapped by the two branches respectively. Step 2: Construct a multi-scale spectral feature perception module to perform weighted fusion based on the lighting environment and the multi-scale features output by the dual branches, constructing multi-modal fusion features. Step 3: Construct an illumination robust feature decoupling module to decouple pedestrian features from background features in the multi-modal fusion features. Step 4: Input the decoupled pedestrian-related features into the pedestrian detection head to complete pedestrian target category prediction and bounding box regression. Step 5: Perform convergence training of the pedestrian detection model using a loss function.

[0022] Optionally, firstly, a YOLO convolutional neural network is used to build a dual-branch extraction layer, constructing a visible light branch and an infrared spectral branch respectively. Visible light images acquired in traffic scenes are input into the visible light branch, and simultaneously acquired infrared spectral images are input into the infrared spectral branch. Through layer-by-layer convolution and downsampling operations of the YOLO network, the output is obtained in the visible light branch. , , Features at three different scales are output in the infrared spectral branch. , , The features at three different scales are used to obtain multi-scale features for two branch mappings.

[0023] For visible light branch , , Using bilinear interpolation upsampling, , The scale is unified to To ensure consistency, a learnable weighting method is used to assign weight parameters to the three types of features after unifying the scale. , , The fused visible light features are obtained by weighted summation. ; For the infrared spectral branch , , Perform the same upsampling and weighted fusion operations to obtain the fused infrared light features. .

[0024] Next, we construct a multi-scale spectral feature sensing module, specifically as follows: = + ,in, This indicates the final fusion characteristics. , These represent the weighting parameters for visible light and infrared light characteristics, respectively. For visible light characteristics... Configure weight parameters Infrared light characteristics Configure weight parameters These weight parameters are continuously optimized through backpropagation during model training to adapt to the contribution levels of the two modalities under different lighting conditions. Then... and Multiplication and Multiply the two results and then sum them to obtain the final multimodal fusion feature. .

[0025] Then, a robust feature decoupling module for illumination is constructed. For the aforementioned multimodal fusion features, pedestrian features and background features are decoupled and separated for the visible light features and infrared light features respectively: For visible light features, the features are first refined through a 3*3 convolutional layer, and then pedestrian and background related features are extracted by max pooling and average pooling layers respectively. After concatenating the two in the channel dimension, the features are normalized by activation function and processed by a multilayer perceptron to complete feature decoupling and obtain the corresponding pedestrian and background related features; For infrared light features, the features are first refined through a 1*1 convolutional layer, and then fine-grained distinction is achieved by capturing nonlinear relationships through a multilayer perceptron. Pedestrian related features are highlighted by max pooling layer, and then pedestrian and background related features are obtained by normalizing by activation function and processing by a 1*1 convolutional layer.

[0026] Subsequently, the pedestrian-related features obtained after decoupling and separation are received, and multiple 3*3 convolutional kernels are configured to perform convolution operations on the input pedestrian-related feature map. By sliding the convolutional kernels across the feature map, local detail information related to the pedestrian target is extracted from the features, enhancing the feature's ability to represent the pedestrian target, resulting in pedestrian features refined through convolution. Then, a pooling window of fixed size is set and slides across the refined pedestrian feature map region by region, selecting the maximum feature value within each window as the output feature for that region. This compresses the spatial dimension of the features, reducing the number of feature parameters while retaining the key feature information of the pedestrian target, resulting in dimensionality-reduced pedestrian features.

[0027] Next, for the pedestrian category prediction task, the dimensionality-reduced pedestrian features are flattened into a one-dimensional feature vector and input into a network structure composed of multiple fully connected layers. Through linear transformations of the fully connected layers and non-linear processing of the activation function, the feature vector is mapped to the output dimension corresponding to the pedestrian category, outputting the category probability value of the pedestrian target, thus completing the pedestrian category prediction. Simultaneously, for the pedestrian bounding box regression task, the flattened one-dimensional feature vector is input into another set of fully connected layers. Through linear transformations of the fully connected layers, the bounding box parameters corresponding to the pedestrian target are output, specifically the predicted values ​​of the top-left and bottom-right corner coordinates of the pedestrian target bounding box, thus achieving the regression of the pedestrian target bounding box.

[0028] Finally, the total model loss is determined to be a weighted sum of the classification loss from pedestrian category prediction, the regression loss from bounding box regression, and the contrastive loss from feature decoupling. The traffic scene pedestrian detection dataset is input into the training model to obtain the predicted pedestrian categories and bounding boxes. The loss between this prediction and the ground truth categories and bounding boxes is calculated, and combined with the contrastive loss after feature decoupling, the total loss is obtained. The gradient of the total loss is backpropagated through backpropagation to update the parameters of each module of the model. This process of inputting data, calculating the loss, and updating parameters is repeated iteratively until the total loss stabilizes at a low level, completing the convergence training of the pedestrian detection model.

[0029] Pedestrian detection in traffic scenes is performed based on the converged pedestrian detection model.

[0030] In one embodiment of this application, a dual-modal image group to be detected is obtained by deploying a visible light sensor and an infrared spectral sensor in a traffic scene to simultaneously collect real-time traffic scene image data. The visible light sensor captures the color and texture details of the image under good lighting conditions, while the infrared spectral sensor captures the object contour information under low light conditions.

[0031] Next, the visible light image to be detected is input into the visible light branch of the converged pedestrian detection model, and the infrared spectral image is input into the infrared spectral branch. With the help of the YOLO backbone network that has been trained, convolution operation is performed to extract local features of the image, normalization operation is performed to stabilize the numerical distribution of features, activation function transformation is performed to enhance the nonlinear representation of features, and pyramid structure processing is performed to generate multi-scale features, thus obtaining visible light multi-scale features and infrared spectral multi-scale features containing pedestrian information of different sizes.

[0032] The model's multi-scale spectral feature perception module processes the multi-scale features output by the dual branches. It uses bilinear interpolation upsampling to unify the spatial size of features at different scales. Then, it calls the converged learnable weight parameters to perform weighted summation on the multi-scale features within the same modality to obtain single-modal fusion features. Subsequently, it performs cross-modal weighted fusion on the single-modal fusion features of visible light and infrared spectra to generate multi-modal fusion features that integrate complementary information from both modes.

[0033] The multimodal fusion features are then input into the illumination robustness feature decoupling module. For the visible light features, the features are refined through a 3*3 convolutional layer, and then pedestrian-related features are extracted using a max pooling layer and background-related features are extracted using an average pooling layer. After channel-dimensional concatenation, sigmoid activation normalization, and multilayer perceptron processing, pedestrian and background separation of visible light features is completed. For the infrared spectral features, the features are refined through a 1*1 convolutional layer, and then nonlinear relationships are captured by a multilayer perceptron, and pedestrian features are highlighted by a max pooling layer. After sigmoid activation normalization and 1*1 convolutional layer processing, pedestrian and background separation of infrared spectral features is completed, and pedestrian-related features are obtained.

[0034] The separated pedestrian-related features are then input into the pedestrian detection head. The features are further refined through convolutional layers and compressed through max pooling layers. The features are then flattened into a one-dimensional vector and input into a fully connected layer to perform pedestrian class prediction, outputting the probability of pedestrian or non-pedestrian class. At the same time, another set of fully connected layers is input to perform bounding box regression, outputting the coordinates of the upper left and lower right corners of the bounding box, thus obtaining the detection result of the pedestrian target in the current image.

[0035] Finally, for the subsequently acquired bimodal images of traffic scenes, the process of image input, bi-branch feature extraction, multimodal feature fusion, feature decoupling, and detection head prediction is repeatedly executed to continuously output real-time pedestrian detection results, covering pedestrian targets in traffic scenes.

[0036] By simultaneously acquiring dual-modal traffic scene images and using a convergent model to sequentially perform feature extraction, fusion, decoupling, and detection head prediction, the system achieves accurate and real-time detection of pedestrian targets in traffic scenes under different lighting conditions.

[0037] Furthermore, the method provided in this application embodiment includes:

[0038] The pedestrian detection dataset includes self-collected data and open-source data.

[0039] Specifically, visible light sensing devices and infrared spectral sensing devices are deployed in traffic scenarios such as traffic intersections, highways, and urban roads. For scenarios with different times, different lighting conditions, and different weather conditions, the two types of devices are activated simultaneously to collect image data, thereby obtaining self-collected data covering multiple scenarios.

[0040] Then, the visible light sensor and the infrared spectral sensor are configured with the same time trigger module, enabling both types of devices to start acquisition operations at the same time. This ensures that the acquisition time of each frame of visible light data and infrared spectral data is consistent, achieving time alignment of the self-acquired data. Next, fixed facilities in the acquisition scene, such as traffic signs, are selected as landmarks. The pixel coordinates of these landmarks in both the visible light and infrared spectral data are determined. The pixel positions in the infrared spectral data are adjusted through coordinate mapping to make the positions of the landmarks coincide in both types of data, achieving spatial alignment of the self-acquired data.

[0041] Next, we used public datasets to retrieve and obtain publicly available traffic scene-related multimodal datasets, including open-source datasets such as Multi-SpectralRoadScenarios and LLVIP. From these open-source datasets, we selected visible light modal data and infrared spectral modal data containing pedestrian information in traffic scenes as open-source data.

[0042] Finally, the self-collected data and the selected open-source data that have been aligned are uniformly converted to the same image resolution and storage format. Then, the two types of data are summarized and integrated to obtain a pedestrian detection dataset containing visible light modal data and infrared spectral modal data.

[0043] By using two sensors to collaboratively collect and align self-collected data, calling a specified public dataset to obtain open-source data, and integrating the two types of data in a unified format, the goal of constructing a pedestrian detection dataset for traffic scenarios that covers multi-scenario and multi-modal information was achieved.

[0044] Furthermore, the method provided in this application embodiment includes:

[0045] The dual-branch extraction layer uses YOLO's convolutional neural network as the backbone network and outputs feature maps of at least three different scales in the visible light branch and the infrared spectral branch, respectively, to characterize different pedestrian sizes and viewing angles.

[0046] Specifically, a dual-branch extraction layer is first constructed using a YOLO convolutional neural network, which builds a visible light branch corresponding to the visible light mode and an infrared spectral branch corresponding to the infrared spectral mode. Visible light images collected in traffic scenarios are input into the visible light branch, and infrared spectral images collected synchronously in the same scenario are input into the infrared spectral branch. With the feature extraction capability of the YOLO convolutional network, preliminary feature mapping processing is performed on the input images of the two modes.

[0047] In the visible light branch, a layer-by-layer convolutional downsampling method is employed. Multiple convolutional layers of the YOLO convolutional network sequentially perform convolution and downsampling operations on the input visible light image, generating three feature maps at different scales, denoted as follows: , , .in The dimensions correspond to 1 / 8 of the height H and 1 / 8 of the width W of the input image, and the number of channels is set to 256; The dimensions correspond to 1 / 16 of the height H and 1 / 16 of the width W of the input image, and the number of channels is set to 512; The dimensions correspond to 1 / 32 of the height H and 1 / 32 of the width W of the input image, and the number of channels is set to 512.

[0048] In the infrared spectral branch, the same layer-by-layer convolutional downsampling method as in the visible light branch is used to perform the same convolution and downsampling process on the input infrared spectral image, generating three feature maps at different scales, denoted as follows: , , .in The dimensions correspond to 1 / 8 of the height H and 1 / 8 of the width W of the input image, and the number of channels is set to 256; The dimensions correspond to 1 / 16 of the height H and 1 / 16 of the width W of the input image, and the number of channels is set to 512; The dimensions correspond to 1 / 32 of the height H and 1 / 32 of the width W of the input image, and the number of channels is set to 512.

[0049] Furthermore, in the feature map of the output above, H and W represent the height and width of the input image, respectively. , , This indicates the number of channels at different scales, which can be set to 256, 512, and 512 respectively.

[0050] By constructing a dual-branch extraction layer using a YOLO convolutional network, and performing layer-by-layer convolutional downsampling operations in the two branches to generate feature maps of a specified scale and number of channels, the effect of outputting multi-scale feature maps that adapt to different pedestrian sizes and viewpoints is achieved.

[0051] Furthermore, the method provided in this application embodiment includes:

[0052] Upsampling is used to unify multi-scale features to the same size, and learnable weight parameters are used to fuse features at different scales, constructing fused visible light and infrared light features, as follows:

[0053] = + + ; = + + ;in, Characterizing the visible light features of fusion, , , They respectively characterize the visible light features at different scales. Characterize the upsampling operation. Characterize the fused infrared light features. , , They respectively characterize infrared light features at different scales.

[0054] Optionally, for pedestrian detection in traffic scenarios, the visible light branch output... , , These three types of features at different scales are analyzed using bilinear interpolation upsampling. and Perform upsampling operations separately. The initial scale is 1 / 16 of the height and 1 / 16 of the width of the input image. The initial scale is 1 / 32 of the height and 1 / 32 of the width of the input image. Through upsampling, the scale of these two elements is unified to the same value. Consistent input image height of 1 / 8 and width of 1 / 8 ensure that the three types of visible light features have the same spatial dimensions.

[0055] Next, learnable weights are used to weight the samples to a uniform size. Upsampling Upsampling Assign learnable weight parameters separately , , These weight parameters are continuously optimized through backpropagation during model training to match the actual contribution of features at different scales. Then... and Multiplication, upsampling and Multiplication, upsampling and Multiply these three results, then sum them to obtain the fused visible light characteristics. .

[0056] For infrared light branch output , , These three different scale features, using the bilinear interpolation upsampling method, are... and Perform an upsampling operation, The initial input image has a height of 1 / 16 and a width of 1 / 16 scale, and The initial input image has a height of 1 / 32 and a width of 1 / 32, which are then uniformly adjusted to match the dimensions of the original image. Using the same input image with a height of 1 / 8 and a width of 1 / 8, the size of infrared light features is standardized.

[0057] Then, learnable weights consistent with the visible light branch are used for weighting, resulting in a uniform-size... Upsampling Upsampling Configure the corresponding learnable weight parameters , , After model training and optimization, and Multiplication, upsampling and Multiplication, upsampling and Multiply these three results, then sum them to obtain the fused infrared light characteristics. .

[0058] By using bilinear interpolation upsampling to unify the multi-scale feature size and a learnable weighted method to fuse features of different scales, the goal of integrating multi-scale information from visible and infrared light was achieved to enhance feature representation capabilities.

[0059] Furthermore, as shown in Figure 2, the method provided in this application embodiment includes:

[0060] The visible light features in the multimodal fusion feature set are first refined using a 3*3 convolutional layer, then pedestrian-related features are extracted using a max pooling layer and background-related features are extracted using an average pooling layer. The outputs of the two pooling layers are concatenated along the channel dimension, and feature normalization is performed using an activation function. Finally, feature decoupling is performed using a multilayer perceptron to obtain pedestrian-related and background-related features. The infrared light features in the multimodal fusion feature set are first refined using a 1*1 convolutional layer, then connected to a multilayer perceptron layer to capture nonlinear relationships. After fine-grained feature differentiation, pedestrian-related features are highlighted using a max pooling layer, and feature normalization is performed using an activation function layer. Finally, decoupling is performed using a 1*1 convolutional layer to obtain pedestrian-related and background-related features.

[0061] Specifically, regarding the visible light characteristics after multimodal fusion A 3x3 convolutional kernel is used to slide pixel-by-pixel across the visible light feature map, extracting detailed feature information through local weighted summation, thus refining the visible light features. Then, a fixed-size pooling window is slid across the refined feature map, selecting the maximum feature value within each window to extract salient features related to pedestrians; similarly, a fixed-size pooling window is used to select the average feature value within each window to extract global features related to the background.

[0062] Next, the pedestrian features output from the max pooling layer and the background features output from the average pooling layer are concatenated along the channel direction to integrate the information from both types of features and expand the feature dimension. Then, a sigmoid activation function is applied to map the concatenated feature values ​​to the range of 0 to 1, achieving feature normalization and enhancing feature stability. Finally, a multilayer perceptron is connected, and multiple fully connected layers perform nonlinear transformations on the normalized features, completing the feature decoupling operation and obtaining pedestrian-related visible light features. Visible light features related to the background .

[0063] Infrared light characteristics after multimodal fusion The infrared light feature map is processed by a 1*1 convolutional kernel to adjust the channel dimension of the features while retaining key information, thus refining the infrared light features. Subsequently, a multilayer perceptron is connected, and multiple fully connected layers perform multiple nonlinear transformations on the refined infrared light features to capture the complex nonlinear relationships between features, achieving fine-grained feature differentiation.

[0064] Next, a fixed-size pooling window is slid across the processed infrared feature map, selecting the maximum feature value within the window to highlight pedestrian-related salient features. Then, a sigmoid activation function is applied to map the feature values ​​to the range of 0 to 1, completing the normalization process for the infrared features. Finally, a 1x1 convolutional kernel is used to perform dimensionality adjustment and nonlinear transformation on the normalized features, completing the feature decoupling operation and obtaining the pedestrian-related infrared features. Infrared features related to the background .

[0065] Through the above steps, the effects of separating pedestrians from background features and enhancing the robustness of pedestrian detection under different lighting conditions are achieved.

[0066] Furthermore, the method provided in this application embodiment includes:

[0067] The loss function is as follows: + ;in, The loss function for action classification is represented by the binary cross-entropy loss function. The regression loss function is represented by the CIoU loss function. The contrast loss function is characterized by a triplet loss function.

[0068] Optionally, it can be first clarified that the loss function for training the pedestrian detection model is derived from the pedestrian action classification loss function. Bounding box regression loss function Comparative loss function The composition is as follows: + ,in Using a binary cross-entropy loss function, Using the CIoU loss function, A triplet loss function is adopted. Then, a pre-constructed pedestrian detection dataset for traffic scenes is selected, and the images in the dataset are input into the pedestrian detection model to be trained in batches to obtain the pedestrian category prediction probability, pedestrian bounding box prediction coordinates, and pedestrian-related features after feature decoupling.

[0069] For the pedestrian category prediction results, the binary cross-entropy loss method is used. The predicted pedestrian category probabilities output by the model and the actual pedestrian category labels in the dataset are substituted into the binary cross-entropy formula to calculate the difference between the predicted probabilities and the actual labels. Pedestrian labels are assigned a value of 1, and non-pedestrian labels are assigned a value of 0, thus obtaining the pedestrian action classification loss. .

[0070] For the pedestrian bounding box prediction results, the CIoU loss method is used. First, the intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box is calculated. Then, the Euclidean distance between their center points is calculated, and the aspect ratio difference between the two is also calculated. Substituting these three indicators into the CIoU loss formula, a regression loss that comprehensively reflects the overlap, position, and shape differences of the bounding boxes is obtained. .

[0071] For pedestrian-related features after feature decoupling, a triplet loss method is used. Anchor pedestrian features, positive examples of the same pedestrian, and negative examples of different pedestrians are selected from the batch data. The Euclidean distances between the anchor features and positive examples, and between the anchor features and negative examples, are calculated. These distances are then substituted into the triplet loss formula to ensure the positive example distance is less than the negative example distance while retaining a predetermined margin, resulting in a contrast loss used to enhance feature discriminative power. .

[0072] Finally, the classification loss Regression loss Compared with loss The total loss L of the model is obtained by summing the results. The backpropagation algorithm is then used to calculate the gradient of the total loss L with respect to the parameters of each module in the model. The Adam optimizer is then used to update the parameters of the dual-branch extraction layer, the multi-scale fusion module, the feature decoupling module, and the pedestrian detection head based on the gradient information. The process of batch data input, model prediction, loss calculation, gradient backpropagation, and parameter update is repeated iteratively until the total loss of the model on the validation set stabilizes and no longer decreases significantly, thus completing the convergence training of the pedestrian detection model.

[0073] By combining multiple types of loss functions to calculate classification, regression, and feature discrimination losses respectively, and combining backpropagation and iterative training steps for updating parameters with optimizers, the pedestrian detection model was able to converge and improve detection accuracy and robustness.

[0074] Furthermore, the method provided in this application embodiment includes:

[0075] The contrastive loss function is expressed as follows: ;in, This represents the total number of triplet samples used in the loss calculation during the training batch. This is the index for triplet samples. Indicates the first Anchor input samples for a triplet sample, Indicates the first The positive matching samples of the triplet samples that belong to the same category as the anchor input sample. This indicates a negative match sample that does not belong to the same category as the anchor input sample. For feature mapping function, It is a constant parameter.

[0076] Specifically, in the training batch of the pedestrian detection model, N sets of triplet samples are selected, and each triplet contains anchor input samples. Positive matching samples negative matching samples The anchor input sample is the feature of a certain pedestrian, the positive matching sample is the pedestrian feature of different modalities corresponding to the same pedestrian, and the negative matching sample is the background feature corresponding to the pedestrian feature.

[0077] Next, the feature mapping function f() of the model is used to input samples to the anchor points in each triplet. Positive matching samples negative matching samples Perform feature mapping to obtain the corresponding feature vectors. , , These feature vectors come from pedestrian-related features and background-related features output by the model feature decoupling module.

[0078] Then, the Euclidean distance squared method is used to calculate the anchor point feature vector. positive matching feature vector The distance between them yields the distance terms for features of the same category. Simultaneously calculate the anchor point feature vector. Matching negative feature vectors The distance between them yields distance terms for different categories of features. .

[0079] Substituting the distance terms of the same category, the distance terms of different categories, and the preset constant parameter α into the triplet loss formula, the loss value of each triplet is calculated. Then, the loss values ​​of N triplet groups in the training batch are summed to obtain the contrastive loss. The constant parameter α is used to set the distance margin, ensuring that the distance between features of the same category is less than the difference between the distances between features of different categories, which is at least α.

[0080] Finally, the losses will be compared. Loss related to pedestrian category Bounding box regression loss The total loss of the model is obtained by summing the results. The gradient of the total loss with respect to the model parameters is calculated using the backpropagation algorithm. The parameters of components such as the feature decoupling module are updated through the optimizer, so that the distance between different modal features of the same pedestrian is reduced and the distance between pedestrian and background features is increased.

[0081] By selecting triplet samples, feature mapping, calculating feature distance, summing to obtain contrast loss and participating in model parameter updates, the effect of strengthening the distinction between pedestrian features and background features and improving the model's feature representation ability is achieved.

[0082] Furthermore, the method provided in this application embodiment includes:

[0083] The backbone network adopts a convolutional neural network based on YOLO, ResNet, and MobileNet structures.

[0084] Specifically, firstly, a dual-branch extraction layer is built, which includes two independent feature extraction branches, corresponding to visible light images and infrared spectral images in traffic scenarios, respectively, to realize feature extraction of two types of modal data and match the feature capture requirements under different lighting conditions.

[0085] Based on convolutional neural networks (CNNs) well-known to those skilled in the art, the possible architectures for the backbone network are determined, including CNNs with structures such as YOLO, ResNet, and MobileNet. To improve the computational efficiency and reduce computational complexity of the model, a YOLO-based backbone network is selected, which integrates a series of basic operations:

[0086] First, a convolution operation is performed, using a convolution kernel of a preset size to slide across local pixel regions of the input image, extracting local features through weighted summation. Next, a normalization operation is performed, adjusting the numerical distribution of features through batch normalization to avoid gradient vanishing or exploding problems during training. Then, a nonlinear transformation is introduced through an activation function to enhance the network's ability to represent complex features. At the same time, this YOLO network structure incorporates a pyramid network structure, which can perform multi-scale downsampling of features to obtain feature maps at different levels, making full use of multi-scale information.

[0087] The visible light image of a well-lit daytime traffic scene is input into the corresponding feature extraction branch. Feature extraction is completed through the operation process of the YOLO backbone network described above: convolution operation captures the color and texture details of the image, normalization operation stabilizes the feature distribution, activation function enhances nonlinear representation, and pyramid structure adapts to pedestrian targets of different sizes, finally obtaining visible light features containing rich color details.

[0088] The infrared spectral image of the traffic scene under low light conditions is input into another feature extraction branch. The feature extraction is also performed through the operation process of the YOLO backbone network: convolution operation captures the contour information of the object, normalization and activation function optimize the feature quality, pyramid structure adapts to pedestrian targets from different perspectives, and finally the infrared spectral features that can enhance the contour of objects in low light are obtained.

[0089] By constructing a dual-branch feature layer, selecting a YOLO-structured backbone network, performing convolutional normalization activation and pyramid structure processing, and extracting features from two types of images respectively, the system effectively captures pedestrian features under different lighting conditions, while improving model computational efficiency and enhancing the robustness of pedestrian detection for different sizes and viewpoints.

[0090] In summary, the pedestrian detection method based on traffic scenarios provided in this application has the following technical effects:

[0091] This application constructs a pedestrian detection dataset by collecting visible light modal data and infrared spectral modal data in traffic scenarios. It performs temporal and spatial alignment on different modal data and annotates pedestrian targets within the images with two-dimensional detection boxes. Based on this annotated dataset, a pedestrian detection model is constructed. A dual-branch extraction layer extracts multi-scale features from the two modal images respectively; a multi-scale spectral feature perception module weights and fuses multi-modal features according to the lighting environment; a lighting robustness feature decoupling module separates pedestrian and background features; and a loss function is used to perform model convergence training. The decoupled pedestrian-related features, pedestrian category prediction, and bounding box regression results are obtained. The category and location information of pedestrian targets are calculated, and the model parameters are iteratively optimized using the loss function for adjustment. This accurately detects pedestrian targets in traffic scenarios, improving the robustness and accuracy of pedestrian detection in traffic scenarios under different lighting conditions. The detection results are more precise and reliable, achieving the technical effects of accurate pedestrian detection and subsequent safety management and efficiency improvement.

[0092] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0093] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A pedestrian detection method based on traffic scenarios, characterized in that, The method includes: constructing a pedestrian detection dataset based on traffic scenes, the pedestrian detection dataset including visible light modal data and infrared spectral modal data, and performing temporal and spatial alignment on the different modal data; and annotating the pedestrian targets in each image of the pedestrian detection dataset with two-dimensional detection boxes; based on the annotated pedestrian detection dataset, constructing and training a pedestrian detection model, the construction and training of the pedestrian detection model including the following steps: Step 1: Constructing a dual-branch extraction layer of the pedestrian detection model, the dual-branch extraction layer extracting features from the visible light image and the infrared spectral image respectively based on the backbone network. Step 1: Output the multi-scale features of the two branch mappings respectively; Step 2: Construct a multi-scale spectral feature perception module to perform weighted fusion based on the lighting environment and the multi-scale features output by the two branches to construct multi-modal fusion features; Step 3: Construct a lighting robust feature decoupling module to decouple and separate pedestrian features from background features in the multi-modal fusion features; Step 4: Input the decoupled and separated pedestrian-related features into the pedestrian detection head to complete the category prediction and bounding box regression of the pedestrian target; Step 5: Perform convergence training of the pedestrian detection model using the loss function; Perform pedestrian detection in traffic scenes based on the converged pedestrian detection model.

2. The pedestrian detection method based on traffic scenarios as described in claim 1, characterized in that, The dual-branch extraction layer uses YOLO's convolutional neural network as the backbone network and outputs feature maps of at least three different scales in the visible light branch and the infrared spectral branch, respectively, to characterize different pedestrian sizes and viewing angles.

3. The pedestrian detection method based on traffic scenarios as described in claim 2, characterized in that, Upsampling is used to unify multi-scale features to the same size, and learnable weight parameters are used to fuse features at different scales, constructing fused visible light and infrared light features, as follows: = + + ； = + + ;in, Characterizing the visible light features of fusion, 、 、 They respectively characterize the visible light features at different scales. Characterize the upsampling operation. Characterize the fused infrared light features. 、 、 They respectively characterize infrared light features at different scales.

4. The pedestrian detection method based on traffic scenarios as described in claim 1, characterized in that, The process involves decoupling pedestrian and background features from the multimodal fusion features. This includes: refining the visible light features from the multimodal fusion features using a 3x3 convolutional layer, extracting pedestrian-related features using a max pooling layer, and extracting background-related features using an average pooling layer; concatenating the channel-dimensional features from the two pooling layers, performing feature normalization using an activation function, and then decoupling the features using a multilayer perceptron to obtain pedestrian-related and background-related features; and refining the infrared light features from the multimodal fusion features using a 1x1 convolutional layer, connecting to a multilayer perceptron layer to capture nonlinear relationships, performing fine-grained feature differentiation, highlighting pedestrian-related features using a max pooling layer, normalizing the features using an activation function layer, and then decoupling the features using a 1x1 convolutional layer to obtain pedestrian-related and background-related features.

5. The pedestrian detection method based on traffic scenarios as described in claim 1, characterized in that, The loss function is as follows: + ;in, The loss function for action classification is represented by the binary cross-entropy loss function. The regression loss function is represented by the CIoU loss function. The contrast loss function is characterized by the triplet loss function.

6. The pedestrian detection method based on a traffic scenario as described in claim 5, characterized in that, The contrastive loss function is expressed as follows: ;in, This represents the total number of triplet samples used in the loss calculation during the training batch. This is the index for triplet samples. Indicates the first Anchor input samples of triplet samples, Indicates the first The positive matching samples of the triplet samples that belong to the same category as the anchor input sample. This indicates a negative match sample that does not belong to the same category as the anchor input sample. For feature mapping function, It is a constant parameter.

7. The pedestrian detection method based on a traffic scenario as described in claim 1, characterized in that, The pedestrian detection dataset includes self-collected data and open-source data.

8. The pedestrian detection method based on traffic scenarios as described in claim 1, characterized in that, The backbone network adopts a convolutional neural network based on YOLO, ResNet, and MobileNet structures.

Images