Multi-scale pedestrian detection method based on information fusion in night scene

Abstract

The present application relates to the technical field of image processing, and more particularly to a multi-scale pedestrian detection method based on information fusion in a night scene, wherein a feature extraction module realizes multiple fusions of multi-scale features while adaptively focusing on different regions through bottom-up feature extraction and a mixed self-attention feature enhancement operation; an auxiliary detection module deeply distinguishes foreground and background in a night low-light scene and identifies the external contour and local features of pedestrians; and a detection output module with a fusion confidence penalty strategy reduces the confidence score of a severely occluded detection box and reselects it, thereby reducing the target miss detection rate in a night low-light scene and improving the detection capability of the model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a multi-scale pedestrian detection method based on information fusion in nighttime scenes. Background Technology

[0002] In the field of computer vision, pedestrian detection technology is one of the most widely applied and popular research areas. The pedestrian detection task mainly refers to the accurate detection and localization of pedestrian targets in images, videos, and video stream data using computer vision-related techniques. This task is essentially a classification and regression process. In real life, pedestrian detection plays a crucial role and has high application value in fields such as autonomous driving, intelligent surveillance, intelligent robots, and human-computer interaction.

[0003] For existing pedestrian detection technologies, most solutions still rely on classic object detection algorithms to detect pedestrian targets in scenes. Based on the detection approach, these can be divided into single-stage detection algorithms represented by the YOLO series network models and two-stage detection algorithms represented by the Faster R-CNN network model. However, most classic object detection algorithms lack specific consideration for the detection scene, especially in low-light nighttime scenes, where the robustness of these mainstream detection algorithms is affected. This includes:

[0004] (1) In low light scene at night, the contrast between pedestrians and background is low due to insufficient light. If the background target has a greater consistency with the pedestrian in appearance, the model has a weak ability to distinguish between pedestrians and background, and is prone to false detection.

[0005] (2) Since most current pedestrian detection datasets are based on camera images and calibration, and cameras exhibit a "near-large, far-small" pattern when capturing images, pedestrian targets at closer viewpoints are generally larger in scale, while those at farther viewpoints are generally smaller in scale. Because small-scale pedestrian targets have relatively low resolution, the algorithm is prone to problems such as limited learned feature information or weak feature representation ability during feature extraction, making it difficult to have high sensitivity for pedestrian targets of different scales, which can easily lead to missed detections or false detections.

[0006] (3) In low-light nighttime scenarios, severe occlusion of pedestrian targets negatively impacts the overall performance of the detector. Analysis of images in the pedestrian detection dataset reveals two main types of pedestrian occlusion: intra-class occlusion and inter-class occlusion. Intra-class occlusion refers to mutual occlusion between pedestrian targets. Inter-class occlusion refers to interference from background information, including buildings, trees, vehicles, items carried by the pedestrian, and items carried by other pedestrians nearby. Both intra-class and inter-class occlusion reduce the proportion of the visible area of ​​the pedestrian's entire body, making feature extraction difficult and reducing the information required for inference by the detection module. Furthermore, it affects the accuracy of pedestrian target localization, thus impacting the overall performance of the pedestrian detection algorithm. Summary of the Invention

[0007] In view of this, the present invention aims to provide a multi-scale pedestrian detection method based on information fusion in nighttime scenes, which aims to improve the ability of pedestrian detectors to detect multi-scale pedestrian targets and occluded pedestrian targets in low-light nighttime scenes, and reduce the target false detection rate.

[0008] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0009] A multi-scale pedestrian detection method based on information fusion for nighttime scenes includes:

[0010] Obtain a dataset including nighttime pedestrian images and their recognition results, and preprocess the dataset to obtain a training set;

[0011] A multi-scale pedestrian detection network is constructed and trained using the obtained training set to obtain a pedestrian detection model. The multi-scale pedestrian detection network includes a feature extraction module, a region proposal module, an auxiliary detection module, and a detection output module. Specifically: the feature extraction module extracts multi-scale features from the input nighttime pedestrian image; the region proposal module receives the multi-scale features from the feature extraction module, generates a candidate region set, and generates predicted detection boxes and their confidence scores based on the candidate region set; the auxiliary detection module extracts the external contour and local features of the pedestrian from the multi-scale features from the feature extraction module; based on the obtained external contour and local features, it forms predicted detection boxes and their confidence scores for the local pedestrian detection from the pedestrian candidate box set output by the region proposal module; the detection output module integrates the predicted detection boxes and their confidence scores output by the region proposal module and the auxiliary detection module, and applies a confidence penalty strategy to the integrated predicted detection boxes based on the integrated confidence scores, outputting the final detection boxes.

[0012] The nighttime pedestrian image to be detected is input into the trained pedestrian detection model, and the corresponding pedestrian recognition result is output.

[0013] Furthermore, in the feature extraction module: multi-scale feature extraction is performed on the nighttime pedestrian image; upsampling is performed on the feature map at each scale; the upsampled features are fused with features of the same scale for the first time; downsampling is performed on each feature after the first feature fusion; the downsampled features are fused with features of the same scale for the second time; and hybrid self-attention enhancement with residual operation is performed on each feature after the second feature fusion to obtain the output features at the corresponding scale.

[0014] Furthermore, the hybrid self-attention enhancement process includes convolutional operations on the input features to obtain three sets of features, and performing parallel operations on the three sets of features in two branches: In the first branch, each of the three sets of features undergoes continuous fully connected operations and feature scaling operations, and the operation results corresponding to the three sets of features are integrated to obtain the output features of the first branch; In the second branch, each of the three sets of features undergoes self-attention operations, and the self-attention operation results corresponding to the three sets of features are integrated to obtain the output features of the second branch; The output features of the first branch and the second branch are weighted and summed to obtain the output features of the hybrid self-attention enhancement.

[0015] Furthermore, the self-attention operation in the second branch includes: projecting each of the three feature groups into a query, key, and value; and performing the following operation on each feature group:

[0016] ;

[0017] in, This represents a local pixel region N centered at (i,j) and with a width of s within the l-th feature group. s Attention features of (i,j), This represents a query for the l-th feature group. The key of the l-th feature group f represents the value of the l-th feature group. ij f represents the input tensor of pixel (i,j). ab For a local pixel region N s The tensor of pixel (a,b) in (i,j), where d represents The number of dimensions; integrate the attention features of each feature group to obtain the attention features corresponding to each feature group.

[0018] Furthermore, the region proposal module includes: an RPN network, which receives multi-scale features from the feature extraction module, performs RoI-Align operation on the output of the RPN network to truncate it, and generates a candidate region set; and an R-CNN network, which identifies the candidate region set to obtain predicted detection boxes and their confidence scores.

[0019] Furthermore, in the auxiliary detection module: it receives a set of candidate regions from the region proposal module and features extracted by the feature extraction module after RoI-Align operation; based on the set of candidate regions, it uses the extracted features to detect pedestrian contours and pedestrian limbs, and outputs pedestrian contour detection boxes and contour probability scores, as well as pedestrian leg / arm detection boxes and limb probability scores.

[0020] Furthermore, the auxiliary detection module includes: a pedestrian contour detection branch, used to detect pedestrian contour feature information from the candidate region set using the truncated features; a pedestrian leg detection branch, used to detect the lower half of the truncated features to obtain pedestrian leg feature information; and a pedestrian arm detection branch, used to divide the upper half of the truncated features into four equal parts, truncating the leftmost and rightmost parts to obtain pedestrian arm feature information. Each of the three detection branches includes three cascaded convolutional layers and three cascaded fully connected layers along the data transmission direction, with the last convolutional layer cascaded with the first fully connected layer. The R-CNN network combines the pedestrian contour feature information, pedestrian leg feature information, and pedestrian arm feature information to output pedestrian contour detection boxes, pedestrian leg detection boxes, and pedestrian arm detection boxes, as well as corresponding probability scores.

[0021] Furthermore, in the detection output module, the confidence scores from the region suggestion module and the auxiliary detection module are first weighted and summed to obtain the integrated confidence score;

[0022] Based on the integrated confidence score, a confidence penalty strategy is applied using the following formula:

[0023] ;

[0024] in, Sorce represents the confidence score after processing with the i-th confidence penalty strategy. i Represents the confidence score of the i-th integrated box. i This represents the i-th detection box output by the region suggestion module and the auxiliary detection module. max This indicates the detection box with the highest confidence score in the output detection boxes of the region suggestion module and the auxiliary detection module. D represents the Euclidean distance calculation, L represents the diagonal length calculation of the smallest rectangle, and N represents the region suggestion module and the auxiliary detection module. t This represents the preset threshold of the traditional nonmaximum suppression algorithm.

[0025] Furthermore, the multi-scale pedestrian detection system is trained using the overall loss function expressed in the following formula:

[0026] L=L cls-rpn +L reg-rpn+L cls-rcnn +L reg-rcnn ;

[0027] Where L represents the overall loss function, L cls-rpn Let L represent the classification loss function of the RPN network. reg-rpn L represents the regression loss function of the RPN network. cls-rcnn Let L represent the classification loss function of the R-CNN network. reg-rcnn Compared with existing technologies, the regression loss function of the R-CNN network can achieve the following beneficial effects:

[0028] This invention presents a multi-scale pedestrian detection method based on information fusion for nighttime scenes. Through a bottom-up feature extraction module, it achieves multiple fusions of multi-scale features, allowing lower-level detail information to be better transmitted to the top-level feature map, thus providing the detection module with richer and more detailed multi-scale feature information. By using a hybrid self-attention feature enhancement operation in the feature extraction module, it can adaptively focus on different regions, obtaining a larger receptive field and richer contextual information, thereby acquiring richer detail features and more accurately distinguishing the background from the target. Furthermore, this invention constructs an auxiliary detection module to effectively achieve deep differentiation between foreground and background in low-light nighttime scenes, and to identify and detect occluded pedestrian targets by recognizing the external contours and local features of pedestrians. In the post-processing stage, a detection output module with a fused confidence penalty strategy is constructed to retain detection boxes of occluded pedestrian targets as much as possible, and to reduce the confidence score of severely occluded detection boxes for re-screening. This alleviates the problem of incorrect suppression of detection boxes to a certain extent, helping to reduce the target false negative rate in low-light nighttime scenes and improve the model's detection capability. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0030] Figure 1 A flowchart illustrating the multi-scale pedestrian detection method based on information fusion in nighttime scenes as described in the embodiments of the present invention;

[0031] Figure 2 A schematic diagram of the multi-scale pedestrian detection network described in the embodiments of the present invention;

[0032] Figure 3 A schematic diagram of the feature extraction module described in an embodiment of the present invention;

[0033] Figure 4A schematic diagram illustrating the hybrid self-attention enhancement described in an embodiment of the present invention;

[0034] Figure 5 A comparison of the multi-scale pedestrian detection method described in the embodiments of the present invention with existing detection methods on the NightSurveillance dataset;

[0035] Figure 6 A comparison of the multi-scale pedestrian detection method described in the embodiments of the present invention with existing detection methods on the NightOwls dataset. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0038] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0039] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0040] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] like Figure 1 As shown in the embodiments of the present invention, the multi-scale pedestrian detection method based on information fusion in nighttime scenes includes:

[0042] S1: Obtain a dataset including nighttime pedestrian images and their recognition results, and preprocess the dataset to obtain a training set.

[0043] In this embodiment of the invention, the NightSurveillance dataset and the NightOwls dataset are used, and data augmentation strategies are applied to the datasets, including random pruning and random horizontal flipping.

[0044] S2: Construct a multi-scale pedestrian detection network and train the multi-scale pedestrian detection network using the obtained training set to obtain a pedestrian detection model.

[0045] Among them, multi-scale pedestrian detection networks, such as Figure 2 As shown, the system includes a feature extraction module, a region proposal module, an auxiliary detection module, and a detection output module. The feature extraction module obtains multi-scale features from the input nighttime pedestrian image; the region proposal module receives the multi-scale features from the feature extraction module, generates a candidate region set, and generates predicted detection boxes and their confidence scores based on the candidate region set; the auxiliary detection module obtains the external contour and local features of the pedestrian from the multi-scale features from the feature extraction module; based on the obtained external contour and local features, it forms predicted detection boxes and their confidence scores for detecting the local area of ​​the pedestrian from the pedestrian candidate box set output by the region proposal module; the detection output module integrates the predicted detection boxes and their confidence scores output by the region proposal module and the auxiliary detection module, and applies a confidence penalty strategy to the integrated predicted detection boxes based on the integrated confidence scores, outputting the final detection boxes.

[0046] In some embodiments, the data processing procedure in the feature extraction module is as follows: Figure 2 As shown, multi-scale feature extraction is performed on nighttime pedestrian images. For each extracted scale feature map, an upsampling operation is performed. The upsampled features are then fused with features of the same scale for the first time. Each feature after the first feature fusion is then downsampled, and the downsampled features are fused with features of the same scale for the second time. Finally, each feature after the second feature fusion is enhanced with hybrid self-attention using residual operations to obtain the output features at the corresponding scale. In this embodiment, bilinear interpolation is preferably used for upsampling, i.e., linear interpolation is performed on the features in two directions, thereby fully utilizing the pixel value information of the four pixels surrounding the target point and reducing information loss during feature map transmission.

[0047] In this embodiment of the invention, ResNet50 is used to learn the features of pedestrian images at night, resulting in feature maps F1, F2, F3, and F4 with gradually decreasing scales. The specific structure of the ResNet50 network is shown in Table 1 below.

[0048] Table 1: Specific Structure of ResNet50 Network

[0049]

[0050] Perform a 1×1 convolution operation on feature maps F1, F2, F3, and F4 respectively to convert them into 256 channels. The feature map F4 is subjected to a 2x bilinear interpolation operation and added to the corresponding elements of the feature map F3 to obtain feature Add1; feature Add1 is subjected to a 2x bilinear interpolation operation and added to the corresponding elements of the feature map F2 to obtain feature Add2; feature Add2 is subjected to a 2x bilinear interpolation operation and added to the corresponding elements of the feature map F1 to obtain feature Add3; feature Add3 is subjected to a 3×3 2D convolution operation and then added to the corresponding elements of feature Add2 to obtain feature Add4; feature Add2 is subjected to a 3×3 2D convolution operation and then added to the corresponding elements of feature Add1 to obtain feature Add5; feature Add5 is subjected to a 3×3 2D convolution operation and then added to the corresponding elements of feature map F4 to obtain feature Add6; features Add3, Add4, Add5, and Add6 are subjected to hybrid self-attention enhancement with residual operation and then subjected to a 3×3 2D convolution operation to obtain output features P1, P2, P3, and P4 respectively.

[0051] In some embodiments, the process of hybrid self-attention enhancement is as follows: Figure 3 As shown, a convolution operation is performed on the input features to obtain three sets of features. These three sets of features are then subjected to a two-branch parallel operation: In the first branch, each of the three sets of features undergoes a continuous fully connected operation and a feature scaling operation. The results of these operations are then integrated to obtain the output features of the first branch. In the second branch, each of the three sets of features undergoes a self-attention operation. The results of these self-attention operations are then integrated to obtain the output features of the second branch. Finally, the output features of the first branch and the second branch are weighted and summed to obtain the hybrid self-attention enhanced output features.

[0052] In the hybrid self-attention enhancement process provided in this embodiment of the invention, three parallel 1×1 convolution operations are performed on the input features of size H×W×C to obtain three sets of features. Each set of features includes N feature maps, and each feature map includes a sub-feature of size H×W×C / 3. That is, after three parallel 1×1 convolution operations, features of size (H×W×C / N)×3N are obtained. Continuous fully connected operations and feature scaling operations are performed on each of the three sets of features. The operation results corresponding to the three sets of features are concatenated to obtain the first branch output feature F. ful In the second branch, a self-attention operation is performed on each of the three feature groups, and then the results of the self-attention operations on the three feature groups are integrated to obtain the output feature F of the second branch. att The output features of the second branch of the first branch are weighted and summed to obtain the hybrid self-attention enhanced output feature F. out As shown in the following formula:

[0053] F out =w1×F ful +w2×F att ;

[0054] Where w1 and w2 represent the output features F of the first branch, respectively. ful Second branch output characteristic F att The corresponding weight values. In this embodiment of the invention, the weight values ​​w1 and w2 are learned and updated to their optimal values ​​through model training.

[0055] In some embodiments, in the second branch, each of the three sets of features is projected as a query, a key, and a value. The query represents the current focus or information need and is used to match the key; the key contains identifier information for each position in the input sequence and is used to be queried; the value stores the information content corresponding to the key and is weighted and summed according to the degree of matching of the query.

[0056] Perform the following self-attention operation on each feature group using the following formula:

[0057] ;

[0058] in, This represents a local pixel region N centered at (i,j) and with a width of s within the l-th feature group. s Attention features of (i,j), This represents a query for the l-th feature group. The key of the l-th feature group f represents the value of the l-th feature group. ij f represents the input tensor of pixel (i,j). ab For a local pixel region Ns The tensor of pixel (a,b) in (i,j), where d represents The number of dimensions;

[0059] The attention features of each feature group are concatenated and integrated to obtain the attention features corresponding to each feature group, as shown in the following formula:

[0060] ;

[0061] in, This represents a cascade of N attention features.

[0062] In some embodiments, the region proposal module includes an RPN network and an R-CNN network. The RPN network receives multi-scale features from the feature extraction module, performs RoI-Align operation on the output of the RPN network to truncate the region and generate a set of candidate regions. The R-CNN network identifies the set of candidate regions to obtain predicted detection boxes and their confidence scores.

[0063] In some embodiments, the auxiliary detection module receives a set of candidate regions from the region proposal module and features truncated by the feature extraction module using the RoI-Align operation. Based on the candidate region set, it uses the truncated features to detect pedestrian contours and limbs, outputting pedestrian contour detection boxes and contour probability scores, as well as pedestrian leg / arm detection boxes and limb probability scores. Specifically, the auxiliary detection module includes a pedestrian contour detection branch, a pedestrian leg detection branch, a pedestrian arm detection branch, and an R-CNN network. The R-CNN network comprises three cascaded convolutional layers and three cascaded fully connected layers along the data transmission direction. The last convolutional layer is cascaded with the first fully connected layer. The pedestrian contour detection branch uses truncated features to detect pedestrian contour features from a candidate region set. The pedestrian leg detection branch detects the lower half of the truncated features to obtain pedestrian leg features. The pedestrian arm detection branch divides the upper half of the truncated features into four equal parts, truncating the leftmost and rightmost parts to obtain pedestrian arm features. The R-CNN network combines these features to output pedestrian contour detection boxes, pedestrian leg detection boxes, and pedestrian arm detection boxes, along with their corresponding probability scores.

[0064] In some embodiments, the detection output module first performs a weighted summation of the confidence scores output from the region suggestion module and the auxiliary detection module to obtain the integrated confidence score. In this embodiment of the invention, the integrated confidence score is obtained by the following formula:

[0065] Sorcei =(1-μ)×P+μ×C;

[0066] Among them, Sorce i Let represent the i-th integrated confidence score, and μ represent the proportional weight of the confidence score. P represents the confidence score output by the region suggestion module, and C represents the confidence score output by the auxiliary detection module. In this embodiment of the invention, the confidence score weight μ is preferably set to 0.1.

[0067] This invention constructs a confidence penalty strategy, which penalizes detection boxes with low predicted confidence scores, rather than directly eliminating them as in traditional non-maximum suppression algorithms. This approach can retain as many candidate boxes as possible in occluded scenes, thus providing sufficient resources for subsequent filtering stages. Specifically, based on the integrated confidence scores, the confidence penalty strategy is implemented using the following formula:

[0068] ;

[0069] in, Box represents the confidence score after processing with the i-th confidence penalty strategy. i This represents the i-th detection box output by the region suggestion module and the auxiliary detection module. max This indicates the detection box with the highest confidence score in the output detection boxes of the region suggestion module and the auxiliary detection module. D represents the Euclidean distance calculation, L represents the diagonal length calculation of the smallest rectangle, and N represents the region suggestion module and the auxiliary detection module. t This represents the preset threshold of the traditional nonmaximum suppression algorithm.

[0070] In some embodiments, the multi-scale pedestrian detection system is trained using the overall loss function expressed in the following formula:

[0071] L=L cls-rpn +L reg-rpn +L cls-rcnn +L reg-rcnn ;

[0072] Where L represents the overall loss function, L cls-rpn Let L represent the classification loss function of the RPN network. reg-rpn L represents the regression loss function of the RPN network. cls-rcnn Let L represent the classification loss function of the R-CNN network. reg-rcnnThis represents the regression loss function of the R-CNN network. This invention mitigates gradient vanishing or exploding by using the outputs of the RPN and R-CNN networks in the multi-scale pedestrian detection network to participate in the loss function calculation. Furthermore, it forces the intermediate layer outputs to conform to the classification task expectations, avoiding the invalidation of intermediate features and ensuring that each layer of the network bears the responsibility for the task. Finally, the intermediate layer loss acts as a regularization term, reducing the risk of overfitting and allowing the model to learn more robust hierarchical features. Specifically:

[0073] Classification loss function L cls-rpn As shown in the following formula:

[0074] ;

[0075] Where, N rpn This represents the number of samples used in the training of the RPN network. This represents the score calculated by the RPN network predicting the category for the nth sample. express The corresponding truth label, CELoss, represents the cross-entropy loss;

[0076] Regression loss function L reg-rpn As shown in the following formula:

[0077] ;

[0078] in, This represents the regression offset of the candidate box predicted by the RPN network for the nth sample. express The corresponding truth label;

[0079] Classification loss function L cls-rcnn As shown in the following formula:

[0080] ;

[0081] Where, N rcnn This represents the number of samples used in the training of the R-CNN network. This represents the score calculated by the R-CNN network predicting the category for the nth sample. express The corresponding truth label;

[0082] Regression loss function L reg-rcnn As shown in the following formula:

[0083] ;

[0084] in, This represents the regression offset of the candidate box predicted by the R-CNN network for the nth sample. express The corresponding truth label.

[0085] Furthermore, in the training process of this embodiment of the invention, a stochastic gradient descent optimizer is used for 50 epochs of training, the momentum factor is set to 0.9, and a warm-up strategy is adopted for the learning rate. The initial learning rate is set to a low value, which is 1.25 × 10 in this invention. -4 In each epoch, from the first iteration to the 800th iteration, the learning rate increases linearly to 1.25 × 10⁻⁶. -3 Then it remains unchanged. By the 35th epoch, the learning rate is reduced to 10% of the original, and by the 40th epoch, the learning rate becomes 1% of the original.

[0086] S3: Input the nighttime pedestrian image to be detected into the trained pedestrian detection model, and output the corresponding pedestrian recognition result.

[0087] In this embodiment of the invention, the average false negative rate is used as the evaluation metric to assess the performance of the pedestrian detection algorithm. A lower average false negative rate for pedestrian targets indicates better algorithm performance. Experimental results are as follows: Figure 5 and Figure 6 As shown, on the NightSurveillance and NightOwls pedestrian detection datasets, the average false negative rate of this invention is lower than that of some popular pedestrian detection algorithms. Therefore, the experimental results fully demonstrate that this invention outperforms some popular pedestrian detection algorithms and exhibits higher detection capabilities in low-light nighttime scenes.

[0088] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-scale pedestrian detection method based on information fusion in nighttime scenes, characterized in that, include: Obtain a dataset including nighttime pedestrian images and their recognition results, and preprocess the dataset to obtain a training set; A multi-scale pedestrian detection network is constructed and trained using the obtained training set to obtain a pedestrian detection model. The multi-scale pedestrian detection network includes a feature extraction module, a region proposal module, an auxiliary detection module, and a detection output module. Specifically: the feature extraction module extracts multi-scale features from the input nighttime pedestrian image; the region proposal module receives the multi-scale features from the feature extraction module, generates a candidate region set, and generates predicted detection boxes and their confidence scores based on the candidate region set; the auxiliary detection module extracts the external contour and local features of the pedestrian from the multi-scale features from the feature extraction module; based on the obtained external contour and local features, it forms predicted detection boxes and their confidence scores for the local pedestrian detection from the pedestrian candidate box set output by the region proposal module; the detection output module integrates the predicted detection boxes and their confidence scores output by the region proposal module and the auxiliary detection module, and applies a confidence penalty strategy to the integrated predicted detection boxes based on the integrated confidence scores, outputting the final detection boxes. In the feature extraction module: multi-scale feature extraction is performed on nighttime pedestrian images. For each extracted feature map at each scale, an upsampling operation is performed. The upsampled features are then fused with features of the same scale for the first time. For each feature fused after the first time, a downsampling process is performed. The downsampled features are then fused with features of the same scale for the second time. For each feature fused after the second time, a hybrid self-attention enhancement with residual operation is performed to obtain the output features of the corresponding scale. The hybrid self-attention enhancement process includes convolutional operations on the input features to obtain three sets of features, followed by parallel operations on these three sets of features in two branches: In the first branch, each of the three sets of features undergoes consecutive fully connected operations and feature scaling operations, and the results of these operations are integrated to obtain the output features of the first branch; In the second branch, each of the three sets of features undergoes a self-attention operation, and the results of these self-attention operations are integrated to obtain the output features of the second branch; Finally, the output features of the first branch and the second branch are weighted and summed to obtain the hybrid self-attention enhanced output features. The nighttime pedestrian image to be detected is input into the trained pedestrian detection model, and the corresponding pedestrian recognition result is output.

2. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 1, characterized in that, The self-attention operations in the second branch include: Project each of the three sets of features into a query, key, and value; Perform the following operation on each feature group using the following formula: ； in, This represents a local pixel region N centered at (i,j) and with a width of s within the l-th feature group. s Attention features of (i,j), This represents a query for the l-th feature group. The key of the l-th feature group f represents the value of the l-th feature group. ij f represents the input tensor of pixel (i,j). ab For a local pixel region N s The tensor of pixel (a,b) in (i,j), where d represents The number of dimensions; The attention features of each feature group are integrated to obtain the attention features corresponding to each feature group.

3. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 1, characterized in that, The regional suggestion module includes: The RPN network receives multi-scale features from the feature extraction module, performs RoI-Align operation on the output of the RPN network to extract candidate region sets; The R-CNN network identifies candidate regions and obtains predicted detection boxes and their confidence scores.

4. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 3, characterized in that, In the auxiliary detection module: Receives a set of candidate regions from the region proposal module, and features extracted by the feature extraction module after RoI-Align operation; Based on the candidate region set, the extracted features are used to detect pedestrian contours and limbs, and output pedestrian contour detection boxes and contour probability scores, as well as pedestrian leg / arm detection boxes and limb probability scores.

5. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 4, characterized in that, The auxiliary detection module includes: The pedestrian contour detection branch is used to detect pedestrian contour feature information from the candidate region set using the truncated features; The pedestrian leg detection branch is used to detect the lower half of the truncated features to obtain pedestrian leg feature information. The pedestrian arm detection branch is used to divide the upper half of the truncated feature into four equal parts, truncating the leftmost part and the rightmost part, thereby obtaining pedestrian arm feature information. Each of the three detection branches includes three cascaded convolutional layers and three cascaded fully connected layers along the data transmission direction, with the last convolutional layer cascaded with the first fully connected layer; The R-CNN network combines pedestrian contour features, pedestrian leg features, and pedestrian arm features to output pedestrian contour detection boxes, pedestrian leg detection boxes, and pedestrian arm detection boxes, along with their corresponding probability scores.

6. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 5, characterized in that, In the detection output module, the confidence scores from the region suggestion module and the auxiliary detection module are first weighted and summed to obtain the integrated confidence score. Based on the integrated confidence score, a confidence penalty strategy is applied using the following formula: ； in, Sorce represents the confidence score after processing with the i-th confidence penalty strategy. i Represents the confidence score of the i-th integrated box. i This represents the i-th detection box output by the region suggestion module and the auxiliary detection module. max This indicates the detection box with the highest confidence score in the output detection boxes of the region suggestion module and the auxiliary detection module. D represents the Euclidean distance calculation, L represents the diagonal length calculation of the smallest rectangle, and N represents the region suggestion module and the auxiliary detection module. t This represents the preset threshold of the traditional nonmaximum suppression algorithm.

7. The multi-scale pedestrian detection method based on information fusion in nighttime scenes according to claim 5, characterized in that, The multi-scale pedestrian detection system is trained using the overall loss function expressed below: L=L cls-rpn +L reg-rpn +L cls-rcnn +L reg-rcnn ; Where L represents the overall loss function, L cls-rpn Let L represent the classification loss function of the RPN network. reg-rpn L represents the regression loss function of the RPN network. cls-rcnn Let L represent the classification loss function of the R-CNN network. reg-rcnn This represents the regression loss function of the R-CNN network.

Images