Pedestrian detection model and training method and detection method thereof

Abstract

The invention provides a pedestrian detection model which is used for detecting pedestrians in an image, and the model comprises a feature extraction module which is used for carrying out the feature extraction of the image, and obtaining an original feature map; the region generation module is used for carrying out candidate target frame selection on the original feature map by adopting an anchor frame to obtain a feature map with the anchor frame; the space attention module is used for performing mask processing on the selected part of the anchor frame in the feature map with the anchor frame to obtain a mask feature map; the calibration module is used for performing pixel calibration on the mask feature map based on the feature map with the anchor frame, performing pooling processing on the feature map after pixel calibration, and performing candidate calibration on the original feature map based on the feature map after pooling processing to obtain a calibrated feature map; and the pedestrian recognition module is used for classifying the calibration feature map to recognize the position of the pedestrian. The model provided by the invention can effectively learn the features of the sheltered pedestrians to improve the detection accuracy of the sheltered pedestrians.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a pedestrian detection technology in the field of machine vision, and more particularly to a pedestrian detection model and a training method and a detection method thereof. Background Art

[0002] Pedestrian detection is one of the most important tasks in the field of machine vision. Its purpose is to accurately locate pedestrians in images or videos. Currently, pedestrian detection technology has been widely used in advanced driver assistance, autonomous driving, regional monitoring, human-computer interaction and other fields. In recent years, with the rapid development of deep convolutional neural networks (CNN), machine vision technology has been driven from target recognition to general target detection and then to pedestrian detection.

[0003] At present, the pedestrian detection method based on deep convolutional neural network mainly utilizes the existing deep convolutional neural network framework, trains the deep convolutional neural network through pedestrian datasets, thereby obtaining a trained pedestrian detection model, and processes images or videos with the trained pedestrian detection model to accurately locate pedestrians.

[0004] In the prior art, the R-CNN series of convolutional neural networks are mainly used as the framework to train pedestrian detection models for pedestrian detection. However, during the model training process, the focus is on the features of unobstructed pedestrians, and no effective learning is performed on the features of obstructed pedestrians. As a result, the pedestrian detection model trained under the prior art can detect pedestrians, but can only detect pedestrians who are not obstructed or are not severely obstructed, and the detection accuracy of severely obstructed pedestrians is poor. Summary of the invention

[0005] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a pedestrian detection model, a training method for a pedestrian detection model and a pedestrian detection method.

[0006] The objectives of the present invention are achieved through the following technical solutions.

[0007] According to a first aspect of the present invention, a pedestrian detection model is provided for detecting pedestrians in an image, the model comprising: a feature extraction module for performing feature extraction on an image to obtain an original feature map; a region generation module for performing candidate target selection on the original feature map using an anchor frame to obtain a feature map with an anchor frame; a spatial attention module for performing masking on a portion of the feature map with an anchor frame selected by the anchor frame to obtain a masked feature map; a calibration module for performing pixel calibration on the masked feature map based on the feature map with the anchor frame, and for performing pooling on the feature map after pixel calibration, and for performing candidate calibration on the original feature map based on the feature map after pooling to obtain a calibrated feature map; and a pedestrian recognition module for classifying the calibrated feature map to identify the location of the pedestrian.

[0008] In some embodiments of the present invention, the region generation module is configured to: after selecting candidate targets from the original feature map using an anchor frame, calculate the probability of each anchor frame being extracted in the original feature map according to a preset extraction rule, and sort the probability of each anchor frame being extracted in order from large to small, then select a preset number of anchor frames in the front order and retain them.

[0009] In some embodiments of the present invention, the preset extraction rule is:

[0010]

[0011]

[0012] Among them, P j represents the probability of extracting the jth positive candidate anchor box in the original feature map, λ j represents the occlusion factor of the jth positive candidate anchor box in the original feature map, n p represents the preset value, N represents the total number of positive candidate anchor boxes in the original feature map, represents the occlusion rate of the jth positive candidate anchor box in the original feature map, It represents the summed mean of the occlusion rates of all positive candidate anchor boxes in the original feature map, and k represents the classification threshold.

[0013] In some embodiments of the present invention, the spatial attention module includes a first convolutional layer, a first activation function, a second convolutional layer, a second activation function, a third convolutional layer and a normalization layer connected in sequence.

[0014] In some embodiments of the present invention, the calibration module includes: a pixel calibration submodule, used to perform feature fusion on the feature map with the anchor frame and the mask feature map to obtain a fused feature map; a region of interest pooling submodule, used to perform pooling on the fused feature map to obtain a candidate feature map; and a candidate calibration submodule, used to select a calibration area from the original feature map based on the candidate feature map to obtain a calibration feature map.

[0015] In some embodiments of the present invention, the pixel calibration submodule includes: an element multiplication unit, used to perform element-by-element multiplication of the feature map with the anchor frame and the mask feature map; an element addition unit, used to perform element-by-element addition of the feature map output by the element multiplication unit and the feature map with the anchor frame.

[0016] In some embodiments of the present invention, the region of interest pooling submodule adopts a RoI Align layer or a RoIpooling layer.

[0017] In some embodiments of the present invention, the candidate calibration submodule is configured to: take the candidate feature map as a reference, select an area corresponding to the candidate feature map from the original feature map, and determine the size of the first area calibration frame based on the area, and use the area as the area where the first area calibration frame is located; reduce the first area calibration frame according to a preset scale factor to obtain a second area calibration frame, and select the area with the largest total pixel value in the area where the first area calibration frame is located from the original feature map based on the second area calibration frame; expand the first area calibration frame according to a preset scale factor to obtain a third area calibration frame, and select an area including both the first area calibration frame and the second area calibration frame from the original feature map based on the third area calibration frame.

[0018] In some embodiments of the present invention, the calibration feature map output by the candidate calibration module is:

[0019]

[0020] in, Represents the calibration feature map; F1 represents the area where the first area calibration box in the original feature map is located; F2 represents the area where the first area calibration box in the original feature map is located, and the regional pixels where the second area calibration box is located in this area are set to 0; F3 represents the area where the third area calibration box in the original feature map is located, and the regional pixels where the first area calibration box is located in this area are set to 0.

[0021] In some embodiments of the present invention, the pedestrian recognition module adopts two fully connected layers or one global average pooling layer.

[0022] According to a second aspect of the present invention, a method for training a pedestrian detection model is provided, the method comprising: S1, obtaining a training set, the training set comprising a plurality of image samples, each image sample being provided with a pedestrian annotation box and its corresponding border annotation and pedestrian visible area annotation; S2, using the training set obtained in step S1 to train the model as described in the first aspect until convergence, and in the training process, back-propagating and updating the model parameters based on a preset loss function, the preset loss function being:

[0023] L total =L1+ηL SSA

[0024]

[0025] Among them, L total represents the total loss function of the pedestrian detection model, L1 is the first loss, L SSA is the loss of the spatial attention module, η is a hyperparameter, M represents the number of anchor boxes in the feature map with anchor boxes, H represents the height of the feature map with anchor boxes, W represents the width of the feature map with anchor boxes, and p m (x,y) represents the true value, Represents the output of the spatial attention module.

[0026] According to a third aspect of the present invention, a pedestrian detection method is provided, the method comprising: T1, obtaining image data to be processed; T2, using the model as described in the first aspect to perform pedestrian detection on the image data to be processed obtained in step T1.

[0027] Compared with the prior art, the advantages of the present invention are: (1) the spatial attention module is set to enable the pedestrian detection model to focus on the visible area information (features) of the pedestrian and the parts with higher recognition, so that the pedestrian detection model can better detect obscured pedestrians; (2) the setting of the calibration module can enhance the visible area information (features) of the pedestrian, and further improve the accuracy of the pedestrian detection model in detecting pedestrians. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0029] Figure 1 It is a schematic diagram of the Faster-RCNN structure according to the prior art;

[0030] Figure 2 is a schematic diagram of a pedestrian detection model according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the structure of a pedestrian detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] As described in the background technology, in the prior art, the pedestrian detection model focuses on the features of unobstructed pedestrians during training, and does not effectively learn the features of obstructed pedestrians, so that although the trained pedestrian detection model can realize pedestrian detection, it can only detect pedestrians who are not obstructed or not severely obstructed, and the detection accuracy of severely obstructed pedestrians is poor. In view of this, the present invention proposes a new pedestrian detection scheme, which improves the detection accuracy of the pedestrian detection model for obstructed pedestrians by effectively learning the features of obstructed pedestrians.

[0034] In order to better understand the present invention, the following is a detailed description from the perspectives of the pedestrian detection model in the prior art, the pedestrian detection model solution of the present invention, and experimental verification.

[0035] 1. Pedestrian Detection Model in Existing Technology

[0036] First, the framework structure of the existing pedestrian detection model is introduced. Since there are many frameworks of the existing pedestrian detection model, such as RCNN, Fast-RCNN, Faster-RCNN, etc., for the convenience of understanding, the present invention mainly uses the framework of Faster-RCNN as an example for explanation. Other frameworks are similar to them and will not be introduced repeatedly. Figure 1 As shown, Faster-RCNN includes a feature extraction network, a feature pyramid network (FPN), a region proposal network (RPN), a RoI Align layer and two fully connected layers (FC) connected in sequence. Among them, the feature extraction network adopts the ResNet-50 structure, which includes a head, a first processing layer, a second processing layer, a third processing layer and a fourth processing layer connected in sequence. The first processing layer includes 3 convolution blocks, the second processing layer includes 4 convolution blocks, the third processing layer includes 6 convolution blocks, and the fourth processing layer includes 3 convolution blocks, and each convolution block includes 3 groups of repeated convolution layers, BN layers and activation functions ReLu. Since the feature pyramid network, the region proposal network, the RoI Align layer and the fully connected layer are all network structures known to those skilled in the art, they will not be described in detail here.

[0037] Then, the working principle of the existing pedestrian detection model is introduced. When the pedestrian detection model is used for pedestrian detection, its basic process is as follows: the feature extraction network extracts pedestrian features from the input image; the feature pyramid network performs fusion processing on the feature map output by the feature extraction network; the region generation network then processes the feature map output by the feature pyramid network to obtain a feature map with an anchor frame, that is, multiple anchor frames are marked on the feature map and several anchor frames are randomly selected to be retained, and each anchor frame is scored to distinguish the background from the object through the score; the RoI Align layer processes the feature map with the anchor frame to obtain a fixed-size candidate feature map (proposal feature map); the fully connected layer performs a fully connected operation on the fixed-size candidate feature map, that is, the Softmax function is used to classify the specific category, and the L1 Loss loss function is used to complete the border regression operation to obtain the precise position of the pedestrian.

[0038] Although the existing technology uses the Faster-RCNN framework as a pedestrian detection model to achieve pedestrian detection and positioning, in the existing pedestrian detection model, the region generation network uses a random extraction method to retain the anchor frame, resulting in a failure to focus on the features of occluded pedestrians. There is no solution for targeted learning of the features of severely occluded pedestrians. As a result, when the Faster-RCNN basic framework is used as a pedestrian detection model, only pedestrians who are not occluded or are not severely occluded can be detected, and the detection accuracy of severely occluded pedestrians is poor.

[0039] 2. Pedestrian Detection Model Solution of the Present Invention

[0040] In order to improve the accuracy of severely obscured pedestrian detection, the inventor proposes a pedestrian detection scheme that focuses on learning the features of severely obscured pedestrians. In the embodiment of the present invention, Faster-RCNN is used as the basic framework to illustrate the scheme of the present invention, but it should be noted that the scheme of the present invention does not mean that only Faster-RCNN can be used as the basic framework. Other structures, such as the RCNN series, can also be used as the basic framework to implement the scheme of the present invention. However, due to the similar principles, the schemes of other architectures are not repeated in the embodiments of the present invention.

[0041] In the solution of the present invention, the inventor proposes to add a spatial attention module and a calibration module to the pedestrian detection model to improve the detection accuracy of the pedestrian detection model for occluded pedestrians. Among them, the spatial attention module is used to mask the feature map with the anchor frame, that is, to extract the pixels belonging to the pedestrian in the anchor frame to focus on the visible area information of the pedestrian, so that the pedestrian detection model can better detect the occluded pedestrian; the calibration module is used to calibrate the feature map processed by the spatial attention module to enhance the pedestrian visible area information (features), and further improve the accuracy of the pedestrian detection model in detecting pedestrians.

[0042] According to one embodiment of the present invention, as shown in the attached Figure 2 As shown, a pedestrian detection model is provided, which includes: a feature extraction module, which is used to extract features from an image to obtain an original feature map; a region generation module, which is used to use an anchor frame to select candidate targets from the original feature map to obtain a feature map with an anchor frame; a spatial attention module, which is used to mask the portion of the feature map with an anchor frame selected by the anchor frame to obtain a mask feature map; a calibration module, which is used to perform pixel calibration on the mask feature map based on the feature map with the anchor frame, and to perform pooling on the feature map after pixel calibration, and to perform candidate calibration on the original feature map based on the feature map after pooling to obtain a calibrated feature map; a pedestrian recognition module, which is used to classify the calibrated feature map output by the calibration module to identify the location of the pedestrian.

[0043] The functions of each module of the pedestrian detection model in the solution of the present invention and the training method of the pedestrian detection model in the solution of the present invention are introduced below.

[0044] 2.1 Introduction to the functions of each module of the pedestrian detection model

[0045] 2.11 Feature Extraction Module

[0046] The feature extraction module is used to perform feature extraction on the input image in order to extract useful pedestrian features from the image.

[0047] As described in the first part above, the feature extraction module includes a ResNet-50 network and a feature pyramid network connected in sequence, wherein the ResNet-50 network is used to extract features from the input image, and the feature pyramid network is used to perform feature fusion on the feature map output by the ResNet-50 network. The ResNet-50 network extracts features from the image and can generate feature maps of four different scales, ensuring that the pedestrian detection model can detect both large targets and small targets effectively; at the same time, the feature pyramid network fuses the feature maps of four different feature scales generated by the ResNet-50 network to solve the problem of multi-scale changes in data.

[0048] 2.12 Area Generation Module

[0049] After the feature extraction module obtains the original feature map, the region generation module selects candidate targets on the original feature map to locate targets that may be pedestrians.

[0050] In the prior art, the region generation network is used to generate a large number of anchor boxes based on the feature map input to it, and use the anchor boxes to select candidate targets on the feature map to form a candidate set (the candidate set includes all candidate targets selected by the positive candidate anchor boxes), and randomly extract several candidate targets selected by the anchor boxes from the candidate set for output. It should be noted that the region generation network in the prior art selects anchor boxes by random extraction, that is, the probability of each anchor box being extracted is the same, and the extraction probability of the anchor box is N represents the total number of positive candidate anchor boxes (positive candidate anchor boxes: the anchor boxes whose anchor point overlap area with the reference box (ground truth) is greater than the threshold 0.5 are set as positive candidate anchor boxes), l p Represents the number of positive candidate anchor boxes that the region generation network needs to extract.

[0051] In the random extraction method, since the probability of each candidate anchor frame being extracted is the same, if the candidate targets selected by the anchor frame extracted by the region generation network are all slightly occluded or unoccluded pedestrians, the pedestrian detection performance of the pedestrian detection model in the severe occlusion mode will be greatly reduced. In order to reduce the negative impact of this random extraction method, in the solution of the present invention, a non-random anchor frame extraction rule based on the occlusion rate is proposed, that is, the extraction probability of the anchor frame is redistributed according to the occlusion rate, so that the candidate targets selected by the anchor frame extracted by the region generation module are severely occluded or more occluded pedestrians as much as possible, so that the pedestrian detection model pays more attention to the occluded pedestrians. Based on this, according to an embodiment of the present invention, the region generation module is configured to: after selecting the candidate targets for the original feature map using the anchor frame, calculate the probability of each anchor frame being extracted in the original feature map according to the preset extraction rule, and sort the probability of each anchor frame being extracted in order from large to small, and then select the preset number of anchor frames ranked in front and retain them. Through the non-random anchor frame extraction rule (preset extraction rule) based on pedestrian occlusion rate proposed in the present invention, the region generation module can obtain more feature information of occluded pedestrians, which is beneficial for the pedestrian detection model to identify occluded pedestrians and improve the detection accuracy of the pedestrian detection model for occluded pedestrians.

[0052] According to one embodiment of the present invention, the preset extraction rule is:

[0053]

[0054]

[0055] Among them, P j represents the probability of extracting the jth positive candidate anchor box in the original feature map, λ j represents the occlusion factor of the jth positive candidate anchor box in the original feature map, n p represents the preset value, N represents the total number of positive candidate anchor boxes in the original feature map, represents the occlusion rate of the jth positive candidate anchor box in the original feature map, represents the summed mean of the occlusion rates of all positive candidate anchor boxes in the original feature map, and k represents the classification threshold. The occlusion rate represents the proportion of effective pixels of pedestrians in the candidate targets selected by the anchor box; the positive candidate anchor box is the anchor box whose anchor point overlap area with the reference box (ground truth) is greater than the threshold 0.5. According to an example of the present invention, the preset number is 1000. It should be noted that the preset extraction rule divides the feature map into difficult and simple classes according to the occlusion rate ( Those higher than k are difficult. Less than or equal to k is the simple class). Assuming k = 0.35, for the difficult class (r occ >0.35), the occlusion factor of the positive candidate anchor box with an occlusion rate less than or equal to 0.35 in the feature map A negative value or 0 indicates that the positive candidate anchor box with a lighter degree of occlusion in the difficult class is difficult to be selected; the occlusion factor λ of the positive candidate anchor box with an occlusion rate higher than 0.35 in the feature map j A positive value indicates that the positive candidate anchor box with a higher degree of occlusion in the difficult class is assigned a higher extraction probability; The occlusion factor λ of the positive candidate anchor box with higher occlusion rate in the feature map j (λ j =1+r j occ ), indicating that the positive candidate anchor boxes with high occlusion degree in the simple class are assigned a higher extraction probability. The non-random anchor box extraction rule proposed in the present invention enables the region generation module to obtain more feature information of the occluded pedestrians, which is beneficial for the pedestrian detection model to identify the occluded pedestrians. It should be noted that the classification threshold k is determined according to actual needs, and the present invention does not impose any special restrictions on it.

[0056] 2.13 Spatial Attention Module

[0057] The original feature map output by the feature extraction module is processed by the region generation module to obtain a feature map with an anchor frame. The spatial attention module (Supervised Spatial Attention, SSA) performs mask processing on the part of the feature map with an anchor frame selected by the anchor frame to obtain a mask feature map, so as to focus on the visible area information (features) of the pedestrian and the parts with high recognition, so as to better detect the obscured pedestrians. It should be noted that masking the part of the feature map with an anchor frame selected by the anchor frame means setting the pedestrian pixels in the part of the feature map with an anchor frame selected by 1 and setting the background pixels in the part of the feature map with an anchor frame selected by 0. After masking, the part of the feature map selected by the anchor frame is binarized and the pedestrian pixels therein are highlighted.

[0058] According to one embodiment of the present invention, the spatial attention module includes a first convolution layer, a first activation function, a second convolution layer, a second activation function, a third convolution layer and a normalization layer connected in sequence. According to one embodiment of the present invention, the convolution kernel size of the first convolution layer and the second convolution layer is 3×3×C (C is the number of channels); the first activation function and the second activation function are both ReLu functions; the convolution kernel size of the third convolution layer is 1×1×1; the normalization layer uses a softmax function. For a better understanding, the settings of the convolution layer, activation function, and normalization layer in the embodiment are taken as an example to illustrate how the spatial attention module processes the feature map with the anchor box output by the region generation module: first, the first convolution layer (3×3×C), the first activation function (ReLu function), the second convolution layer (3×3×C), and the second activation function (ReLu function) extract features from the feature map with the anchor box, and then the third convolution layer (1×1×1) and the normalization layer (softmax function) process the feature map with the anchor box after feature extraction to obtain a mask feature map; if the feature map with the anchor box is F, the height is H′, the width is W′, and the number of channels is C, the spatial attention module processes the feature map F with the anchor box to obtain a mask feature map At this time, the mask feature map The height remains H′, the width remains W′, and the number of channels is reduced to 1.

[0059] 2.14 Calibration Module

[0060] After the spatial attention module masks the part selected by the anchor box in the feature map with the anchor box to obtain the mask feature map, the calibration module calibrates the mask feature map to enhance the information (features) of the visible area of ​​the pedestrian.

[0061] According to an embodiment of the present invention, the calibration module includes a pixel calibration submodule, a region of interest pooling submodule and a candidate calibration submodule. The pixel calibration submodule is used to perform feature fusion on the feature map with the anchor frame and the mask feature map to obtain a fused feature map; the region of interest pooling submodule is used to perform pooling on the fused feature map to obtain a candidate feature map; the candidate calibration submodule is used to select a calibration region from the original feature map based on the candidate feature map to obtain a calibration feature map. The following describes each submodule in the calibration module.

[0062] According to an embodiment of the present invention, the pixel calibration submodule (Pixel Calibration, PiC) includes: an element product unit, which is used to perform element-by-element multiplication on the feature map with the anchor frame and the mask feature map; an element addition unit, which is used to perform element-by-element addition on the feature map output by the element product unit and the feature map with the anchor frame. If the output processed by the pixel calibration submodule is defined as The pixel calibration operation can be expressed as:

[0063] i=1,2,…C

[0064] in, represents the fused feature map obtained after pixel calibration, represents element-wise product, and “+” represents element-wise addition. represents the mask feature map output by the spatial attention module, F represents the feature map with anchor box output by the region generation module, and C represents the number of channels. During the pixel calibration process, the element-by-element product operation converts the mask feature map The reflected effective features are transferred to each feature channel of the feature map with anchor frame, so that the focus of feature mapping is placed on the visible area information of pedestrians and the parts with higher recognition; further, due to the complexity of the actual scene, the mask feature map sometimes cannot accurately describe the visible area information of pedestrians. Therefore, the element-by-element addition operation is used after the element-by-element product operation to retain the original feature information, so that the occluded parts have a relatively high confidence.

[0065] According to one embodiment of the present invention, the region of interest pooling submodule adopts a RoI Align layer or a RoI pooling layer.

[0066] According to one embodiment of the present invention, the candidate calibration submodule is configured as follows: based on the candidate feature map, select an area corresponding to the candidate feature map from the original feature map, and determine the size of the first area calibration frame based on the area, and use the area as the area where the first area calibration frame is located; reduce the first area calibration frame according to a preset scale factor to obtain a second area calibration frame, and use the second area calibration frame as a reference to select the area with the largest total pixel count in the area where the first area calibration frame is located from the original feature map; expand the first area calibration frame according to a preset scale factor to obtain a third area calibration frame, and use the third area calibration frame as a reference to select an area including both the first area calibration frame and the second area calibration frame from the original feature map. In order to better understand the process of candidate calibration performed by the calibration submodule, based on the candidate feature map The process of candidate calibration is explained by taking the height as H″ and the width as W″ as an example: First, based on the candidate feature map The height and width of the first area calibration frame P1 are determined, P1∈[H″×W″]; then, the first area calibration frame P is reduced according to a preset scale factor s to obtain a second area calibration frame And taking the second area calibration frame P2 as a reference, select the area with the largest total pixel in the area where the first area calibration frame P is located from the feature map output by the feature extraction module; finally, expand the first area calibration frame P according to the preset scale factor s to obtain the third area calibration frame P3∈[sH″×sW″], and taking the third area calibration frame P3 as a reference, select the area including both the first area calibration frame P1 and the second area calibration frame P2 from the feature map output by the feature extraction module. At this time, the relationship between the area where the first area calibration frame is located, the area where the second area calibration frame is located, and the area where the third area calibration frame is located can be simply expressed as: It should be noted that the preset scale factor s is continuously adjusted according to actual needs, and its initial value can be set to 1.3. After the first area calibration frame P1, the second area calibration frame P2 and the third area calibration frame P3 are determined through candidate calibration, the calibration feature map is calculated based on them:

[0067]

[0068] in, represents the calibration feature map output by the candidate calibration module; F1 represents the area where the first area calibration frame P1 in the original feature map is located; F2 represents the area where the first area calibration frame P1 in the original feature map is located, and the pixels of the area where the second area calibration frame P2 is located in this area are set to 0; F3 represents the area where the third area calibration frame P3 in the original feature map is located, and the pixels of the area where the first area calibration frame P1 is located in this area are set to 0. The calibration feature map obtained after the candidate calibration process It can retain as much context information as possible so that regional feature information is not discarded, providing more distinguishing information for pedestrian detection.

[0069] 2.15 Pedestrian Recognition Module

[0070] The pedestrian recognition module is used to classify the calibration feature map output by the calibration module to identify the location of the pedestrian. According to an embodiment of the present invention, the pedestrian recognition module uses two fully connected layers or one global average pooling layer.

[0071] According to one embodiment of the present invention, ResNet-50 and Feature Pyramid Network (FPN) are used to form a feature extraction module, the RoI Align layer is used as a region of interest pooling submodule, and the Global Average Pooling (GAP) is used as a pedestrian recognition module to obtain the following Figure 3 The schematic diagram of the pedestrian detection model network structure shown in the figure, wherein the pedestrian detection model includes a feature extraction module (ResNet-50 (head, first processing layer, second processing layer, third processing layer and fourth processing layer) and a feature pyramid network (FeaturePyramid Network, FPN)) connected in sequence, a region generation module (Region Proposal Network, RPN), a spatial attention module (Supervised Spatial Attention, SSA), a calibration module (Pixel Calibration submodule (Pixel Calibration, PiC), a region of interest pooling submodule (RoI Align) and a candidate calibration submodule (Proposal Calibration, PrC)) and a pedestrian recognition module. The functions of each module are consistent with those described in the previous embodiment, and will not be repeated here.

[0072] 2.2 Pedestrian Detection Model Training Method

[0073] The above embodiments introduce the framework structure of the pedestrian detection model corresponding to the solution of the present invention and the functions of each module. The following will introduce the training method of the pedestrian detection model in the solution of the present invention.

[0074] According to one embodiment of the present invention, a training method for a pedestrian detection model in the solution of the present invention includes: S1, obtaining a training set, wherein the training set includes a plurality of image samples, each image sample is provided with a pedestrian annotation box and its corresponding border annotation and pedestrian visible area annotation; S2, using the training set obtained in step S1 to train the pedestrian detection model described in the above embodiment until convergence, and in the training process, back-propagating and updating the model parameters based on a preset loss function, wherein the preset loss function is:

[0075] L total =L1+ηL SSA

[0076]

[0077] Among them, L total represents the total loss function of the pedestrian detection model, L1 is the first loss, L SSA is the loss of the spatial attention module, η is a hyperparameter, M represents the number of anchor boxes in the feature map with anchor boxes, H represents the height of the feature map with anchor boxes, W represents the width of the feature map with anchor boxes, and p m (x,y) represents the true value, represents the output of the spatial attention module. It should be noted that the true value represents the annotation of the pedestrian visible area in the input image. It should also be noted that the first loss L1 is:

[0078] L1=L cls_1 +L cls_2 +L reg_1 +L reg_2

[0079]

[0080]

[0081]

[0082] Among them, L cls_1 represents the first classification loss; L cls_2 Second category loss; L reg_1 represents the first regression loss; L reg_2 represents the second regression loss; p i represents the probability of being predicted as a pedestrian; represents the true value label, N cls represents the number of positive candidate anchor boxes in the feature map; λ represents a hyperparameter and λ=10; N reg is the size of the feature map, t i ={t x ,t y ,tw ,t h} represents the 4 parameterized coordinates of the predicted bounding box; is with t i A vector of the same dimension represents the coordinates of the real bounding box corresponding to the predicted bounding box. In the solution of the present invention, a spatial attention module is introduced into the pedestrian detection model, and the loss of the spatial attention module is introduced in the training process to train the pedestrian detection model, so that the model can pay more attention to the features of the occluded pedestrians and learn to improve the detection accuracy of the occluded pedestrians.

[0083] The trained pedestrian detection model of the present invention is directly used for pedestrian detection in an image, so that a more accurate pedestrian detection result can be obtained. It should be noted that when performing pedestrian detection on an image, the image to be detected is generally preprocessed, including but not limited to scaling the image data to the same size using the nearest point interpolation algorithm, the bilinear interpolation algorithm or the bicubic interpolation method, and then performing data enhancement processing on the image data, i.e. adjusting the image saturation, mirroring and noise processing.

[0084] 3. Verification experiment

[0085] 3.1 Dataset Selection

[0086] In order to verify the effectiveness of the pedestrian detection model proposed in this paper, two commonly used datasets, Caltech and CityPersons, are used to evaluate it. Among them, the scenes of CityPersons are recorded by vehicle-mounted cameras in many cities. The CityPersons dataset includes 5,000 pictures, each with a size of 2048×1024. The average number of pedestrians in each picture is 7, and each picture is set with pedestrian visible area and full body annotations. 2,975 pictures are randomly selected from the dataset for training, 500 pictures for verification, and 1,575 pictures for testing. Caltech is similar to CityPersons in terms of data collection, but the location is more single and the number of people is much smaller. The Caltech dataset includes 5,000 images, each with a size of 640×480, and each image is set with pedestrian visible area and full body annotations. The dataset randomly divides all image data into two parts. 2,975 images are used for training and the images are evenly distributed into six sequences from S0 to S5. 2,025 images are used for testing and the images are evenly distributed into five sequences from S6 to S10.

[0087] 3.2 Setting evaluation indicators

[0088] Evaluation indicators The pedestrian detection performance of the pedestrian detection model is evaluated using the two most commonly used standard sets, Reasonable and Heavy. Among them, the visible area of ​​the pedestrian in the Reasonable set exceeds 65% of the full body frame; the occluded area of ​​the pedestrian in the Heavy set exceeds 35% of the full body frame. It should be noted that the height of the pedestrians in the two standard sets is at least 50 pixels. In measuring the detection ability of the pedestrian detection model for pedestrians, the miss detection rate (MR) of the false alarm rate per picture (FPPI) is used as an evaluation indicator, that is, the standard log-average miss rate over the FPPI range of [10 -2 ,10 0 ](In [10 -2 ,10 0 ]The logarithmic mean leakage rate within the FPPI range, namely MR -2 , as an evaluation indicator.

[0089] 3.3 Verification experiment and result analysis

[0090] Faster R-CNN is used as the standard pedestrian detection model (see Appendix Figure 1As shown in the figure, different modules are added to the standard pedestrian detection model in turn, namely, the spatial attention module SSA, the feature calibration module FC (due to the RoI Align layer already existing in the FasterR-CNN network, only the pixel calibration submodule (Pixel Calibration, PiC) and the candidate calibration submodule (Proposal Calibration, PrC) are added at this time), and the difficult example mining module HEMOR (difficult example mining is set in the region generation module (RPN), which is a preset extraction rule), and the performance of the pedestrian detection model containing different modules on the CityPersons dataset and the Caltech dataset is observed. The experimental results are shown in Table 1. It should also be noted that during the verification experiment, the image scale in the CityPersons dataset does not change; the image scale in the Caltech dataset is transformed to 1.3 times the original. As shown in Table 1, on the two standard sets of CityPersons, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model is 15.7%; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model is 56.3%; after adding the spatial attention module SSA to the standard pedestrian detection model, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model + SSA is 13.3%, which is 2.4% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model + SSA is 52.0%, which is 4.3% lower than the miss rate of the standard pedestrian detector; after further adding the calibration module FC, for the Reasonable set, the standard pedestrian detection model The logarithmic average miss rate of +SSA+FC is 12.1%, which is 3.6% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model +SSA+FC is 47.6%, which is 8.7% lower than the miss rate of the standard pedestrian detector; finally, after adding the difficult example mining module HEMOR, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model +SSA+FC+HEMOR is 11.9%, which is 3.8% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model +SSA+FC+HEMOR (the pedestrian detection model described in the embodiment of the present invention) is 46.6%, which is 9.7% lower than the miss rate of the standard pedestrian detector.

[0091] On the two standard sets of Caltech, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model is 13.5%; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model is 46.8%; after adding the spatial attention module SSA to the standard pedestrian detection model, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model + SSA is 11.8%, which is 1.7% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model + SSA is 44.9%, which is 1.9% lower than the miss rate of the standard pedestrian detector; after further adding the calibration module FC, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model + SSA + FC is 11.8%. The miss rate is 10.2%, which is 3.3% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model + SSA + FC is 42.8%, which is 4% lower than the miss rate of the standard pedestrian detector; finally, after adding the difficult example mining module HEMOR, for the Reasonable set, the logarithmic average miss rate of the standard pedestrian detection model + SSA + FC + HEMOR (the pedestrian detection model described in the embodiment of the present invention) is 11.9%, which is 3.8% lower than the miss rate of the standard pedestrian detector; for the Heavy set, the logarithmic average miss rate of the standard pedestrian detection model + SSA + FC + HEMOR (the pedestrian detection model described in the embodiment of the present invention) is 42.4%, which is 4.4% lower than the miss rate of the standard pedestrian detector. It can be seen from the contents of Table 1 that the pedestrian detection model described in the embodiment of the present invention achieves the best pedestrian detection performance.

[0092] Table 1

[0093]

[0094] The beneficial effects of the present invention are: (1) the spatial attention module is set to enable the pedestrian detection model to focus on the visible area information (features) of the pedestrian and the parts with higher recognition, so that the pedestrian detection model can better detect obscured pedestrians; (2) the setting of the calibration module can enhance the visible area information (features) of the pedestrian, and further improve the accuracy of the pedestrian detection model in detecting pedestrians.

[0095] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0096] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0097] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0098] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A pedestrian detection model for detecting pedestrians in an image, characterized in that: The model includes: A feature extraction module is used to extract features from an image to obtain an original feature map; A region generation module, used for selecting candidate targets from the original feature map using an anchor frame to obtain a feature map with an anchor frame; A spatial attention module, used for performing mask processing on the portion of the feature map with the anchor frame selected by the anchor frame to obtain a mask feature map; A calibration module, configured to perform pixel calibration on the mask feature map based on the feature map with the anchor frame, and to perform pooling processing on the pixel-calibrated feature map, and to perform candidate calibration on the original feature map based on the feature map after the pooling processing to obtain a calibrated feature map; The pedestrian recognition module is used to classify the calibration feature map to identify the location of the pedestrian.

2. The model according to claim 1, characterized in that The region generation module is configured as follows: After selecting candidate targets from the original feature map using anchor frames, the probability of each anchor frame being extracted in the original feature map is calculated according to a preset extraction rule, and the probability of each anchor frame being extracted is sorted in descending order, and then a preset number of anchor frames at the front are selected and retained.

3. The model according to claim 2, characterized in that The preset extraction rules are: Among them, P j represents the probability of extracting the jth positive candidate anchor box in the original feature map, λ j represents the occlusion factor of the jth positive candidate anchor box in the original feature map, n p represents the preset value, N represents the total number of positive candidate anchor boxes in the original feature map, r j occ represents the occlusion rate of the jth positive candidate anchor box in the original feature map, It represents the summed mean of the occlusion rates of all positive candidate anchor boxes in the original feature map, and k represents the classification threshold.

4. The model according to claim 1, characterized in that The spatial attention module includes a first convolutional layer, a first activation function, a second convolutional layer, a second activation function, a third convolutional layer and a normalization layer which are connected in sequence.

5. The model according to claim 1, characterized in that The calibration module comprises: A pixel calibration submodule, used for performing feature fusion on the feature map with the anchor frame and the mask feature map to obtain a fused feature map; A region of interest pooling submodule, used for performing pooling processing on the fused feature map to obtain a candidate feature map; The candidate calibration submodule is used to select a calibration area from the original feature map based on the candidate feature map to obtain a calibration feature map.

6. The model according to claim 5, characterized in that The pixel calibration submodule comprises: An element product unit, used for performing element-by-element multiplication of the feature map with the anchor frame and the mask feature map; An element addition unit is used to perform element-by-element addition on the feature map output by the element product unit and the feature map with the anchor frame.

7. The model according to claim 6, characterized in that The region of interest pooling submodule adopts a RoIAlign layer or a RoIpooling layer.

8. The model according to claim 7, characterized in that The candidate calibration submodule is configured as follows: Taking the candidate feature map as a reference, selecting an area corresponding to the candidate feature map from the original feature map, determining a size of a first area calibration frame based on the area, and using the area as an area where the first area calibration frame is located; Reducing the first area calibration frame according to a preset scale factor to obtain a second area calibration frame, and selecting an area with the largest total pixel value in the area where the first area calibration frame is located from the original feature map based on the second area calibration frame; The first region calibration frame is enlarged according to a preset scale factor to obtain a third region calibration frame, and a region including both the first region calibration frame and the second region calibration frame is selected from the original feature map based on the third region calibration frame.

9. The model according to claim 8, characterized in that The calibration feature map output by the candidate calibration module is: in, Represents the calibration feature map; F1 represents the area where the first area calibration box in the original feature map is located; F2 represents the area where the first area calibration box in the original feature map is located, and the regional pixels where the second area calibration box is located in this area are set to 0; F3 represents the area where the third area calibration box in the original feature map is located, and the regional pixels where the first area calibration box is located in this area are set to 0.

10. The model according to claim 1, characterized in that The pedestrian recognition module adopts two fully connected layers or one global average pooling layer.

11. A method for training a pedestrian detection model, characterized in that: The method comprises: S1. Obtain a training set, wherein the training set includes a plurality of image samples, each of which is provided with a pedestrian annotation box and its corresponding border annotation and pedestrian visible area annotation; S2. Using the training set obtained in step S1, the model according to any one of claims 1 to 10 is trained until convergence. During the training process, the model parameters are updated by back propagation based on a preset loss function, wherein the preset loss function is: L total =L1+ηL ssA Among them, L total represents the total loss function of the pedestrian detection model, L1 is the first loss, L SSA is the loss of the spatial attention module, η is a hyperparameter, M represents the number of anchor boxes in the feature map with anchor boxes, H represents the height of the feature map with anchor boxes, W represents the width of the feature map with anchor boxes, and p m (x,y) represents the true value, Represents the output of the spatial attention module.

12. A pedestrian detection method, characterized in that: The method comprises: T1. Obtain image data to be processed; T2. Use the model described in any one of claims 1 to 10 to perform pedestrian detection on the image data to be processed obtained in step T1.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 11-12.

14. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method as described in any one of claims 11-12.

Images