Dense crowded pedestrian detection method and system based on target detection model

Abstract

The invention discloses a dense crowded pedestrian detection method and system based on a target detection model, and relates to the technical field of image detection, and the method comprises the steps: inputting a to-be-detected crowded crowd image into a crowded pedestrian detection model; the crowded pedestrian detection model comprises a backbone network used for extracting a multi-scale feature map of a crowded crowd image; the feature enhancement and refinement module is used for carrying out refinement and enhancement on low-layer features in the multi-scale feature map to obtain an enhanced multi-scale feature map; the hybrid encoder is used for encoding the enhanced multi-scale feature map and transmitting the encoded multi-scale feature map to each layer of the decoder; all layers of the decoder are connected in sequence, the last layer of the decoder is replaced by a prediction suppression module, and the prediction suppression module is used for screening out clear prediction and non-clear prediction and suppressing the non-clear prediction so as to output a prediction result of a target pedestrian; according to the densely-crowded pedestrian detection method, the detection capability of densely-crowded crowds is improved, and the detection speed is high.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a densely crowded pedestrian detection method and system based on a target detection model. Background Art

[0002] Pedestrian detection is a classic sub-problem in the field of object detection in computer vision. It is widely used in video surveillance, autonomous driving, robotics and other fields. It also serves as the cornerstone of many downstream tasks such as pedestrian re-identification, tracking, trajectory prediction, etc. At present, pedestrian detection algorithms based on deep learning have made great progress. Classic models such as object detection models based on convolutional neural networks (CNN) (such as Faster R-CNN, YOLO, FCOS) have achieved remarkable success. With the emergence of Transformer-based object detection models (such as DETR, DeformableDETR, RT-DETR), pedestrian detection has been brought to a new level and can support end-to-end detection. Faster R-CNN is a deep learning model for object detection, which combines the Region Proposal Network (RPN) and the standard convolutional neural network (CNN) for efficient object detection. YOLO (You Only Look Once) is a deep learning model for real-time object detection. FCOS (Fully Convolutional One-Stage) is a fully convolutional single-stage anchor-free detection model. DETR (Detection Transformer) is an end-to-end object detection model based on Transformer. Deformable DETR is a Transformer-based end-to-end object detection model. DETR is an end-to-end object detection model based on Transformer, and RT-DETR is a real-time end-to-end target detection algorithm based on Transformer architecture.

[0003] However, pedestrian detection still has some difficult problems to be solved due to the complexity of the actual situation. In particular, in scenes with dense pedestrians or pedestrians blocked by objects, the detection of blocked pedestrians is often not so effective. In addition, there are environmental influences such as lighting and weather, and there are relatively few studies on this aspect, which makes the detection of dense pedestrians more challenging. Existing technologies such as the YOLO series, although they have high detection capabilities and detection speeds, have weak ability to distinguish and process similar features and features of blocked pedestrians, resulting in poor results in the detection of densely blocked pedestrians. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a densely crowded pedestrian detection method and system based on a target detection model, which improves the detection capability of densely crowded people and has a higher detection speed.

[0005] The densely crowded pedestrian detection method based on the target detection model proposed in the present invention includes: Inputting a dense crowd image to be detected into a crowded pedestrian detection model, which includes a backbone network, a feature enhancement and refinement module, a hybrid encoder and a decoder connected in sequence; The backbone network is used to extract multi-scale feature maps of dense and crowded crowd images; The feature enhancement and refinement module is used to refine and enhance the low-level features in the multi-scale feature map to obtain an enhanced multi-scale feature map; The hybrid encoder is used to encode the enhanced multi-scale feature map and transmit it to each layer of the decoder; The layers of the decoder are connected in sequence, and the last layer of the decoder is replaced by a prediction suppression module. The prediction suppression module is used to filter out explicit predictions and implicit predictions, and suppress implicit predictions to output the prediction results of the target pedestrian.

[0006] Furthermore, the feature enhancement and refinement module includes a channel attention block, a spatial attention block, and a depth separation convolution block; The low-level features in the multi-scale feature map pass through the channel attention block, spatial attention block and depth separation convolution block in sequence to obtain the enhanced multi-scale feature map.

[0007] Further, the channel attention block includes a first channel module, a second channel module, a first activation layer and a first product layer; The inputs of the first channel module and the second channel module are both low-level features in the multi-scale feature map; The input of the first activation layer is connected to the output of the first channel module and the output of the second channel module respectively to evaluate the channel attention weight; The input of the first product layer is connected to the output of the first activation layer and the output of the backbone network respectively to superimpose the channel attention weights on the low-level features in the multi-scale feature map. The first product layer outputs the channel-enhanced feature map.

[0008] Further, the spatial attention block includes a channel maximum pooling layer, a channel average pooling layer, a first convolutional layer, a second activation layer, and a second product layer; The channel maximum pooling layer and the channel average pooling layer are connected in parallel, and the input is connected to the output of the first product layer, and the output is connected to the input of the first convolutional layer to obtain local feature information; The input of the second activation layer is connected to the output of the first convolutional layer, and the output is connected to the second product layer to evaluate the spatial attention weight; The input of the second product layer is connected to the output of the second activation layer and the output of the first product layer respectively to superimpose the spatial attention weights on the channel-enhanced feature map, and the second product layer outputs the spatially enhanced feature map.

[0009] Further, the prediction suppression module includes a prediction filter, a relation information extractor, a local self-attention block, and a cross-attention block; The prediction filter receives the output from the decoder and filters out explicit predictions and non-explicit predictions based on the confidence of the predictions; The relation information extractor is used to extract the spatial feature relationship between the non-explicit prediction and the adjacent explicit prediction, and acts on the non-explicit prediction; The local self-attention block is used to self-update the non-explicit prediction by paying local attention to the neighboring predictions around the non-explicit prediction; The criss-cross attention block is used to learn updates that are not explicitly predicted from the features output by the decoder.

[0010] Furthermore, in the prediction filter, the output of the decoder is predicted by the confidence detection head to obtain the confidence, and the predictions with confidence > 0.7 constitute the explicit prediction set, and the predictions with confidence ≤ 0.7 constitute the implicit prediction set.

[0011] Furthermore, the implicit prediction learns the neighboring features through the relation information extractor, specifically: Based on the spatial position information between predictions, Unspecified predictions Bounding box prediction Bounding box predictions for neighboring explicit predictions ; Through the sine-cosine spatial position coding and Relationship modeling, obtaining location relationship information; The position relationship information is sequentially passed through the linear layer and the maximum pooling layer, and then element-by-element is superimposed on the non-explicit prediction. On the features after the linear layer, the superimposed features are output through the linear layer to obtain the implicit prediction of the learned neighboring features.

[0012] Furthermore, the final prediction output process of the target pedestrian is as follows: The non-explicit predictions and the additionally introduced predictions constitute a supplemented non-explicit prediction set; The local self-attention block is a multi-head self-attention module with an attention mask added, and local attention is performed on the supplemented non-explicit prediction set based on the attention mask; The masked features are input into the deformable attention module to perform reinforcement learning on non-explicit predictions. After reinforcement learning, the supplementary predictions are discarded and merged with the explicit predictions as the final prediction output of the target pedestrian.

[0013] Furthermore, the training process of the crowded pedestrian detection model is as follows: Obtain a public dataset of crowded crowd images as a training set for the crowded pedestrian detection model; Input the training set into the crowded pedestrian detection model; In the real box allocation, the prediction boxes that are clearly predicted are allocated first, and then the remaining prediction boxes are allocated; A hybrid matching strategy is introduced for the first few layers of the decoder. The hybrid matching strategy is as follows: parallel queries are introduced for the first few layers of the decoder, and the introduced parallel queries are isolated from other queries through an attention mask mechanism to form parallel training branches. Based on the matching strategy of matching each real box to multiple predictions, losses are constructed for the parallel training branches to train the crowded pedestrian detection model.

[0014] A densely crowded pedestrian detection system based on a target detection model inputs a densely crowded crowd image to be detected into a crowded pedestrian detection model; The crowded pedestrian detection model includes a backbone network, a feature enhancement and refinement module, a hybrid encoder, and a decoder connected in sequence; The backbone network is used to extract multi-scale feature maps of dense and crowded crowd images; The feature enhancement and refinement module is used to refine and enhance the low-level features in the multi-scale feature map to obtain an enhanced multi-scale feature map; The hybrid encoder is used to encode the enhanced multi-scale feature map and transmit it to each layer of the decoder; The layers of the decoder are connected in sequence, and the last layer of the decoder is replaced by a prediction suppression module. The prediction suppression module is used to filter out explicit predictions and implicit predictions, and suppress implicit predictions to output the prediction results of the target pedestrian.

[0015] The advantages of the densely crowded pedestrian detection method and system based on the target detection model provided by the present invention are: while ensuring the stability of the parameter quantity and the calculation amount, it is possible to better distinguish dense pedestrians from the image and mine the semantic and spatial feature information of the obscured pedestrians, thereby reducing the omission rate of detection; the introduction of the feature refinement module can effectively enhance the multi-scale features; the introduction of the prediction suppression module can effectively suppress the repeated prediction of the same target, thereby enhancing the model's ability to handle densely crowded pedestrian detection; the introduction of priority matching and hybrid matching strategies can effectively accelerate the training convergence speed of the model and improve the prediction accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; Figure 2 for Figure 1 The network structure diagram of the feature enhancement and refinement module; Figure 3 for Figure 1 Schematic diagram of the structure of the prediction inhibition module; Figure 4 for Figure 3 Schematic diagram of the structure of the relation information extractor in ; Figure 5 Schematic diagram of the hybrid matching strategy branch. DETAILED DESCRIPTION

[0017] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0018] like Figures 1 to 5 As shown, the densely crowded pedestrian detection method based on the target detection model proposed in the present invention includes: The image of dense and crowded crowd to be detected is input into the crowded pedestrian detection model; the crowded pedestrian detection model includes a backbone network, a feature enhancement and refinement module, a hybrid encoder and a decoder connected in sequence; the backbone network is used to extract a multi-scale feature map of the dense and crowded crowd image; the feature enhancement and refinement module is used to refine and enhance the low-level features to obtain enhanced low-level features; the hybrid encoder is used to encode the enhanced multi-scale feature map and transmit it to each layer of the decoder; the layers of the decoder are connected in sequence, and the last layer of the decoder is replaced by a prediction suppression module, which is used to screen out explicit predictions and implicit predictions, and suppress implicit predictions to output the prediction results of the target pedestrians.

[0019] The densely crowded pedestrian detection method provided in this embodiment can better distinguish dense pedestrians from images and mine the semantic and spatial feature information of obscured pedestrians while ensuring the stability of parameter quantity and calculation amount, thereby reducing the omission rate of detection.

[0020] The densely crowded pedestrian detection method of this embodiment includes the following aspects: constructing a crowded pedestrian detection model, training the crowded pedestrian detection model based on the constructed data set, and testing the trained crowded pedestrian detection model, so as to obtain a crowded pedestrian detection model for densely crowded pedestrian detection.

[0021] 1. Obtain a public dataset of crowded crowd images as a training set for the crowded pedestrian detection model; In this embodiment, the first step is to obtain a crowded pedestrian dataset. This embodiment uses a crowded crowd image dataset (crowdhuman dataset). The crowded crowd image dataset contains 15,000 training sets, 5,000 test sets, and 4,370 validation sets. Each image contains 23 people on average. There are also various occlusion situations. The crowded crowd image dataset provides annotations such as the pedestrian visible area bounding box and the human body bounding box for each pedestrian instance in the image. Since the dataset is annotated in odgt file format, it needs to be converted into a json file in coco format for universality. And since the resolution of the images in the crowded crowd dataset varies greatly, it needs to be preprocessed to randomly scale, flip, fill, and other operations to enhance the data.

[0022] 2. Crowded Pedestrian Detection Model Based on the RT-DETR framework, a crowded pedestrian detection model (Crowd RT-DETR) for dense and crowded pedestrian detection is constructed. RT-DETR consists of a backbone network, a hybrid encoder, a decoder, and a detection head. Among them, RT-DETR is a real-time end-to-end target detection algorithm based on the DETR architecture open sourced by Baidu. DETR (Detection Transformer) proposes a new end-to-end target detection framework based on Transformer.

[0023] The crowded pedestrian detection model in this embodiment is different from RT-DETR in the following ways: a feature enhancement and refinement module (FRM) is introduced before the hybrid encoder, and the last layer of the decoder is replaced by a prediction suppression module. That is, the crowded pedestrian detection model includes a backbone network, a feature enhancement and refinement module, a hybrid encoder, and a decoder connected in sequence, wherein the backbone network in the crowded pedestrian detection model has the same structure as the backbone network in RT-DETR, and the hybrid encoder in the crowded pedestrian detection model has the same structure as the hybrid encoder in RT-DETR.

[0024] (1) Introducing the feature enhancement refinement module FRM before the hybrid encoder; Combination Figure 1 and 2 To illustrate, the feature enhancement and refinement module acts between the backbone network and the hybrid encoder. As the multi-scale feature map contains more detailed features, the feature enhancement and refinement module is used to Perform refinement enhancement to obtain low-level features in the enhanced multi-scale feature map That is, the feature enhancement and refinement module includes a channel attention block, a spatial attention block, and a depth separation convolution block; the low-level features pass through the channel attention block, the spatial attention block, and the depth separation convolution block in sequence to obtain an enhanced multi-scale feature map.

[0025] like Figure 2 As shown, the channel attention block includes a first channel module, a second channel module, a first activation layer and a first product layer; the inputs of the first channel module and the second channel module are both low-level features in the multi-scale feature map; the input of the first activation layer is connected to the output of the first channel module and the output of the second channel module, respectively, to evaluate the channel attention weight; the input of the first product layer is connected to the output of the first activation layer and the output of the backbone network, respectively, to superimpose the channel attention weight on the low-level features in the multi-scale feature map, and the first product layer outputs the channel-enhanced feature map.

[0026] That is, in the first channel module, the channel attention block (CAB) performs global maximum pooling on the input feature map (low-level features), and first reduces the channel dimension through two 1x1 convolutions, and then restores it to the initial dimension to obtain the first feature map. In the second channel module, the channel attention block (CAB) performs global average pooling on the input feature map (low-level features in the multi-scale feature map), and first reduces the channel dimension through two 1x1 convolutions, and then restores it to the initial dimension to obtain the second feature map. The two parallel feature maps (i.e., the first feature map and the second feature map) are added together, and then the sigmoid of the first activation layer is used to evaluate the channel attention weight. Finally, the Hadamard product of the first product layer is used to superimpose the channel attention weight on the initial input feature map (low-level features in the multi-scale feature map). The first product layer outputs the channel-enhanced feature map.

[0027] like Figure 2 As shown, the spatial attention block includes a channel maximum pooling layer, a channel average pooling layer, a first convolutional layer, a second activation layer and a second product layer; after the channel maximum pooling layer and the channel average pooling layer are parallelized, the input is connected to the output of the first product layer, and the output is connected to the input of the first convolutional layer to obtain local feature information; the input of the second activation layer is connected to the output of the first convolutional layer, and the output is connected to the second product layer to evaluate the spatial attention weight; the input of the second product layer is connected to the output of the second activation layer and the output of the first product layer respectively to superimpose the spatial attention weight on the input feature map, and the second product layer outputs the spatially enhanced feature map.

[0028] The spatial attention block (SAB) uses channel-wise maximum pooling and average pooling in parallel to obtain local feature information for the input feature map (channel-enhanced feature map), and then uses a large kernel convolution (7x7 convolution) to enhance the local contextual relationship between features. Similar to CAB, the sigmoid activation function of the second activation layer is used to evaluate the spatial attention weight. Finally, the Hadamard product of the second product layer is also used to superimpose the spatial attention weight on the initial input feature map (channel-enhanced feature map).

[0029] like Figure 2 As shown in the figure, the deep separable convolution block (DSCB) uses 1x1 convolution on the input feature map (feature map after spatial enhancement) to increase the channel dimension, and then uses deep convolution to extract features. The key point is that since deep convolution ignores the relationship between channels, a channel shuffle operation is used to fuse the feature information between channels, and then another 1x1 convolution is used to restore the channel dimension to the initial dimension. Finally, a residual connection is used to generate the final output, that is, the enhanced multi-scale feature map.

[0030] (2) Replace the last layer of the decoder with a prediction suppression module; The motivation for introducing the prediction suppression module in this embodiment is mainly that the prediction results tend to be stable (the target of interest tends to be stable) as the decoder layers progress. Therefore, if the confidence of the prediction results exceeds a certain threshold, it can be considered that there is a high probability that it is the final prediction. After all, this part of the prediction has almost no possibility of false alarms. Therefore, this part of the prediction can be "locked" in advance and called explicit predictions, while the remaining predictions can be called implicit predictions. Therefore, the prediction suppression module allows those implicit predictions to "sense" that the target they are concerned about has been detected by those explicit predictions, thereby reducing their confidence and being filtered out.

[0031] like Figure 3 As shown, the prediction suppression module includes a prediction filter, a relation information extractor, a local self-attention block and a cross-attention block; the prediction filter receives the output from the decoder and filters out explicit predictions and non-explicit predictions according to the confidence of the prediction; the relation information extractor is used to extract the spatial feature relationship between non-explicit predictions and neighboring explicit predictions, and acts on non-explicit predictions; the local self-attention block is used to update the non-explicit prediction by paying local attention to the neighboring predictions around the non-explicit prediction; the cross-attention block is used to learn the update of the non-explicit prediction from the features output by the decoder.

[0032] (2-1) Prediction filter; The prediction filter accepts the output prediction from the decoder , is the number of predictions output by the decoder, The decoder outputs predictions. The confidence predicted by the confidence detection head is screened, and the predictions with confidence > 0.7 constitute a clear prediction set , predictions with confidence ≤ 0.7 constitute the non-explicit prediction set , and Specifically expressed as: (1); (2); in, It is Predictions The confidence of the detection box obtained by the confidence detection head; for the decoder part of RT-DETR, each layer of the decoder will have a bounding box detection head and a confidence detection head To predict Get bounding box coordinates and confidence predictions, According to the prediction Middle Predictions The corresponding bounding box coordinates are predicted. According to the prediction Middle Predictions Get the confidence level of the prediction.

[0033] (2-2) Relation information extractor; like Figure 4 As shown, first for each non-explicit prediction , according to its spatial location information Finding nearby clear predictions , is a 4-dimensional feature vector, indicating The spatial information of interest is the common learnable spatial information provided by the decoder to instruct the decoder The spatial location of the query is concerned, are the center point coordinates of the focus frame respectively , frame width ,high Therefore, this embodiment selects at most ten nearest neighboring explicit predictions based on the IoU (intersection over union) of the spatial position information between predictions; then the implicit bounding box prediction is encoded by sine and cosine spatial position. and the bounding box predictions of the neighboring explicit predictions Relationship modeling is used to obtain position relationship information. Then, two linear layers are used to learn spatial position relationship features and IoU information. Finally, a maximum pooling operation is performed to superimpose them on the left branch. Two linear layers are used for feature learning; after element-by-element superposition of the results of the left branch, a linear layer is used to further learn the relational features, and an implicit prediction of the learned neighboring features is obtained, which can be expressed as follows: (3); (4); (5); (6); in, is the intersection-and-union ratio, For the Unspecified predictions Spatial location information, For the A clear prediction Spatial location information, is the set of all non-explicitly predicted spatial location information, is the set of all explicitly predicted spatial location information, Indicates the concat operation. represents a fully connected layer + ReLU activation + a fully connected layer, is the sine-cosine spatial position encoding, is the feature of the position relationship information after passing through the linear layer, For non-definite prediction The features after the linear layer are For the obtained implicit predictions of the learned neighboring features, For the maximum pooling process, The threshold for determining proximity (usually set to 0.4).

[0034] (2-3) Local self-attention block; like Figure 3 As shown in Figure 2, the local attention module is actually a multi-head self-attention module plus an attention mask. The attention mask is an important function provided by the multi-head self-attention module. The mask matrix , the mask matrix Line The columns represent the time when calculating attention Will you pay attention? Therefore, we use the attention mask to pay local attention to the implicit predictions. That is, for the implicit predictions, if they do not have neighboring predictions (explicit predictions or implicit predictions), we hope that they will only focus on themselves when doing self-attention (equivalent to direct transfer); if they have neighboring predictions (explicit predictions or implicit predictions), we hope that they can avoid paying attention to those equally implicit and non-neighboring predictions when doing self-attention, because these predictions cannot provide them with useful information and may even interfere with their learning.

[0035] However, it should be noted that the local attention module targets the non-explicit prediction after the relation information extractor, that is, the non-explicit prediction after learning the neighboring features. For the convenience of expression, the following is still based on the non-explicit prediction Abbreviation. Due to the removal of the explicit prediction set , rather than an explicit prediction set The data distribution is obviously different from the prediction Different, so this embodiment introduces a set of supplementary prediction , the supplementary prediction Override explicit prediction set , to keep the prediction The distribution of remains unchanged. The cross attention module uses a deformable attention module, and after the deformable attention module performs reinforcement learning on the non-explicit predictions, the previously introduced supplementary predictions are Eliminate and with explicit prediction The final prediction is expressed as follows: (7); (8); (9); in, represents the deformable attention module, represents the multi-head self-attention module, To supplement the non-explicit prediction set, To non-explicitly predict the features after reinforcement learning, is the final prediction output.

[0036] 3. Training of crowded pedestrian detection model; The training process of the crowded pedestrian detection model is as follows: Step 1 to Step 4: Step 1: Obtain a public dataset of crowded crowd images as a training set for the crowded pedestrian detection model; The construction of the training set is detailed in the above step 1: Obtain a public crowd image dataset as a training set for the crowded pedestrian detection model.

[0037] Step 2: Input the training set into the crowded pedestrian detection model; Step 3: In the real box allocation, the prediction boxes that are clearly predicted are allocated first, and then the remaining prediction boxes are allocated; Step 4: introduce a hybrid matching strategy to the first few layers of the decoder. The hybrid matching strategy is: introduce parallel queries to the first few layers of the decoder, isolate the introduced parallel queries from other queries through the attention mask mechanism, and form parallel training branches. Based on the matching strategy of matching each real box to multiple predictions, a loss is constructed for the parallel training branches to train the crowded pedestrian detection model.

[0038] Among them, a hybrid matching strategy is introduced for the first few layers of the decoder, and the specific number of the first few layers of the decoder can be set according to the actual training situation, for example Figure 5 As shown, this embodiment chooses to introduce a hybrid matching strategy into the first three layers of the encoder.

[0039] The original RT-DETR first uses Hungarian matching when calculating the loss during training, that is, predicting With real box Perform one-to-one matching so that each Match only one , that is, by calculating and The loss matrix is ​​obtained by summing the Focal loss (cross entropy loss function), L1 loss (absolute error loss) and GIoU loss (IoU loss function), and then the loss matrix is ​​used for matching through the Hungarian algorithm; According to the prediction Middle Predictions The corresponding bounding box coordinates are predicted. According to the prediction Middle Predictions The confidence prediction obtained, For the A real frame, For the A real frame, is the number of true boxes, For prediction Middle Predictions The corresponding bounding box prediction.

[0040] Then the prediction Calculate the Varifocal loss (an advanced version of the Focal loss function, used to train dense target detectors) and matched The L1 loss (absolute error loss) and GIoU loss (IoU loss function) are calculated, and finally the loss of a single-layer decoder is obtained. The final loss is obtained by combining the losses of each decoder layer.

[0041] Because the predictions are divided into clear predictions according to their confidence and non-explicit predictions , so we need to ensure that the prediction matches the real box, so we must match it first Then match the remaining predictions with the true boxes The subsequent loss calculation is consistent with the original RT-DETR. Match for Hungary.

[0042] A new matching mode is introduced in the training stage of the crowded pedestrian detection model. The original RT-DETR adopts the Hungarian matching mode when calculating the loss, that is, in the training stage, only one predicted box is matched for a real box. 1) In order to adapt to the prediction suppression module of this embodiment, a priority matching strategy is introduced, that is, when allocating real boxes, it is necessary to give priority to the clearly predicted prediction boxes, and then allocate the remaining prediction boxes; 2) In addition, in order to accelerate the convergence of the crowded pedestrian detection model, the training effect, and the robustness of the model, a mixed matching strategy is adopted for the first few layers of the decoder. This is because the first few decoder layers have uncertainty about their own prediction targets (that is, the previous layer of a query may still predict target A, and the next layer may switch to predict target B). In order to accelerate the speed of prediction entering stability, in the training stage, a parallel prediction branch is introduced for the decoder, which is isolated from the normal training branch by the attention mask, and one-to-many matching is adopted when matching them, that is, one real box can match multiple prediction boxes.

[0043] A mixed matching strategy is introduced for the first few layers of the decoder, that is, training is carried out by mixing one-to-one matching and one-to-many matching: Figure 5 As shown, during the training phase, 300 parallel queries are introduced to the first three layers in the decoder. , through the attention mask mechanism to isolate it from other common queries, forming a parallel training branch, and using one-to-many matching for this part of the branch when calculating the loss, that is, each true box will be matched to multiple predictions, by Copy once to get , which is used to match , so that each real box is realized on this branch A maximum of two predictions can be matched.

[0044] The specific parameters of the crowded pedestrian detection model in this embodiment are: trained on cuda11.6, deep learning framework pytorch2.0.1, and 4 NVIDIAGeForceRTX 3090 GPUs. The input image size of the model training is 960x960, and the training epoch is 50.

[0045] 4. Testing of crowded pedestrian detection model; The trained crowded pedestrian detection model is tested and evaluated on the validation set. The specific content is: use the trained crowded pedestrian detection model to make inference predictions on the validation set, and calculate based on the predicted box data and the real box data , Indicators, using the average precision AP (Averageprecision) and the average of the average precision To evaluate the detection effect and performance of the crowded pedestrian detection model, the larger the index, the better. To evaluate the missed detection of the crowded pedestrian detection model, the smaller the index, the better.

[0046] Compared with the prior art, this embodiment introduces a feature refinement module, which can effectively enhance the multi-scale feature map; introduces a prediction suppression module, which can effectively suppress repeated predictions of the same target, thereby enhancing the model's ability to handle densely crowded pedestrian detection; introduces priority matching and mixed matching strategies, which can effectively accelerate the training convergence speed of the model and improve prediction accuracy and robustness. In summary, the crowded pedestrian detection model of this embodiment can improve the model's ability to detect densely crowded people and has a higher detection speed.

[0047] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A densely crowded pedestrian detection method based on a target detection model, characterized in that: include: Inputting the dense crowd image to be detected into the crowded pedestrian detection model, which includes a backbone network, a feature enhancement and refinement module, a hybrid encoder and a decoder connected in sequence; The backbone network is used to extract multi-scale feature maps of dense and crowded crowd images; The feature enhancement and refinement module is used to refine and enhance the low-level features in the multi-scale feature map to obtain an enhanced multi-scale feature map; The hybrid encoder is used to encode the enhanced multi-scale feature map and transmit it to each layer of the decoder; The layers of the decoder are connected in sequence, and the last layer of the decoder is replaced by a prediction suppression module. The prediction suppression module is used to filter out explicit predictions and implicit predictions, and suppress implicit predictions to output the prediction results of the target pedestrian.

2. The densely crowded pedestrian detection method based on the target detection model according to claim 1 is characterized in that: The feature enhancement and refinement module includes a channel attention block, a spatial attention block and a depth separation convolution block; The low-level features in the multi-scale feature map pass through the channel attention block, spatial attention block and depth separation convolution block in sequence to obtain the enhanced multi-scale feature map.

3. The densely crowded pedestrian detection method based on the target detection model according to claim 2 is characterized in that: The channel attention block includes a first channel module, a second channel module, a first activation layer and a first product layer; The inputs of the first channel module and the second channel module are both low-level features in the multi-scale feature map; The input of the first activation layer is connected to the output of the first channel module and the output of the second channel module respectively to evaluate the channel attention weight; The input of the first product layer is connected to the output of the first activation layer and the output of the backbone network respectively to superimpose the channel attention weights on the low-level features in the multi-scale feature map. The first product layer outputs the channel-enhanced feature map.

4. The densely crowded pedestrian detection method based on the target detection model according to claim 3 is characterized in that: The spatial attention block includes a channel maximum pooling layer, a channel average pooling layer, a first convolutional layer, a second activation layer, and a second product layer; The channel maximum pooling layer and the channel average pooling layer are connected in parallel, and the input is connected to the output of the first product layer, and the output is connected to the input of the first convolutional layer to obtain local feature information; The input of the second activation layer is connected to the output of the first convolutional layer, and the output is connected to the second product layer to evaluate the spatial attention weight; The input of the second product layer is connected to the output of the second activation layer and the output of the first product layer respectively to superimpose the spatial attention weights on the channel-enhanced feature map, and the second product layer outputs the spatially enhanced feature map.

5. The densely crowded pedestrian detection method based on the target detection model according to claim 1 is characterized in that: The prediction suppression module includes a prediction filter, a relation information extractor, a local self-attention block, and a cross-attention block; The prediction filter receives the output from the decoder and filters out explicit predictions and non-explicit predictions based on the confidence of the predictions; The relation information extractor is used to extract the spatial feature relationship between the non-explicit prediction and the adjacent explicit prediction, and acts on the non-explicit prediction; The local self-attention block is used to self-update the non-explicit prediction by paying local attention to the neighboring predictions around the non-explicit prediction; The criss-cross attention block is used to learn updates that are not explicitly predicted from the features output by the decoder.

6. The densely crowded pedestrian detection method based on the target detection model according to claim 5 is characterized in that: In the prediction filter, the output of the decoder is predicted by the confidence detection head to obtain the confidence. The predictions with confidence > 0.7 constitute the explicit prediction set, and the predictions with confidence ≤ 0.7 constitute the implicit prediction set.

7. The densely crowded pedestrian detection method based on the target detection model according to claim 5 is characterized in that: The implicit prediction learns the neighboring features through the relation information extractor, specifically: Based on the spatial position information between predictions, Unspecified predictions Bounding box prediction for Bounding box predictions for neighboring explicit predictions ; Through the sine-cosine spatial position coding and Relationship modeling, obtaining location relationship information; The position relationship information is sequentially passed through the linear layer and the maximum pooling layer, and then element-by-element is superimposed on the non-explicit prediction. On the features after the linear layer, the superimposed features are output through the linear layer to obtain the implicit prediction of the learned neighboring features.

8. The densely crowded pedestrian detection method based on the target detection model according to claim 7 is characterized in that: The final prediction output process of the target pedestrian is as follows: The non-explicit predictions and the additionally introduced predictions constitute a supplemented non-explicit prediction set; The local self-attention block is a multi-head self-attention module with an attention mask added, and local attention is performed on the supplemented non-explicit prediction set based on the attention mask; The masked features are input into the deformable attention module to perform reinforcement learning on non-explicit predictions. After reinforcement learning, the supplementary predictions are discarded and merged with the explicit predictions as the final prediction output of the target pedestrian.

9. The densely crowded pedestrian detection method based on the target detection model according to claim 7 is characterized in that: The training process of the crowded pedestrian detection model is as follows: Obtain a public dataset of crowded crowd images as a training set for the crowded pedestrian detection model; Input the training set into the crowded pedestrian detection model; In the real box allocation, the prediction boxes that are clearly predicted are allocated first, and then the remaining prediction boxes are allocated; A hybrid matching strategy is introduced for the first few layers of the decoder. The hybrid matching strategy is as follows: parallel queries are introduced for the first few layers of the decoder, and the introduced parallel queries are isolated from other queries through an attention mask mechanism to form parallel training branches. Based on the matching strategy of matching each real box to multiple predictions, losses are constructed for the parallel training branches to train the crowded pedestrian detection model.

10. A densely crowded pedestrian detection system based on a target detection model, characterized in that: Inputting the dense crowd image to be detected into the crowded pedestrian detection model, which includes a backbone network, a feature enhancement and refinement module, a hybrid encoder and a decoder connected in sequence; The backbone network is used to extract multi-scale feature maps of dense and crowded crowd images; The feature enhancement and refinement module is used to refine and enhance the low-level features in the multi-scale feature map to obtain an enhanced multi-scale feature map; The hybrid encoder is used to encode the enhanced multi-scale feature map and transmit it to each layer of the decoder; The layers of the decoder are connected in sequence, and the last layer of the decoder is replaced by a prediction suppression module. The prediction suppression module is used to filter out explicit predictions and implicit predictions, and suppress implicit predictions to output the prediction results of the target pedestrian.

Images