A pedestrian detection method and system based on MPDNet model

Abstract

A pedestrian detection method and system based on the MPDNet model of a subway scene pedestrian detection network includes: obtaining raw data from subway stations and processing it to form a MetroStation dataset; inputting the training set of the dataset into a PVT model to extract sample image feature maps, which are then input into an ASFF‑FPN feature pyramid network for feature fusion to obtain a feature map after feature fusion; sending the feature map after feature fusion to a RetinaNet head module, and improving the bounding box regression loss function GIoU loss to obtain the category and position information of the target; forming an MPDNet model after model training using the aforementioned steps; and using the MPDNet model for pedestrian detection. The present invention solves the problem of varying degrees of angular distortion and occlusion of pedestrians caused by factors such as camera layout angles by using the PVT model in combination with an improved ASFF‑FPN structure to output feature maps and using the bounding box loss function GIoU loss to calculate the bounding box loss. This solves the problem of varying degrees of angular distortion and occlusion of pedestrians caused by factors such as camera layout angles, meets real-time requirements, and improves detection performance and training efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian detection methods and systems based on computer vision, and in particular relates to a pedestrian detection method and system based on the MPDNet model of a subway scene pedestrian detection network. Background Art

[0002] When using computer vision methods to detect pedestrians in underground spaces such as subways, the human body is quite flexible and has various postures and shapes. Its appearance is greatly affected by clothing, posture, viewing angle, etc., and it is also affected by factors such as occlusion and lighting. This makes pedestrian detection in complex scenes such as subways an extremely challenging problem in the field of computer vision.

[0003] In the era of deep learning, object detection algorithms are primarily categorized into two types: regression-based one-stage algorithms (such as SSD and the YOLO series) and candidate-box-based two-stage algorithms (such as R-CNN and Fast RCNN). The key difference between these two methods is that two-stage algorithms generate candidate regions and then use classification and regression to determine the location and category of objects in an image. However, this approach offers high detection accuracy but slow detection speed. One-stage algorithms use a simpler, more brute-force approach to directly predict the location and category of objects, improving detection speed. Their superior detection performance is achieved through high-resolution and multi-scale feature maps.

[0004] The Transformer is a self-attention-based model originally used in natural language processing (NLP). The Vision Transformer (ViT) was the first to apply a pure Transformer to image classification tasks. Since then, research on ViT has exploded, focusing on two main areas: improving ViT's image classification performance and applying it to other image tasks, such as segmentation and detection. However, existing methods cannot meet the real-time requirements of complex scenarios and lack verification in multiple scenarios such as subway stations. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a pedestrian detection method and system based on the subway scene pedestrian detection network MPDNet model, which can solve the problems of varying degrees of angular distortion and occlusion of pedestrians caused by factors such as camera layout angles, and can meet real-time requirements.

[0006] The embodiment of the present invention provides a pedestrian detection method based on the subway scene pedestrian detection network MPDNet model, comprising the following steps:

[0007] Step (1): Obtain the original data of the subway station, and process the original data to form the subway station MetroStation dataset;

[0008] Step (2): inputting the training set of the subway station MetroStation dataset into the PVT model to extract the feature map of the sample image;

[0009] Step (3): inputting the feature map into the ASFF-FPN feature pyramid network to perform feature fusion to obtain a feature map after feature fusion;

[0010] Step (4): Send the feature map after feature fusion to the RetinaNet head module, and modify the bounding box loss function to the improved bounding box regression loss function GIoU loss to obtain the category and location information of the target;

[0011] Step (5): After model training using the above steps (2), (3) and (4), an MPDNet model is formed;

[0012] Step (6): Input the real-time all-directional monitoring video stream data of the subway station into the MPDNet model for pedestrian detection.

[0013] Furthermore, the original data of the subway station in step (1) is derived from the subway station monitoring video stream data. According to the different passenger movement speeds in the various operating areas of the subway station, frame extraction processing is performed at different time intervals to obtain key frame images, and the key frame images are scaled and processed into a unified format. The processed key frame images are marked with pedestrian positions and the parameters of the rectangular frame are recorded to form a MetroStation data set.

[0014] Furthermore, the pyramid visual transformer PVT model in step (2) includes four stages, and the four stages generate feature maps of different scales, namely the first layer feature map P1, the second layer feature map P2, the third layer feature map P3 and the fourth layer feature map P4;

[0015] In step (3), the feature map is input into the ASFF-FPN feature pyramid network to perform feature fusion to obtain the feature map after feature fusion, which specifically includes:

[0016] Step (3.1): Convolve the fourth-layer feature map P4 to obtain the feature F4 of the feature pyramid network FPN, upsample the feature F4 by a factor of 2 and add the result of the convolution operation with the third-layer feature map P3 to obtain the feature F3, upsample the feature F3 by a factor of 2 and add the result of the convolution operation with the second-layer feature map P2 to obtain the feature F2, upsample the feature F2 by a factor of 2 and add the result of the convolution operation with the first-layer feature map P1 to obtain the feature F1;

[0017] Step (3.2): sequentially assign weight coefficients α, β, γ, and δ to the feature F1, the feature F2, the feature F3, and the feature F4;

[0018] Step (3.3): The adaptive feature fusion ASFF structure includes the first layer Layer1, the second layer Layer2, the third layer Layer3 and the fourth layer Layer4;

[0019] The calculation method of the fourth layer Layer4 is as follows:

[0020] The feature F4 is respectively multiplied by the corresponding weight coefficient δ, the feature F3 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 8 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, and the four dot multiplication results are added to obtain the feature map of the fourth layer Layer4;

[0021] The calculation method of the third layer Layer3 is as follows:

[0022] The feature F3 is respectively multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, and the four dot multiplication results are added to obtain the feature map of the third layer Layer3;

[0023] The calculation method of the second layer Layer2 is as follows:

[0024] The feature F2 is respectively multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, the feature F3 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the four dot multiplication results are added to obtain the feature map of the second layer Layer2;

[0025] The calculation method of the first layer Layer1 is as follows:

[0026] The feature F1 is respectively multiplied by the corresponding weight coefficient α, the feature F2 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient β, the feature F3 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the feature F4 is upsampled by 8 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ. The results of the four dot multiplication operations are added together to obtain the feature map of the first layer Layer1.

[0027] Furthermore, the RetinaNet head module in step (4) includes a classification module and a position detection module, each branch includes 4 convolutional layers, the classification module is used to obtain the category information of the target, and the position detection module is used to obtain the position information of the target.

[0028] Furthermore, in step (4), the improved bounding box regression loss function GIoUloss is used to calculate the bounding box loss, and the calculation formula of the GIoU is as follows:

[0029]

[0030] Where A is the predicted box, B is the real box, C is the minimum box that surrounds both the predicted box A and the real box B, IoU(A,B) is the ratio of the intersection and union of the predicted box A and the real box B, indicating the degree of overlap between the predicted box and the real box; the calculation formula of the bounding box regression loss function GIoUloss is:

[0031] L GIoU =1-GIoU.

[0032] Furthermore, if IoU(A, B) is greater than 0.7, the prediction box A is a positive sample, and the prediction box A is used for the model training; if IoU(A, B) is less than 0.3, the prediction box A is a negative sample, the prediction box does not contain the target, and the prediction box A is used for the model training; the prediction box A with IoU(A, B) between 0.3 and 0.7 is not used for the model training.

[0033] Furthermore, when IoU(A,B) is 0, the predicted box A deviates from the real box B, A∪B / C is infinitely close to 0, and GIoU approaches -1. When IoU(A,B) is 1, the predicted box A coincides with the real box B, A∪B / C is 1, so the value of GIoU is (-1,1].

[0034] Furthermore, the size of the input image of the pyramid visual converter PVT model in step (2) is H×W×3, where H represents the height of the input image and W represents the width of the input image. The pyramid visual converter PVT model outputs a first-layer feature map P1, a second-layer feature map P2, a third-layer feature map P3 and a fourth-layer feature map P4, and the corresponding output strides are 4, 8, 16, and 32, respectively.

[0035] An embodiment of the present invention provides a pedestrian detection system based on the subway scene pedestrian detection network MPDNet model, the system including the following modules:

[0036] A data set generation module, which is used to obtain original data of subway stations and process the original data to form a subway station MetroStation data set;

[0037] An extraction module, the extraction module is used to input the training set of the subway station MetroStation data set into the PVT model to extract the feature map of the sample image;

[0038] A feature fusion module is used to input the feature map into the ASFF-FPN feature pyramid network to perform feature fusion to obtain a feature map after feature fusion;

[0039] A recognition module is configured to send the feature map after the feature fusion to the RetinaNet head module and modify the bounding box loss function to an improved bounding box regression loss function GIoU loss to obtain the category and location information of the target;

[0040] A training module, which is used to form an MPDNet model after model training using the aforementioned extraction module, feature fusion module and recognition module;

[0041] The detection module is used to input the real-time all-directional monitoring video stream data of the subway station into the MPDNet model for pedestrian detection.

[0042] Furthermore, the original data of the subway station in the dataset generation module is derived from the subway station monitoring video stream data. According to the different passenger movement speeds in each operating area of ​​the subway station, frame extraction processing is performed at different time intervals to obtain key frame images, and the key frame images are scaled and processed into a unified format. The processed key frame images are marked with pedestrian positions and the parameters of the rectangular frame are recorded to form the MetroStation dataset.

[0043] Furthermore, the pyramid visual transformer (PVT) model in the extraction module includes four stages, which generate feature maps of different scales, namely, the first layer feature map P1, the second layer feature map P2, the third layer feature map P3 and the fourth layer feature map P4;

[0044] The feature fusion module inputs the feature map into the ASFF-FPN feature pyramid network to perform feature fusion to obtain the feature map after feature fusion, specifically including:

[0045] A feature pyramid network (FPN) construction module is configured to convolve the fourth-layer feature map P4 to obtain feature F4 of the feature pyramid network (FPN), upsample feature F4 by a factor of 2 and add the result of the convolution operation with the third-layer feature map P3 to obtain feature F3, upsample feature F3 by a factor of 2 and add the result of the convolution operation with the second-layer feature map P2 to obtain feature F2, and upsample feature F2 by a factor of 2 and add the result of the convolution operation with the first-layer feature map P1 to obtain feature F1;

[0046] A weight module, the weight module is used to sequentially assign weight coefficients α, β, γ, and δ to the feature F1, the feature F2, the feature F3, and the feature F4;

[0047] An adaptive feature fusion ASFF module, wherein the adaptive feature fusion ASFF module is used to construct an adaptive feature fusion ASFF structure, including a first layer Layer 1, a second layer Layer 2, a third layer Layer 3 and a fourth layer Layer 4;

[0048] The calculation method of the fourth layer Layer4 is as follows:

[0049] The feature F4 is respectively multiplied by the corresponding weight coefficient δ, the feature F3 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 8 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, and the four dot multiplication results are added to obtain the feature map of the fourth layer Layer4;

[0050] The calculation method of the third layer Layer3 is as follows:

[0051] The feature F3 is respectively multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, and the four dot multiplication results are added to obtain the feature map of the third layer Layer3;

[0052] The calculation method of the second layer Layer2 is as follows:

[0053] The feature F2 is respectively multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, the feature F3 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the four dot multiplication results are added to obtain the feature map of the second layer Layer2;

[0054] The calculation method of the first layer Layer1 is as follows:

[0055] The feature F1 is respectively multiplied by the corresponding weight coefficient α, the feature F2 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient β, the feature F3 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the feature F4 is upsampled by 8 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ. The results of the four dot multiplication operations are added together to obtain the feature map of the first layer Layer1.

[0056] Furthermore, the RetinaNet head module in the recognition module includes a classification module and a position detection module, each branch includes 4 convolutional layers, the classification module is used to obtain the category information of the target, and the position detection module is used to obtain the position information of the target.

[0057] Furthermore, the recognition module uses an improved bounding box regression loss function GIoU loss to calculate the bounding box loss. The calculation formula of the GIoU is as follows:

[0058]

[0059] Where A is the predicted box, B is the real box, C is the minimum box that surrounds both the predicted box A and the real box B, IoU(A,B) is the ratio of the intersection and union of the predicted box A and the real box B, indicating the degree of overlap between the predicted box and the real box; the calculation formula of the bounding box regression loss function GIoUloss is:

[0060] L GIoU =1-GIoU.

[0061] Furthermore, if IoU(A, B) is greater than 0.7, the prediction box A is a positive sample, and the prediction box A is used for the model training; if IoU(A, B) is less than 0.3, the prediction box A is a negative sample, the prediction box does not contain the target, and the prediction box A is used for the model training; the prediction box A with IoU(A, B) between 0.3 and 0.7 is not used for the model training.

[0062] Furthermore, when IoU(A,B) is 0, the predicted box A deviates from the real box B, A∪B / C is infinitely close to 0, and GIoU approaches -1. When IoU(A,B) is 1, the predicted box A coincides with the real box B, A∪B / C is 1, so the value of GIoU is (-1,1].

[0063] Furthermore, the size of the input image of the pyramid visual converter PVT model in the extraction module is H×W×3, where H represents the height of the input image and W represents the width of the input image. The pyramid visual converter PVT model outputs the first layer feature map P1, the second layer feature map P2, the third layer feature map P3 and the fourth layer feature map P4, and the corresponding output strides are 4, 8, 16, and 32 respectively.

[0064] The above technical solutions of the embodiments of the present invention have the following beneficial technical effects:

[0065] This paper proposes a real-time subway scene detection model MPDNet. By using the PVT model in conjunction with the improved ASFF-FPN structure to output feature maps, the present invention solves the problem of varying degrees of angular distortion and occlusion of pedestrians caused by factors such as camera placement angles, meets real-time requirements, and improves detection performance and training efficiency. The PVT model obtains global feature information from the image and, in conjunction with the improved ASFF-FPN structure, fuses low-level and high-level features in the feature pyramid network (FPN). It utilizes both the high resolution of low-level features and the high semantic information of high-level features, sharing semantic information across all feature layers. Furthermore, to address the structural defect of FPN that causes semantic information loss during feature fusion, the present invention abandons the previous method of directly adding features with large semantic information gaps. Instead, features at different levels are weighted in the feature space, retaining useful information for combination, achieving adaptive spatial feature fusion and enhancing the effect of feature fusion. In the head, the bounding box loss function (GIoU) ​​solves the problem of the inability to calculate gradients when two boxes overlap when IoU is used as the loss function, and adds a minimum outer bounding box as a penalty term. Focusing not only on overlapping areas but also on non-overlapping regions better reflects the proximity of two boxes, making it more effective in predicting box regression in complex scenes with densely populated objects, such as subways. This method's excellent performance in subway scenarios demonstrates its suitability for real-time regional passenger flow detection in subway stations and is particularly well-suited for pedestrian detection in high-density scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a model diagram of the subway scene pedestrian detection network MPDNet (MetroStationPedestrian detection Net) provided by an embodiment of the present invention.

[0067] Figure 2 It is a schematic diagram of the processing process of the subway scene pedestrian detection network MPDNet (MetroStationPedestrian detection Net) provided by an embodiment of the present invention.

[0068] Figure 3 This is a structural diagram of the adaptive feature fusion ASFF-FPN feature pyramid network provided by an embodiment of the present invention.

[0069] Figure 4 This is a schematic diagram showing the visualization of the detection effect of the MPDNet model provided by an embodiment of the present invention on the MetroStation dataset. DETAILED DESCRIPTION

[0070] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0071] The pedestrian detection method and system based on the MetroStation Pedestrian Detection Net (MPDNet) model proposed in the embodiments of the present invention are mainly used in complex scenes in underground spaces such as subways.

[0072] The complete MPDNet network model structure in this invention Figure 1 shown.

[0073] The pedestrian detection method and system of the present invention include the following steps:

[0074] (1) Obtaining the video stream data of all directions of the subway station to form the subway station MetroStation dataset

[0075] When creating the MetroStation dataset for subway stations, taking the Beijing subway scene as an example, the goal is to present different operating scenarios of subway stations as comprehensively as possible, including different camera angles, different lighting conditions, and different passenger flow densities. In this invention, it is observed that the pedestrian movement speeds in various areas of dense scenes are different. In order to accurately obtain key frame images, data analysis is performed on historical monitoring video stream data, or mathematical models are used to predict historical monitoring video stream data, or real-time data monitoring is performed to obtain pedestrian movement speeds in different areas of dense scenes, thereby obtaining different frame extraction time intervals for different areas. For subway stations, frames are extracted at different time intervals according to the different passenger movement speeds in various operating areas of the station. After the frame extraction process, key frame images are obtained. After the key frame images are scaled and processed into a unified format, the LabelImg tool is used to mark the pedestrian positions and record the parameters of the rectangular box to create the MetroStation dataset;

[0076] In this example, the complex scene selected is a subway station. The created MetroStation dataset contains 2123 annotated images of size 640*480, with a total of 12974 labels. 1699 of them are divided into training sets, and the rest are used for testing.

[0077] (2) Input the training set sample images of the MetroStation dataset into the PVT (Pyramid Vision Transformer) model to extract the feature maps of the training set sample images;

[0078] The PVT model is superior to the CNN model that only focuses on local features. The present invention innovatively uses PVT (Pyramid Vision Transformer) as the backbone network of the model combined with the structure of the adaptive feature fusion ASFF-FPN feature pyramid network to improve detection performance and training efficiency.

[0079] In this embodiment, the training set of the MetroStation data set is input into the PVT model to extract the feature map of the sample image.

[0080] like Figure 1 As shown in the figure, PVT has four stages, each generating feature maps of different scales. Each stage has a similar structure, consisting of a patch embedding layer and a Transformer Encoder layer. The input image size is H×W×3, where H represents the height and W represents the width. The output feature maps of stages 1 to 4 are the first-layer feature map P1, the second-layer feature map P2, the third-layer feature map P3, and the fourth-layer feature map P4, respectively, with corresponding output strides of 4, 8, 16, and 32, respectively.

[0081] (3) Input the feature map into the adaptive feature fusion ASFF-FPN feature pyramid network for feature fusion:

[0082] like Figure 1 As shown, in the present invention, PVT is used as the backbone network of the model in combination with the structure of the adaptive feature fusion ASFF-FPN feature pyramid network to build an ASFF-FPN coordinated with PVT. First, FPN is built: after obtaining the four outputs P1, P2, P3, and P4 of PVT, the fourth-layer feature map P4 is subjected to a 1×1 convolution operation to obtain the feature F4 of the feature pyramid network FPN, the feature F4 is upsampled by 2 times the nearest neighbor interpolation method and convolved with the third-layer feature map P3 to obtain the feature F3, the feature F3 is upsampled by 2 times the nearest neighbor interpolation method and convolved with the second-layer feature map P2 to obtain the feature F2, the feature F2 is upsampled by 2 times the nearest neighbor interpolation method and convolved with the first-layer feature map P1 to obtain the feature F1; at this point, F1, F2, and F3 complete the first step of feature fusion, and the underlying features such as F1 have now fused the features of P2-P4, and a stronger feature expression capability has been initially obtained.

[0083] Next, we use the ASFF method to improve the FPN performance. After obtaining the four FPN outputs F1, F2, F3, and F4, we assign weight coefficients α, β, γ, and δ in sequence.

[0084] like Figure 1 and Figure 2 As shown, the ASFF structure of the present invention combined with PVT includes four layers: the first layer Layer 1, the second layer Layer 2, the third layer Layer 3 and the fourth layer Layer 4;

[0085] The calculation method of the fourth layer Layer4 is as follows:

[0086] Feature F4 is multiplied by the corresponding weight coefficient δ, feature F3 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient γ, feature F2 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, feature F1 is downsampled by 8 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, and the four dot multiplication results are added together to obtain the feature map of the fourth layer Layer4;

[0087] The calculation method of the third layer Layer3 is as follows:

[0088] Feature F3 is multiplied by the corresponding weight coefficient γ, feature F2 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, feature F1 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, feature F4 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ. The results of the four dot multiplication operations are added together to obtain the feature map of the third layer Layer3;

[0089] The calculation method of the second layer Layer2 is as follows:

[0090] Feature F2 is multiplied by the corresponding weight coefficient β, feature F1 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, feature F4 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, feature F3 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ. The results of the four dot multiplication operations are added together to obtain the feature map of the second layer Layer2;

[0091] The calculation method of the first layer Layer1 is as follows:

[0092] Feature F1 is multiplied by the corresponding weight coefficient α, feature F2 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient β, feature F3 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, feature F4 is upsampled by 8 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, and the results of the four dot multiplication operations are added together to obtain the feature map of the first layer Layer1.

[0093] (4) Send the feature map after feature fusion to the RetinaNet head module, and modify the bounding box loss function to GIoU loss to obtain the category and location information of the target

[0094] Step (3) obtains the feature maps of the four layers respectively. The obtained four feature maps are input into the RetinaNet head structure. The RetinaNet head module includes two branches: classification and position detection. Each branch includes four convolutional layers. The classification module is used to obtain the category information of the target, and the position detection module is used to obtain the position information of the target.

[0095] In order to address the shortcoming that the loss function L1 Loss used in RetinaNet head does not consider the correlation between direction and coordinates, the bounding box regression loss function GIoU loss is used to calculate the bounding box loss. The GIoU formula is as follows:

[0096]

[0097] Where A is the predicted box, B is the real box, and C is the minimum bounding box that surrounds both A and B. IoU(A,B) represents the ratio of the intersection to the union of A and B, indicating the degree of overlap between the predicted box and the real box. If IoU is greater than 0.7, the predicted box is called a positive sample; if IoU is less than 0.3, it is called a negative sample, and it is considered that the predicted box does not contain the target. Prediction boxes with IoU between 0.3 and 0.7 are not used for subsequent model training. When IoU is 0, the predicted box deviates from the real box, and the two have no intersection. A∪B / C is infinitely close to 0, and GIoU approaches -1. Similarly, when IoU is 1, the predicted box coincides with the real box, A∪B / C is 1, so the value of GIoU is (-1,1]. The bounding box regression loss function GIoU loss is expressed as:

[0098] L GIoU =1-GIoU

[0099] (5) After the training of steps (2), (3) and (4) is completed, the MPDNet network model is formed.

[0100] The real-time monitoring video stream data of the subway station in all directions is input into the MPDNet model for pedestrian detection. The present invention tests the detection effect of the MPDNet model on the test set of the MetroStation dataset. The detection effect is visualized as shown in the following diagram: Figure 4 As shown in Figure 2, using mean Average Percision (mAP) as the accuracy measure, the MPDNet model detection accuracy reached 94.0.

[0101] An embodiment of the present invention further provides a pedestrian detection system based on the subway scene pedestrian detection network MPDNet model, the system comprising:

[0102] A dataset generation module for obtaining raw data of subway stations and processing the raw data to form a subway station MetroStation dataset;

[0103] An extraction module for inputting the training set of the subway station MetroStation data set into the PVT model to extract the feature map of the sample image;

[0104] A feature fusion module for inputting the feature map into the ASFF-FPN feature pyramid network to perform feature fusion to obtain a feature map after feature fusion;

[0105] A recognition module for sending the feature map after the feature fusion to the RetinaNet head module and modifying the bounding box loss function to the improved bounding box regression loss function GIoU loss to obtain the category and location information of the target;

[0106] A training module for forming an MPDNet model after model training using the aforementioned extraction module, feature fusion module, and recognition module;

[0107] A detection module is used to input the real-time all-directional monitoring video stream data of the subway station into the MPDNet model for pedestrian detection.

[0108] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A pedestrian detection method based on the subway scene pedestrian detection network MPDNet model, comprising the following steps: Step (1): Obtain the original data of the subway station, and process the original data to form the subway station MetroStation dataset; Step (2): inputting the training set of the subway station MetroStation dataset into the pyramid visual transformer PVT model to extract the feature map of the sample image; Step (3): inputting the feature map into the adaptive feature fusion ASFF-FPN feature pyramid network to perform feature fusion to obtain a feature map after feature fusion; Step (4): Send the feature map after feature fusion to the RetinaNet head module, and modify the bounding box loss function to the improved bounding box regression loss function GIoU loss to obtain the category and location information of the target; Step (5): After model training using the above steps (2), (3) and (4), an MPDNet model is formed; Step (6): Input the real-time all-directional monitoring video stream data of the subway station into the MPDNet model for pedestrian detection.

2. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 1, characterized in that: The original data of the subway station in step (1) is derived from the subway station monitoring video stream data. According to the different passenger movement speeds in the various operating areas of the subway station, frame extraction processing is performed at different time intervals to obtain key frame images, and the key frame images are scaled and processed into a unified format. The processed key frame images are marked with pedestrian positions and the parameters of the rectangular frame are recorded to form a MetroStation data set.

3. A pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 1 or 2, characterized in that: The pyramid visual converter (PVT) model in step (2) includes four stages, and the four stages generate feature maps of different scales, namely, a first-layer feature map P1, a second-layer feature map P2, a third-layer feature map P3, and a fourth-layer feature map P4; In the step (3), the feature map is input into the adaptive feature fusion ASFF-FPN feature pyramid network to perform feature fusion to obtain the feature map after feature fusion, which specifically includes: Step (3.1): Convolve the fourth-layer feature map P4 to obtain the feature F4 of the feature pyramid network FPN, upsample the feature F4 by a factor of 2 and add the result of the convolution operation with the third-layer feature map P3 to obtain the feature F3, upsample the feature F3 by a factor of 2 and add the result of the convolution operation with the second-layer feature map P2 to obtain the feature F2, upsample the feature F2 by a factor of 2 and add the result of the convolution operation with the first-layer feature map P1 to obtain the feature F1; Step (3.2): assign weight coefficients α, β, γ, δ to the feature F1, the feature F2, the feature F3, and the feature F4 in sequence; Step (3.3): The structure of the adaptive feature fusion ASFF includes the first layer Layer1, the second layer Layer2, the third layer Layer3 and the fourth layer Layer4; The calculation method of the fourth layer Layer4 is as follows: The feature F4 is respectively multiplied by the corresponding weight coefficient δ, the feature F3 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 8 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, and the four dot multiplication results are added to obtain the feature map of the fourth layer Layer4; The calculation method of the third layer Layer3 is as follows: The feature F3 is respectively multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, and the four dot multiplication results are added to obtain the feature map of the third layer Layer3; The calculation method of the second layer Layer2 is as follows: The feature F2 is respectively multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, the feature F3 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the four dot multiplication results are added to obtain the feature map of the second layer Layer2; The calculation method of the first layer Layer1 is as follows: The feature F1 is respectively multiplied by the corresponding weight coefficient α, the feature F2 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient β, the feature F3 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the feature F4 is upsampled by 8 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ. The results of the four dot multiplication operations are added together to obtain the feature map of the first layer Layer1.

4. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 3, characterized in that: The RetinaNet head module in step (4) includes a classification module and a position detection module, each branch includes 4 convolutional layers, the classification module is used to obtain the category information of the target, and the position detection module is used to obtain the position information of the target.

5. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 3 is characterized in that: In step (4), the improved bounding box regression loss function GIoU loss is used to calculate the bounding box loss. The calculation formula of GIoU is as follows: Where A is the predicted box, B is the real box, C is the minimum box that surrounds both the predicted box A and the real box B, IoU(A,B) is the ratio of the intersection and union of the predicted box A and the real box B, indicating the degree of overlap between the predicted box and the real box; the calculation formula of the bounding box regression loss function GIoUloss is: L GIoU = 1-GIoU.

6. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 5, characterized in that: If IoU(A,B) is greater than 0.7, the prediction box A is a positive sample, and the prediction box A is used for the model training; if IoU(A,B) is less than 0.3, the prediction box A is a negative sample, the prediction box does not contain an object, and the prediction box A is used for the model training; the prediction box A with IoU(A,B) between 0.3 and 0.7 is not used for the model training.

7. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 6, characterized in that: When IoU(A,B) is 0, the predicted box A deviates from the real box B, A∪B / C is infinitely close to 0, and GIoU approaches -1. When IoU(A,B) is 1, the predicted box A coincides with the real box B, A∪B / C is 1, so the value of GIoU is (-1,1].

8. The pedestrian detection method based on the subway scene pedestrian detection network MPDNet model according to claim 1 or 2, characterized in that: The size of the input image of the pyramid visual converter PVT model in step (2) is H×W×3, where H represents the height of the input image and W represents the width of the input image. The pyramid visual converter PVT model outputs a first-layer feature map P1, a second-layer feature map P2, a third-layer feature map P3 and a fourth-layer feature map P4, and the corresponding output strides are 4, 8, 16, and 32 respectively.

9. A pedestrian detection system based on the subway scene pedestrian detection network MPDNet model, the system includes the following modules: A data set generation module, which is used to obtain original data of subway stations and process the original data to form a subway station MetroStation data set; An extraction module, wherein the extraction module is used to input the training set of the subway station MetroStation data set into a pyramid visual transformer (PVT) model to extract a feature map of a sample image; A feature fusion module, wherein the feature fusion module is used to input the feature map into an adaptive feature fusion ASFF-FPN feature pyramid network to perform feature fusion to obtain a feature map after feature fusion; A recognition module is configured to send the feature map after the feature fusion to the RetinaNet head module and modify the bounding box loss function to an improved bounding box regression loss function GIoU loss to obtain the category and location information of the target; A training module, which is used to form an MPDNet model after model training using the aforementioned extraction module, feature fusion module and recognition module; The detection module is used to input the real-time all-directional monitoring video stream data of the subway station into the MPDNet model for pedestrian detection.

10. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 9, characterized in that: The raw data of the subway station in the dataset generation module is derived from the subway station surveillance video stream data. According to the different passenger movement speeds in each operating area of ​​the subway station, frame extraction processing is performed at different time intervals to obtain key frame images. The key frame images are scaled and processed into a unified format. The processed key frame images are marked with pedestrian positions and the parameters of the rectangular frame are recorded to form the MetroStation dataset.

11. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 9, characterized in that: The pyramid visual transformer (PVT) model in the extraction module includes four stages, which generate feature maps of different scales, namely the first layer feature map P1, the second layer feature map P2, the third layer feature map P3 and the fourth layer feature map P4; The feature fusion module inputs the feature map into the adaptive feature fusion ASFF-FPN feature pyramid network to perform feature fusion to obtain the feature map after feature fusion, which specifically includes: A feature pyramid network (FPN) construction module is configured to convolve the fourth-layer feature map P4 to obtain feature F4 of the feature pyramid network (FPN), upsample feature F4 by a factor of 2 and add the result of the convolution operation with the third-layer feature map P3 to obtain feature F3, upsample feature F3 by a factor of 2 and add the result of the convolution operation with the second-layer feature map P2 to obtain feature F2, and upsample feature F2 by a factor of 2 and add the result of the convolution operation with the first-layer feature map P1 to obtain feature F1; A weight module, the weight module is used to sequentially assign weight coefficients α, β, γ, and δ to the feature F1, the feature F2, the feature F3, and the feature F4; An adaptive feature fusion ASFF module, wherein the adaptive feature fusion ASFF module is used to construct an adaptive feature fusion ASFF structure, wherein the structure includes a first layer Layer 1, a second layer Layer 2, a third layer Layer 3, and a fourth layer Layer 4; The calculation method of the fourth layer Layer4 is as follows: The feature F4 is respectively multiplied by the corresponding weight coefficient δ, the feature F3 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 8 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, and the four dot multiplication results are added to obtain the feature map of the fourth layer Layer4; The calculation method of the third layer Layer3 is as follows: The feature F3 is respectively multiplied by the corresponding weight coefficient γ, the feature F2 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 4 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, and the four dot multiplication results are added to obtain the feature map of the third layer Layer3; The calculation method of the second layer Layer2 is as follows: The feature F2 is respectively multiplied by the corresponding weight coefficient β, the feature F1 is downsampled by 2 times using the maximum pooling method and then multiplied by the corresponding weight coefficient α, the feature F4 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ, the feature F3 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the four dot multiplication results are added to obtain the feature map of the second layer Layer2; The calculation method of the first layer Layer1 is as follows: The feature F1 is respectively multiplied by the corresponding weight coefficient α, the feature F2 is upsampled by 2 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient β, the feature F3 is upsampled by 4 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient γ, and the feature F4 is upsampled by 8 times using the nearest neighbor interpolation method and then multiplied by the corresponding weight coefficient δ. The results of the four dot multiplication operations are added together to obtain the feature map of the first layer Layer1.

12. A pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 9 or 10, characterized in that: The RetinaNet head module in the recognition module includes a classification module and a position detection module. Each branch includes 4 convolutional layers. The classification module is used to obtain the category information of the target, and the position detection module is used to obtain the position information of the target.

13. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 12, characterized in that: The recognition module uses the improved bounding box regression loss function GIoU loss to calculate the bounding box loss. The calculation formula of the GIoU is as follows: Where A is the predicted box, B is the real box, C is the minimum box that surrounds both the predicted box A and the real box B, IoU(A,B) is the ratio of the intersection and union of the predicted box A and the real box B, indicating the degree of overlap between the predicted box and the real box; the calculation formula of the bounding box regression loss function GIoUloss is: L GIoU = 1-GIoU.

14. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 13, characterized in that: If IoU(A,B) is greater than 0.7, the prediction box A is a positive sample, and the prediction box A is used for the model training; if IoU(A,B) is less than 0.3, the prediction box A is a negative sample, the prediction box does not contain an object, and the prediction box A is used for the model training; the prediction box A with IoU(A,B) between 0.3 and 0.7 is not used for the model training.

15. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 14, characterized in that: When IoU(A,B) is 0, the predicted box A deviates from the real box B, A∪B / C is infinitely close to 0, and GIoU approaches -1. When IoU(A,B) is 1, the predicted box A coincides with the real box B, A∪B / C is 1, so the value of GIoU is (-1,1].

16. The pedestrian detection system based on the subway scene pedestrian detection network MPDNet model according to claim 9, characterized in that: The size of the input image of the pyramid visual converter PVT model in the extraction module is H×W×3, where H represents the height of the input image and W represents the width of the input image. The pyramid visual converter PVT model outputs the first layer feature map P1, the second layer feature map P2, the third layer feature map P3 and the fourth layer feature map P4, and the corresponding output strides are 4, 8, 16, and 32 respectively.

Images