Entrance and exit traffic monitoring method based on deep learning

Abstract

The invention discloses a method for monitoring entrance and exit traffic based on deep learning, which specifically relates to the technical field of smart library. The method comprises the following steps: obtaining pedestrian detection data in crowded scenes, constructing a data set A, and performing preprocessing; retrieving videos shot by cameras at the entrances and exits of a monitoring area, constructing a data set B, and performing manual annotation and preprocessing; training and verifying the data set A by using a CCS-YOLOv7 model, and updating the parameters of the CCS-YOLOv7 model; migrating the pre-trained weights to the CCS-YOLOv7 model, training and testing the data set B, and updating the parameters of the CCS-YOLOv7 model; combining the CCS-YOLOv7 model with a DeepSORT algorithm to construct a monitoring model for the entrance and exit traffic of a monitoring area, so as to learn and detect specific features of pedestrians at the entrances and exits of the monitoring area; and evaluating the monitoring model for the entrance and exit traffic of a monitoring area.

Description

Technical Field

[0001] The present invention relates to the technical field of human flow monitoring, and in particular to a method for monitoring human flow at an entrance and exit based on deep learning. Background Art

[0002] In recent years, artificial intelligence technologies represented by deep learning are rapidly becoming a research hotspot and are widely used. Libraries, hospitals, stations, etc. are important places for people to learn, receive medical treatment, and travel. The flow of people at their entrances and exits can provide key data, thereby helping managers to effectively manage venue resources and provide better services. By accurately counting the number of pedestrians at the entrances and exits, managers can understand the busy hours and passenger flow peaks, and then reasonably arrange personnel and resources to optimize the operational efficiency of the venue. In addition, crowd flow detection can also help to promptly detect crowds and dense places to ensure the health and safety of the crowd.

[0003] For example, in libraries, in recent years, with the development of deep learning technology, CNN has made some progress in the field of pedestrian detection. However, in complex scenarios such as smart libraries where there is frequent flow of people and information interference from non-readers (managers), current pedestrian flow counting algorithms face many challenges. Existing research has tried a variety of algorithms, Sermanet, Pierre, et al. "Pedestrian detection with unsupervised multi-stage feature learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2013. Use convolutional coefficient encoding scheme to train the convolutional layer [1],Ouyang, Wanli, et al. "Deepid-net: Deformable deep convolutional neural networks for object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. Try to combine contextual learning into human detection to improve accuracy, Li, Haoxiang, et al. "A convolutional neural network cascade for face detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. Try to use memory neural network to extract features from image sequences [3] , and Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances inneural information processing systems 28 (2015). Use R-CNN to process pedestrian detection [4] .

[0004] However, these methods suffer from problems such as insufficient feature extraction, poor real-time performance, and decreased recognition effect in crowded scenes. Summary of the invention

[0005] In view of the fact that the current algorithms have not been able to effectively solve the problem of pedestrian detection in complex scenarios such as smart libraries, this paper proposes a pedestrian detection algorithm that combines CCS-YOLOv7 with DeepSORT. By realizing high-order interaction between information, the algorithm can increase the focus on important features, thereby effectively meeting the needs of reader flow statistics in complex scenarios such as smart libraries.

[0006] To this end, the present invention provides an entrance and exit traffic monitoring method based on deep learning to solve the problems raised in the background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a method for monitoring entrance and exit traffic based on deep learning, comprising the following steps:

[0008] S1. Obtain pedestrian detection data in crowded scenes, construct dataset A, and perform preprocessing;

[0009] S2, retrieve the videos shot by the cameras at the entrance and exit of the monitoring area, construct data set B, and perform manual annotation and preprocessing;

[0010] S3. Use the CCS-YOLOv7 model to train and verify the data set A, and update the CCS-YOLOv7 model parameters to save the pre-trained weights with the minimum loss;

[0011] S4, migrating the pre-trained weights to the CCS-YOLOv7 model, training and testing the data set B, and updating the CCS-YOLOv7 model parameters;

[0012] S5. The CCS-YOLOv7 model is combined with the DeepSORT algorithm to build a pedestrian flow monitoring model at the entrance and exit of the monitoring area to learn and detect the specific characteristics of pedestrians at the entrance and exit of the monitoring area, thereby improving the accuracy and timeliness of the detection of the pedestrian flow monitoring model at the entrance and exit of the monitoring area;

[0013] S6. Evaluate the pedestrian flow monitoring model at the entrance and exit of the monitoring area, and test the effectiveness of the pedestrian flow monitoring model at the entrance and exit of the monitoring area under real-time camera.

[0014] Preferably, in S1, the preprocessing operation on the crowded scene pedestrian detection data includes: performing data enhancement on the data set A by random horizontal rotation and random vertical flipping, and dividing it into a training set and a test set.

[0015] Preferably, in S2, the annotated labels include three categories: in, out and others; wherein in represents non-staff entering, out represents non-staff leaving, and others represents staff. Through this annotation method, stationary objects in the video sequence are excluded; the preprocessing operation in S2 is the same as the preprocessing in S1.

[0016] Preferably, in S3, the loss function CIOU in the CCS-YOLOv7 model is used to compare the difference between the detection result and the true result, and the parameter gradient of the loss function with respect to the neural network is calculated, and the parameter gradient is used to update the CCS-YOLOv7 model parameters, so that the detection result of the model is close to the true result manually labeled; CCS-YOLOv7 is a CCS module that adds a cross-channel information fusion mechanism before the SPPCSPC module of the YOLOv7 model.

[0017] Preferably, the CCS module of the cross-channel information fusion mechanism is designed as follows:

[0018] The CCS module divides the input into three branches, namely, branch 1, branch 2 and branch 3, for feature fusion. First, branch 1 uses convolution layer 10 to halve the number of channels of input x to obtain feature map Feature10. Feature10 is activated by AvgPool and Sigmoid to obtain attention vector g1. At the same time, Feature10 is converted to four times the original number of channels by convolution layer 11, and then reduced to one-fourth of the number of channels of the input by convolution layer 12 to obtain feature map Feature12.

[0019] Secondly, Feature12 is multiplied by g1 element by element to obtain feature map Feature13. Finally, Feature13 is added to feature vector Feature12 through convolution layer 13 to obtain Multi Feature1.

[0020] Branch 2 first uses convolution layer 20 to halve the number of channels of input x to obtain feature map Feature20. After Feature20 passes through MaxPool and Sigmoid activation functions, it obtains attention vector g2. At the same time, Feature20 passes through convolution layer 21, and the number of channels becomes four times the original. After passing through convolution layer 22, the number of channels is converted to one-fourth of the input to obtain feature map Feature22. Then Feature22 is element-wise multiplied by g2 to obtain feature map Feature23. Finally, Feature23 passes through convolution layer 23 and is added to feature vector Feature22 to obtain Multi Feature2. Multi Feature1 and MultiFeature2 are concatenated in the channel dimension to generate feature map Concat Feature.

[0021] Branch 3 is the residual link, which adds the input x to the Concat Feature to get the output of the CCS module.

[0022] Preferably, in S3, the torch.save() method in the pytorch toolkit is used to save the pre-trained weights with the minimum loss.

[0023] Preferably, in S4, the pre-trained weights described in S3 are loaded using the torch.load() method in the pytorch toolkit, and then the data set B is trained and tested, and the CCS-YOLOv7 model parameters are updated; wherein, updating the CCS-YOLOv7 model parameters is to use CIOU to compare the difference between the monitoring results and the true results, and calculate the parameter gradient of the loss function with respect to the neural network, and use the parameter gradient to update the network parameters of the CCS-YOLOv7 model, so that the detection results of the model are close to the true results manually labeled.

[0024] Preferably, in S5, the specific steps of constructing a monitoring model for the flow of people at the entrance and exit of the monitoring area include:

[0025] First, filter the labels generated by S4, exclude labels with the category of others, and filter labels with a box width less than 20px and a height less than 50px. Use the CCS-YOLOv7 model to detect readers in dataset B and generate bounding boxes and category labels.

[0026] Use the DeepSORT model to track pedestrians in the video and generate unique identifiers to track pedestrians.

[0027] Preferably, in S6, under the test real-time camera, a pedestrian flow monitoring model at the entrance and exit of the monitoring area is used to detect and track pedestrians in each frame of the video stream.

[0028] The present invention has the following advantages:

[0029] The present invention provides an advanced target detection model, namely the CCS-YOLOv7 model. By introducing the CCS module, the CCS-YOLOv7 model can make full use of different branches and cross-channel information fusion mechanisms to more effectively learn the characteristics of input data and improve the representation ability of the model. The CCS module facilitates information interaction between channels by fusing information from different channels, enhancing model understanding and learning features. In addition, the multiple residual connections within the CSS module help alleviate the problem of gradient vanishing or gradient exploding, and enhance the training stability and generalization ability of the model;

[0030] In addition, the CCS-YOLOv7 model is pre-trained in a crowded scene dataset to overcome the challenges of the model when dealing with dense targets. By pre-training on real-world crowded scene data, the CCS-YOLOv7 model can better adapt to target detection tasks in various complex environments and perform more robustly and reliably in practical applications.

[0031] The present invention uses the Labelme tool to label stationary objects in a video sequence as others, thereby solving the problem of unstable detection results that may be caused in a video sequence in which moving targets and stationary objects are mixed. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a smart library entrance and exit traffic monitoring model constructed by combining the CCS-YOLOv7 model provided by the present invention with the DeepSORT algorithm;

[0033] Figure 2 The present invention provides Figure 1 Schematic diagram of training and testing of the model;

[0034] Figure 3 Schematic diagram of the confusion matrix without setting the others category and with setting the others category provided by the present invention. DETAILED DESCRIPTION

[0035] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] This embodiment takes the library as an example and provides a method for monitoring the flow of people at the entrances and exits of a smart library based on deep learning. The method for monitoring the flow of people at the entrances and exits of a smart library based on deep learning first collects video data of crowded scenes and library entrances and exits to construct data sets A and B respectively. Then, a CCS-YOLOv7 model is built and pre-trained in data set A to obtain the pre-trained weights with the lowest loss value. Secondly, the CCS-YOLO v7 model loads the pre-trained weights and trains and tests data set B. Then, combined with the DeepSORT algorithm, a CCS-YOLOv7 model with a DeepSort algorithm is formed to improve the accuracy and timeliness of reader detection in crowded scenes. The method and device for monitoring the flow of people at the entrances and exits of smart libraries based on deep learning mainly include the following steps:

[0037] Step 1: Collect pedestrian detection data in occluded scenes, such as the CUHK Occlusion dataset, to form a dataset A, and perform preprocessing, wherein the preprocessing operation includes: performing data enhancement on the dataset A by random horizontal rotation and random vertical flipping to prevent overfitting of training and improve the generalization ability of the model, and randomly dividing it into a training set and a test set in a ratio of 8:2.

[0038] Step 2: Retrieve the video captured by the camera at the entrance and exit of the library of the Jiangjun Road Campus of Nanjing University of Aeronautics and Astronautics, extract a picture every 30 frames, and construct Dataset B. Dataset B has a total of 2237 data, which are randomly divided into training set and test set in a ratio of 8:2. Then use the LabelImg tool to annotate, where the annotated categories are in (entry), out (exit) and others (others). By annotating static objects in the video sequence as "others", the problem of unstable detection results that may be caused in video sequences with mixed moving targets and static objects is solved, and the influence of irrelevant personnel such as librarians on the judgment is solved.

[0039] Step 3: Model building and pre-training. This embodiment provides a CCS module with a cross-channel information fusion mechanism. The structure of the CCS module is as follows: Figure 1 As shown. Among them, all convolutional layer structures are the same, including convolutional layers, batch normalization, and activation functions. k1 and k3 represent the number of convolution kernels as 1 and 3 respectively, s1 represents the convolution step size as 1, p0 represents no pixel padding, and p1 represents the pixel padding as 1. Concat means concatenating feature maps in the channel dimension. AvgPool and MaxPool represent global average pooling and global maximum pooling respectively. Among them, the activation function is GELU, and the formula is as follows:

[0040]

[0041] Then, the model is trained and tested on the training set and validation set in data set A, that is, CIOU is used to compare the difference between the detection result and the real result, and the parameter gradient of the loss function with respect to the neural network is calculated, and the network parameters of the CCS-CCS-YOLOv7 model are updated with the gradient, so that the network classification result is close to the real result of manual annotation, thereby continuously improving the network performance, and finally torch.save() in the pytorch toolkit is used to save the pre-trained weights with the minimum loss;

[0042] Step 4: If Figure 2 As shown, after migrating the pre-trained weights, the dataset B is trained and tested. First, the pre-trained weights described in step 3 are loaded using the torch.load() function in the pytorch toolkit, and then the dataset B is trained and tested. CIOU is also used to compare the difference between the detection results and the true results, and the loss function is calculated with respect to the parameter gradient of the neural network, and the gradient is used to update the network parameters of the CCS-YOLOv7 model.

[0043] Step 5: Specifically, use the CCS-YOLOv7 model to detect readers in dataset B and generate bounding boxes and category labels; then filter the labels, exclude labels with the category of others, and filter labels with a box width less than 20px and a height less than 50px. Finally, use the DeepSORT model to track readers in the video and generate a unique identity.

[0044] Step 6: Evaluate the model. To verify the performance of the model, test it on the test set. Figure 3 The confusion matrix without setting the others category and setting the others category is shown.

[0045] The results show that by identifying and judging moving and still objects in video frames, it is possible to effectively solve the problem of unstable detection results in video sequences with mixed moving targets and still objects. In addition, the mAPs of YOLOv7 and CCS-YOLOv7 are 0.969 and 0.977 respectively, proving the effectiveness of the CCS module.

[0046] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for monitoring entrance and exit traffic based on deep learning, characterized by: The following steps are involved: S1. Obtain pedestrian detection data in crowded scenes, construct dataset A, and perform preprocessing; S2, retrieve the videos shot by the cameras at the entrance and exit of the monitoring area, construct data set B, and perform manual annotation and preprocessing; S3. Use the CCS-YOLOv7 model to train and verify the data set A, and update the CCS-YOLOv7 model parameters to save the pre-trained weights with the minimum loss; The loss function CIOU in the CCS-YOLOv7 model is used to compare the difference between the detection result and the true result, and the parameter gradient of the loss function with respect to the neural network is calculated, and the parameter gradient is used to update the CCS-YOLOv7 model parameters, so that the detection result of the model is close to the true result of manual annotation; CCS-YOLOv7 is a CCS module that adds a cross-channel information fusion mechanism before the SPPCSPC module of the YOLOv7 model; The CCS module design of the cross-channel information fusion mechanism is as follows: The CCS module divides the input into three branches, namely, branch 1, branch 2 and branch 3, for feature fusion. First, branch 1 uses convolution layer 10 to halve the number of channels of input x to obtain feature map Feature10. Feature10 is activated by AvgPool and Sigmoid to obtain attention vector g1. At the same time, Feature10 is converted to four times the original number of channels by convolution layer 11, and then reduced to one-fourth of the number of channels of the input by convolution layer 12 to obtain feature map Feature12. Secondly, Feature12 is multiplied by g1 element by element to obtain feature map Feature13. Finally, Feature13 is added to feature vector Feature12 through convolution layer 13 to obtain Multi Feature1. Branch 2 first uses convolution layer 20 to halve the number of channels of input x to obtain feature map Feature20. After Feature20 passes through MaxPool and Sigmoid activation functions, it obtains attention vector g2. At the same time, Feature20 passes through convolution layer 21, and the number of channels becomes four times the original. After passing through convolution layer 22, the number of channels is converted to one-fourth of the input to obtain feature map Feature22. Then Feature22 is element-wise multiplied by g2 to obtain feature map Feature23. Finally, Feature23 passes through convolution layer 23 and is added to feature vector Feature22 to obtain Multi Feature2. Multi Feature1 and Multi Feature2 are concatenated in the channel dimension to generate feature map Concat Feature. Branch 3 is the residual link, which adds the input x to the Concat Feature to get the output of the CCS module; S4, migrating the pre-trained weights to the CCS-YOLOv7 model, training and testing the data set B, and updating the CCS-YOLOv7 model parameters; S5. The CCS-YOLOv7 model is combined with the DeepSORT algorithm to build a pedestrian flow monitoring model at the entrance and exit of the monitoring area to learn and detect the specific characteristics of pedestrians at the entrance and exit of the monitoring area, thereby improving the accuracy and timeliness of the detection of the pedestrian flow monitoring model at the entrance and exit of the monitoring area; S6. Evaluate the pedestrian flow monitoring model at the entrance and exit of the monitoring area, and test the effectiveness of the pedestrian flow monitoring model at the entrance and exit of the monitoring area under real-time camera.

2. The entrance and exit traffic monitoring method based on deep learning according to claim 1 is characterized by: In S1, the preprocessing operation of the crowded scene pedestrian detection data includes: performing data enhancement on the data set A by random horizontal rotation and random vertical flipping, and dividing it into a training set and a test set.

3. The entrance and exit traffic monitoring method based on deep learning according to claim 2 is characterized by: In S2, the annotated labels include three categories: in, out and others; in represents non-staff entering, out represents non-staff leaving, and others represents staff. Through this annotation method, stationary objects in the video sequence are excluded; the preprocessing operation in S2 is the same as the preprocessing in S1.

4. The entrance and exit traffic monitoring method based on deep learning according to claim 1 is characterized in that: In S3, the torch.save() method in the pytorch toolkit is used to save the pre-trained weights with the minimum loss.

5. The entrance and exit traffic monitoring method based on deep learning according to claim 1 is characterized by: In S4, the pre-trained weights described in S3 are loaded using the torch.load() method in the pytorch toolkit, and then the data set B is trained and tested, and the CCS-YOLOv7 model parameters are updated; wherein, updating the CCS-YOLOv7 model parameters is to use CIOU to compare the difference between the monitoring results and the true results, and calculate the parameter gradient of the loss function with respect to the neural network, and use the parameter gradient to update the network parameters of the CCS-YOLOv7 model, so that the detection results of the model are close to the true results of manual annotation.

6. The entrance and exit traffic monitoring method based on deep learning according to claim 1 is characterized by: In S5, the specific steps of constructing a monitoring model for the flow of people at the entrance and exit of the monitoring area include: First, filter the labels generated by S4, exclude labels with the category of others, and filter labels with a box width less than 20px and a height less than 50px. Use the CCS-YOLOv7 model to detect readers in dataset B and generate bounding boxes and category labels. Use the DeepSORT model to track pedestrians in the video and generate unique identifiers to track pedestrians.

7. The entrance and exit traffic monitoring method based on deep learning according to claim 1 is characterized by: In S6, under the test real-time camera, the pedestrian flow monitoring model at the entrance and exit of the monitoring area is used to detect and track pedestrians in each frame of the video stream.

Images