A ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system

Abstract

The application discloses a ferry cargo vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry monitoring platform. The method comprises the following steps: collecting monitoring videos from ferry entrance cameras, frame extraction is performed on the videos, and a vehicle and pedestrian detection dataset based on ferry cargo is established; the self-constructed dataset is sent into a target detection network for training, and an optimal ferry cargo detection model is deployed on a service end to perform fact detection on ferry monitoring videos; a target tracking algorithm is used to complete real-time tracking of the same target in the monitoring videos and obtain the number of each category in the videos; the video stream after target detection and tracking is sent to the ferry monitoring platform through a socket protocol; and users can query ferry monitoring information and the number of each category of vehicles through a ferry management platform, compare the number with ferry load, so as to determine whether the ferry is overloaded.

Description

TECHNICAL FIELD

[0001] The application relates to a ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system. BACKGROUND

[0002] China's port trade is developed, and the port throughput increases year by year. The inland river shipping is also developing rapidly, and the safety of the inland river shipping is also a top priority. One of them is the vehicle and pedestrian detection at the ferry entrance. Real-time control and tracking of boarding vehicles can provide safety protection for the ferry. Taking simple vehicle control as an example, each ferry has its own carrying capacity, such as how many cars, trains and cross-river buses a ferry can carry. The present application can be used for real-time counting of boarding vehicles and pedestrians, and the boarding vehicles and pedestrians are counted one by one. When the boarding vehicles and pedestrians reach the carrying capacity of the ferry, an alarm is sent to realize the control of the vehicle and pedestrian at the ferry entrance. There are mainly two problems at present: first, the high-precision target detection algorithm has a large parameter quantity, and the model occupies a lot of memory, which cannot meet the real-time needs. Although the lightweight target detection algorithm has a high frame rate and can meet the real-time needs, the precision is not enough. The traditional anchor-based target detection algorithm lacks flexibility and produces a large parameter quantity, which is not conducive to real-time target tracking. Second, in the boarding process of vehicles and pedestrians, occlusion is easy to occur, causing the loss of some frames, so that the counting and actual vehicles and pedestrians do not match. Target tracking is needed to determine the driving route of vehicles and pedestrians to achieve the purpose of real-time control. The related technologies of target detection and target tracking are needed. SUMMARY

[0003] The purpose of the present application is to control the vehicles and pedestrians at the ferry entrance, realize target detection and tracking counting of boarding vehicles and pedestrians, and propose a ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system, and build a corresponding platform.

[0004] Technical scheme: A ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system, comprising the following steps:

[0005] (1) A ferry entrance vehicle and pedestrian monitoring video is shot by a ferry entrance monitoring camera, and the picture part of the ferry vehicle and pedestrian detection dataset is constructed through frame extraction processing. The collected pictures are labeled by labelimg to construct the final training dataset. And divide it into training set and verification set by 8:2.

[0006] (2) Build a target detection model and a target tracking model.

[0007] (3) Train the ferry entrance vehicle and pedestrian detection model, frame the video captured by the ferry monitoring camera, build the ferry vehicle and pedestrian detection dataset, and send the pictures to the target detection network in batches for training to obtain a detection model suitable for ferry vehicle and pedestrian detection.

[0008] (4) Train the ferry entrance vehicle and pedestrian tracking model. The ferry entrance vehicle tracking model includes three components: target feature re-identification, Kalman filtering and Hungarian matching. The result ensures the ID invariance of the target in the video.

[0009] (5) Start the service and deliver the ferry vehicle and pedestrian information to the ferry monitoring platform. Users can query the number of boarding vehicles and pedestrians in real time through the platform.

[0010] The ferry entrance vehicle detection network model in step (2) is an improved YOLOX-S model, which is used for target detection on incoming video; the ferry entrance vehicle and pedestrian tracking model is a deepsort model, which mainly includes feature re-extraction, Kalman filtering and Hungarian matching algorithm.

[0011] The ferry entrance vehicle detection model mainly includes the following improvements: replacing the YOLOX-S backbone network with a self-built CSP-Efficentnet to replace the original CSP-Darknet; using an improved SPP module to replace the existing SPP module; adding an attention mechanism in the enhanced feature extraction network.

[0012] The ferry entrance vehicle detection model backbone network contains five times of downsampling, and each downsampling is a large residual block containing a CSPLayer layer. The CSPLayer input feature layer is composed of a residual edge with ordinary convolution layers and a small number of residual blocks. After passing through the small residual block, the large residual edge is stacked back, and finally the channels are integrated. The number of small residual blocks in the large residual block is 1, 3, 3, and 1, respectively.

[0013] The SPP module of the ferry entrance vehicle detection and tracking model originally has three maximum pooling layers. This method uses a dilated convolution layer instead of a maximum pooling layer.

[0014] However, due to the interval value of its convolution kernel, there are two problems: loss of local information; and lack of relevance of information obtained at a distance. Regarding these two problems, this paper refers to the solution to this problem in ESPNet proposed by Sachin et al. The idea of feature layering is used to directly add different dilated convolution layers with different dilated convolution rates, that is, to add different receptive fields, thereby effectively reducing the network effect caused by dilated convolution.

[0015] The ferry entrance vehicle detection and tracking model mainly includes the following improvements: using CIOU as a loss function in data cascade matching.

[0016] The judgment criteria for the final effect of the ferry entrance vehicle and pedestrian detection model include recall rate, accuracy rate, weight file size, and mean average precision (mAP).

[0017] The ferry monitoring platform is composed of three parts: the vue framework realizes the front-end page, the springboot framework realizes the access and management of the database and the ship, and the python constructs the target tracking and target detection network. Information is transmitted between vue and springboot through the http protocol, and information is transmitted between springboot and python through socket.

[0018] The ferry detection platform includes a front-end display module, a back-end video stream service module, a target detection and tracking service module, and an external camera.

[0019] The front-end display module is used for users to call the camera and realize real-time monitoring of ferry entrance vehicles and pedestrians.

[0020] The back-end video stream service module mainly includes converting the video captured by the camera into base64 format and transmitting it to the background database, simultaneously converting it into the format required by the target detection network, calling the target detection and tracking service module, realizing ferry vehicle and pedestrian tracking, finally, parsing the output video of the network into base64 format, returning it to the ship management platform together with the counting information, and simultaneously storing it in the database.

[0021] The target detection and tracking service module is deployed as:

[0022] Build a target detection and tracking model.

[0023] Create a socket service and get the host address, and configure the corresponding port number.

[0024] Start the service.

[0025] Beneficial effects: Through experimental results, the improved target detection model has improved the original accuracy by 4.4%, the average precision has improved the original accuracy by 1.8%, and the model size has been reduced by 2M, which is sufficient to prove that the algorithm has good real-time performance. Finally, the combination of the target detection algorithm and the target tracking algorithm improves the difficulty of ferry entrance vehicle and pedestrian detection, and the platform also provides a one-stop service solution for ferry management. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1A ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system diagram;

[0027] Figure 2 A target detection network model diagram;

[0028] Figure 3 A CSP-MBConv module network structure diagram;

[0029] Figure 4 An improved SPP module network structure diagram;

[0030] Figure 5 A target tracking network model diagram;

[0031] Figure 6 A ferry management platform module diagram; DETAILED DESCRIPTION

[0032] The technical solutions of the present application will be further described below in conjunction with the accompanying drawings.

[0033] Example 1:

[0034] The present application provides a ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system, as shown in Figure 1 The following steps are included:

[0035] First, the camera at the ferry entrance needs to be called to obtain real-time vehicle and pedestrian monitoring information, frame extraction is performed from the captured video, and ferry vehicle and pedestrian information pictures are collected, the picture size is adjusted to 416*416, labeled by labelimg, and divided into four categories: car, truck, motorbike, and person. Due to the possible slight shortage of the number of collected pictures, the present application adopts data enhancement, expands the original 1500 pictures to 6000 pictures through vertical flipping, horizontal flipping, color gamut transformation, etc.

[0036] As shown in Figure 2 , a target detection network model is built. It is composed of three parts, including a backbone feature extraction network, a strengthened feature extraction network, and a detection head. The present application mainly improves the network structure of the backbone feature network and the SPP module in the strengthened feature extraction network, and combines the channel attention mechanism in the strengthened feature extraction network.

[0037] As shown in Figure 3As shown in the figure, the network structure diagram of the CSP-MBCov module is composed of CSP and MBCov modules. It consists of three parts. First, the depth-wise separable convolution is used to adjust the number of channels, adjust the length and width, and extract the local information of the input feature image; secondly, the ECA channel attention mechanism module is used to enhance the extraction of feature information. The function of this module is to give different network features different weights, which helps to grasp the global information; finally, the number of channels is adjusted through 3×3 convolution, and feature fusion is performed with the input feature map to fully extract the feature information of the image.

[0038] Its outer structure is CSPNet, a cross-stage local network that can be understood as a large residual structure. This network structure, proposed by Chien-Yao Wang et al., splits the gradient flow, propagating it through different network paths. This represents a cross-stage hierarchical structure. This approach can reduce the computational load in neural networks and achieve higher accuracy. It is easy to implement and can be easily combined with various networks, such as ResNet and DenseNet.

[0039] like Figure 4 The figure shows the improved SPP network structure. The improvements include the following aspects: removing the global pooling branch contained in the original module and reducing the four dilated convolution branches contained in the original module to three; reducing the dilated convolution rate of the branch, and selecting dilated convolution layers with dilated convolution rates of 2, 4, and 6 respectively; changing the direct merging branches of the original network to skip connections to obtain more contextual semantic information; finally, replacing the ordinary dilated convolution with a depthwise separable convolution, this operation can further reduce the number of parameters in the ASPP module. Since dilated convolution may lose local information while obtaining more contextual semantic information, the idea of ​​feature layering is used to directly add dilated convolution layers with different dilated convolution rates, that is, to add different receptive fields, thereby effectively reducing the network effect caused by dilation.

[0040] The ECA module is also applied to the enhanced feature extraction network, and through channel squeezing and excitation, the result of efficient feature extraction is finally achieved.

[0041] The detection head used in the application is a Decoupled Head detection head, which is proposed in the YOLOX target detection network released by PatSnap in 2021. Unlike previous detection heads, the classification and regression in previous detection heads are implemented in a 1x1 convolution, while Decoupled Head is different. Decoupled Head believes that this will adversely affect identification, so in this target detection head, the detection head is divided into two parts, which are implemented separately and then combined together.

[0042] The loss function used in target detection is CIOU-Loss loss function, which is a penalty parameter applied on the basis of IOU-Loss. The loss function formula is as follows:

[0043]

[0044] Where b, b gt represent the center points between the predicted frame and the actual frame, and p represents the Euclidean distance between the two center points. c represents, which can contain the diagonal distance between the predicted frame and the actual frame. The formulas of alpha and v are shown in equations (2) and (3):

[0045]

[0046]

[0047] The IOU loss function calculation formula is shown in equation (4):

[0048]

[0049] In summary, due to the addition of the concept of v, the overlap information of the two frames calculated by CIOU-Loss is more accurate, and it is beneficial to back propagation.

[0050] The target tracking part flow chart is shown in Figure 5 It introduces a deep learning model and extracts target features for matching in real-time target tracking. It is divided into two stages. The first stage is the Kalman filter stage, which is mainly used for motion trajectory prediction. The second stage is the Hungarian matching stage, which is mainly used for data association of predicted data and detection data, so as to assign a corresponding target ID to each object.

[0051] Use self-made data set as input to train target detection network and target tracking network, train corresponding target detection and target tracking model, finally, deploy the model on the corresponding network port of the server, realize the inter-process communication through the socket protocol and the ferry management platform.

[0052] Embodiment 2:

[0053] The application discloses a ferry vehicle and pedestrian detection and tracking method based on an improved YOLOX algorithm and a ferry detection system, Figure 6 The platform concept design diagram is a ferry monitoring platform mainly composed of a ferry entrance pedestrian and vehicle detection and tracking module, a safety alarm module, a ship dispatch management module, a weather management module and a map module, and roles thereof are divided into on-ship personnel and on-shore administrators.

[0054] The ferry entrance pedestrian and vehicle detection and tracking module is embedded with a deep learning neural network model, is deployed at a corresponding network port of a server through construction of a socket service, and a background service of the ferry management platform is used to realize interaction with the constructed target detection and tracking service. Monitoring video is transmitted in a Base64 mode, the target detection and tracking service is used to analyze the video stream transmitted into the service, and detection and tracking are realized. The video with completed detection and tracking counting is fed back to the ferry management platform in a video stream mode, and is finally displayed on a UI interface of a user end.

[0055] The safety alarm module is composed of three parts. The first part is an on-ship alarm page, which comprises a marine map and an alarm button. The second part is an on-shore early warning receiving page, which comprises a report of possible early warning types on the ferry. The third part is a service agent. A communication protocol thereof is an MQTT protocol. Firstly, a topic of an MQTT transmission message is defined, such as ship body fire, person falling into water and grounding. An on-shore administrator subscribes to a message of a corresponding topic through the early warning receiving page. When a corresponding event occurs on the ship, an on-ship personnel publishes a message through an early warning button. The on-shore administrator can receive a push of the corresponding message.

[0056] The ship dispatch management module is completed through a database management system. When a ferry is put into use, an administrator inputs information of the ferry into the ferry management platform. Fields include a ship id, a ship type, a ship name, a company where the ferry is located, a ship captain, whether the ship is dispatched, a starting point, a terminal point, a departure time, an expected arrival time, an actual arrival time, whether the ship arrives, and whether there is an early warning message.

[0057] When a ferry is dispatched, firstly, the ferry monitoring platform sends a ship departure instruction. The instruction is transmitted to a background service management program. The background management program changes whether the ship is dispatched in the database to yes, and fills in the starting point, the terminal point and other information of the ship navigation, and the expected arrival time. If an early warning is sent during the navigation, the ferry management platform can also modify the whether the ship is early warned field through a corresponding service.

[0058] When the ferry arrives at the destination, the corresponding actual arrival time is filled in, and the whether the ship is dispatched field is changed to no.

[0059] On this basis, the ferry monitoring platform is also equipped with corresponding schedule system, weather detection system, etc., realizing integrated management of ferry navigation.

[0060] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely divergences, modifications, replacements and variations of the embodiments of the application, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. A YOLOX-based method for detecting and tracking pedestrians on ferry cargo vehicles, characterized in that: The following steps are involved: (1) Use the ferry camera to obtain video of vehicles and pedestrians boarding the ship; (2) Build a vehicle and pedestrian detection model at the ferry entrance and a vehicle and pedestrian tracking model at the ferry crossing; (3) Train the ferry entrance vehicle detection model. Extract frames from the video captured by the ferry surveillance camera to construct a ferry vehicle and pedestrian detection dataset. Then, feed the images in batches into the target detection network for training, and obtain a detection model suitable for ferry vehicle and pedestrian detection. (4) Training the ferry entrance vehicle tracking model; the ferry entrance vehicle tracking model consists of three components: re-identification of target features in the input video, Kalman filtering and Hungarian matching, and the result is to ensure the invariance of the ID of the target in the video; (5) Start the service and transmit the information of vehicles and pedestrians on the ferry to the ferry monitoring platform, through which users can query the number of vehicles and pedestrians on board in real time; The ferry entrance vehicle detection network model in (2) is an improved YOLOX-S model, which is used to detect targets in the incoming video; the ferry entrance vehicle and pedestrian tracking model is a deepsort model, which includes feature re-extraction, Kalman filtering and Hungarian matching algorithm; The ferry entrance vehicle detection network model includes the SP-MBCov module, which is composed of the CSP combined with the MBCov module. It consists of three parts. First, the number of channels is adjusted by using depthwise separable convolution, adjusting the length and width to extract local information of the input feature image. Second, the ECA channel attention mechanism module is used to enhance the extraction of feature information. This module is used to give different network features different weights to help grasp global information. Finally, the number of channels is adjusted by 3×3 convolution, and feature fusion is performed with the input feature map to fully extract the feature information of the image. The ferry entrance vehicle detection network model replaces the existing SPP module with an improved SPP module, including: removing the global pooling branch contained in the original module and reducing the four dilated convolution branches contained in the original module to three; reducing the size of the branch dilated convolution rate and selecting dilated convolution layers with dilated convolution rates of 2, 4, and 6 respectively; changing the direct merging branches of the original network to skip connections to obtain more contextual semantic information; and finally, replacing the ordinary dilated convolution with depthwise separable convolution. This operation can further reduce the number of parameters in the ASPP module.

2. The method for detecting and tracking pedestrians on a ferry cargo vehicle according to claim 1, characterized in that: The ferry entrance vehicle detection and tracking model includes: using CIOU as a loss function in data cascade matching.

3. The method for detecting and tracking pedestrians on a ferry cargo vehicle according to claim 1, characterized in that: The final effect of the ferry entrance vehicle and pedestrian detection model is judged by the following criteria: recall rate, accuracy rate, weight file size, and average precision index.

4. The method for detecting and tracking pedestrians on a ferry cargo vehicle according to claim 1, characterized in that: The ferry detection platform consists of three parts: the Vue framework implements the front-end page, the Springboot framework implements database access and ship management, and Python builds the target tracking and target detection network; information is transmitted between Vue and Springboot through the HTTP protocol, and information is transmitted between Springboot and Python through socket.

5. A ferry cargo vehicle pedestrian detection and tracking platform, implementing a ferry cargo vehicle pedestrian detection and tracking method based on YOLOX as described in any one of claims 1-4, characterized in that: Including front-end display module, back-end video stream acquisition service module, target detection and tracking service module, camera; The front-end display module is used by the user to call the camera to achieve real-time monitoring of vehicles and pedestrians at the ferry entrance; The back-end video stream acquisition service module mainly includes converting the camera video into base64 format and transmitting it to the back-end database. It also converts it into the format required by the target detection network, calls the target detection and tracking service module to realize the tracking of vehicles and pedestrians at the ferry. Finally, the network output video is parsed into base64 format, returned to the ship management platform together with the counting information, and stored in the database at the same time. The target detection and tracking service module is deployed as: Build target detection and tracking models; Create a socker service, obtain the host address, and configure the corresponding port number; Start the service.

6. The ferry cargo vehicle pedestrian detection and tracking platform according to claim 5 is characterized in that: By connecting an external surveillance camera to the PC, the camera's video of vehicles and pedestrians at the ferry crossing is streamed to the Spring Boot backend service. The target detection and tracking service is called to perform real-time detection, tracking, and counting. The detection results are then returned to the Spring Boot backend service, which then feeds the information back to the front-end user. Step 5: Collection: After drying, the rapeseed seeds can be collected.

Images