A traffic flow statistical method based on reinforcement learning

Abstract

The application relates to a traffic flow statistical method based on reinforcement learning, which comprises the following steps: collecting video data of a traffic scene to obtain initial video samples; performing pretreatment; inputting the pretreated video images into a YOLOv8 network model to perform target detection; tracking the vehicles in the pretreated video images detected by using an improved SORT algorithm; combining a DDPG model to optimize a target tracking strategy and obtain a target tracking model; and inputting the traffic scene video data to be counted into the target tracking model, so that the target tracking model outputs statistical results. The YOLOv8 model is used for target detection, the vehicles in the traffic scene can be quickly and accurately detected, the SORT algorithm is improved, the accuracy of target tracking is improved, the reinforcement learning technology, namely the DDPG model, is additionally introduced, the characteristics of vehicle driving are learned, target prediction is performed for each position, and the prediction accuracy of the target is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a traffic flow statistics method based on reinforcement learning. Background Technology

[0002] The rapid development of traffic flow statistics is primarily driven by the advancements in intelligent transportation and urbanization. The efficiency and accuracy of traffic management benefit from the widespread application of intelligent transportation technologies, while accelerated urbanization leads to a continuous increase in traffic data, further enhancing the demand for and importance of traffic flow statistics. Traffic flow statistics play a crucial role in public transportation management, providing essential data for traffic planning, road network optimization, traffic signal control, and traffic safety management.

[0003] Currently, traditional traffic flow statistics algorithms mainly include optical flow, inter-frame difference, and background difference. Optical flow is particularly sensitive to changes in illumination and background; significant variations in ambient lighting conditions can severely impact the calculated optical flow field distribution. When the moving target is similar to the background or is occluded, optical flow struggles to accurately extract the target's shape, potentially leading to inaccurate target tracking. Inter-frame difference, when the moving target has a uniform color, primarily differentiates adjacent frames along the target's direction of motion, resulting in small differences within the target's internal regions, creating holes and hindering the acquisition of a complete and accurate target contour. Furthermore, inter-frame difference is highly sensitive to environmental noise, making threshold selection crucial; too low a threshold is insufficient to suppress noise, while too high a threshold may ignore useful changes. Background difference requires sophisticated background modeling, necessitating a background image that is updated in real-time with changes in illumination or the external environment. Background model updates and disturbances can negatively impact moving target detection. When the background and moving target are similar, or when there is occlusion, some moving targets may be misidentified as background, or some background may be misidentified as moving targets, resulting in inaccurate detection results. Summary of the Invention

[0004] To address the problem that traditional traffic flow statistics systems easily lose detection targets and produce inaccurate results due to occlusion issues, the present invention aims to provide a reinforcement learning-based traffic flow statistics method that can quickly and accurately detect vehicles in traffic scenes, improve the accuracy of target tracking, and enhance the accuracy of target prediction.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a traffic flow statistics method based on reinforcement learning, which includes the following sequential steps:

[0006] (1) Collect video data of traffic scenes to obtain initial video samples;

[0007] (2) Preprocess the video images in the initial video samples to obtain the preprocessed video images;

[0008] (3) Input the preprocessed video image into the YOLOv8 network model to perform target detection on the preprocessed video image and detect the vehicles in the preprocessed video image;

[0009] (4) The improved SORT algorithm is used to track vehicles in the detected preprocessed video images. The target's current position is predicted based on its historical motion trajectory and matched with the target detected in the current frame to achieve continuous tracking of the target.

[0010] (5) Based on step (4), combine the DDPG model to optimize the target tracking strategy, select appropriate actions according to the current environmental state, and obtain the target tracking model;

[0011] (6) Input the traffic scene video data to be statistically analyzed into the target tracking model, and the target tracking model outputs the statistical results.

[0012] Step (1) specifically refers to: collecting video data in traffic scenarios using a camera or existing traffic monitoring cameras.

[0013] In step (2), the preprocessing includes noise removal, camera shake reduction, image size and format adjustment; and setting a detection line or detection area in the video for traffic flow statistics.

[0014] In step (4), the improved SORT algorithm specifically refers to: during the tracking process, saving trajectory objects and performing cascade matching and IOU matching: for confirmed trajectory objects, the cascade matching method is used, and the cosine similarity cost function is used for matching, with higher priority; unmatched trajectory objects, unmatched detection objects, and unconfirmed trajectory objects are all matched together using IOU matching; IOU matching performs IOU matching between detection objects and trajectory objects, and outputs the matching result;

[0015] Based on the matching results, the detected objects are classified into successfully matched objects, unmatched detected objects, and unmatched trajectory objects. For successfully matched objects, the parameters of the trajectory objects are updated using a Kalman filter to improve the accuracy and stability of tracking. After three consecutive matches, the object is confirmed. For unmatched detected objects, new trajectory objects are initialized, with one trajectory object corresponding to each detected object for the next object matching. For unmatched trajectory objects, if their status is not confirmed (i.e., they have not undergone three consecutive matches), they are deleted. If their status is confirmed (i.e., they have undergone three consecutive matches), whether to delete them is determined based on the set disappearance frame limit. If the disappearance frame limit is exceeded, they are deleted; if the disappearance frame limit is not exceeded, they are retained.

[0016] In the cascaded matching stage, in addition to location information, the appearance feature vector of the detected object is matched with all appearance feature vectors of the tracked object since tracking began. The feature vector with the highest similarity is selected as the cost matrix, and then the Hungarian algorithm is used for matching.

[0017] Step (5) specifically includes the following steps:

[0018] (5a) Define the state space as a six-dimensional vector {x,y,vx,vy,ax,ay}, and define the coordinates x,y, velocity vector v = {vx,vy} and acceleration vector a = {ax,ay} respectively. Each set of data represents the feature state of an object in the detection map.

[0019] (5b) Define the action space as a continuous direction vector {dx,dy}, which indicates the direction in which the object will actively choose to move in the next state.

[0020] (5c) During the Kalman filter update process, the matched trajectory objects and detection objects are obtained, and DDPG prediction sampling is performed on 300 randomly selected directions; the DDPG model includes an Actor network, a Critic network, a replay buffer, and a DDPG master control. The Actor network and the Critic network adopt a fully connected neural network with two hidden layers. The two hidden layers refer to the first hidden layer and the second hidden layer. The first hidden layer has 400 nodes, and the second hidden layer has 300 nodes. The replay buffer is used as an experience cache to store each data point and provide an interface for batch reading of data.

[0021] The DDPG master controller is responsible for model initialization, reward function, model storage and updates. After model initialization, it creates the Actor and Critic networks and initializes the replay buffer, while defining hyperparameters for model updates. Model updates include Actor and Critic network updates. The model reads data from the replay buffer multiple times in batches, then uses the target Critic network to obtain the target Q-value. This target Q-value, along with the action, is then fed into the Critic network to obtain its loss value. The Critic network is then updated using gradient descent backpropagation. The updated Critic network is then used to calculate the Actor network's loss value, and gradient descent backpropagation is used to update the Actor network again. Finally, the Actor and Critic network parameters are softly updated to the corresponding target Actor and Critic networks using a moving average method.

[0022] (5d) For each DDPG prediction result, the reward is calculated by combining the Kalman filter prediction result. The reward function consists of two parts: the direction information reward function R1 and the distance information reward function R2.

[0023] For directional information, the cosine of the angle between the directions of the two vectors is used as the reward function for DDPG. The specific formula is as follows:

[0024]

[0025] Where d is the direction vector given by the agent, d′ is the direction vector measured by the Kalman filter, and R1 ranges from [-1,1], where -1 represents the worst predicted direction and 1 represents the best predicted direction.

[0026] For distance information, Euclidean distance is used, and an exponential function is employed to map the range of R² to [0,1]. The specific formula is as follows:

[0027]

[0028] Where x,y are the positions predicted by the direction vector given by the agent, x′,y′ are the positions obtained by the Kalman filter, and the range of R2 is (0,1], where 0 represents infinite distance and 1 represents closest distance;

[0029] Therefore, the total reward function R is defined as follows:

[0030]

[0031] When the Kalman filter tracks well, i.e., achieves three accurate matches, the total reward function R is defined as the sum of R1 and R2. However, when the object is occluded or the Kalman filter is not accurate enough, R = 0. In this case, there is no data available for learning, so no reward is provided.

[0032] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention uses the YOLOv8 model for target detection, which can quickly and accurately detect vehicles in traffic scenes; Second, the present invention improves the SORT algorithm by introducing feature recognition, cascade matching, and target re-identification technologies, thereby improving the accuracy of target tracking. In particular, the cascade matching stage uses a cosine similarity cost function and combines it with the appearance feature vector of the target for matching, making the tracked object more distinguishable and reducing the target ID switching problem caused by occlusion or detection failure; Third, the present invention additionally introduces reinforcement learning technology, namely the DDPG model, which learns the features of the vehicle while it is moving and performs target prediction for each position, effectively improving the prediction accuracy of the target. Attached Figure Description

[0033] Figure 1 , 3 All of these are flowcharts of the method of the present invention;

[0034] Figure 2 This is a flowchart of the improved SORT algorithm in this invention;

[0035] Figure 4 The logical diagram of the DDPG model;

[0036] Figure 5 , 6 7 represents the traffic flow statistics principle diagram, system settings interface, and system operation interface, respectively. Detailed Implementation

[0037] like Figure 1 , Figure 3 As shown, a traffic flow statistics method based on reinforcement learning includes the following sequential steps:

[0038] (1) Collect video data of traffic scenes to obtain initial video samples;

[0039] (2) Preprocess the video images in the initial video samples to obtain the preprocessed video images;

[0040] (3) Input the preprocessed video image into the YOLOv8 network model to perform target detection on the preprocessed video image and detect the vehicles in the preprocessed video image;

[0041] (4) The improved SORT algorithm is used to track vehicles in the detected preprocessed video images. The target's current position is predicted based on its historical motion trajectory and matched with the target detected in the current frame to achieve continuous tracking of the target.

[0042] (5) Based on step (4), combine the DDPG model to optimize the target tracking strategy, select appropriate actions according to the current environmental state, and obtain the target tracking model;

[0043] (6) Input the traffic scene video data to be statistically analyzed into the target tracking model, and the target tracking model outputs the statistical results.

[0044] Step (1) specifically refers to: collecting video data in traffic scenarios using a camera or existing traffic monitoring cameras.

[0045] In step (2), the preprocessing includes noise removal, camera shake reduction, and image size and format adjustment; a detection line or detection area is set in the video for traffic flow statistics. The system provides a customizable detection line, and when the bottom midpoint of a vehicle's tracking frame passes through the detection line, the system will count the traffic flow based on its direction.

[0046] like Figure 2 As shown, in step (4), the improved SORT algorithm specifically refers to: during the tracking process, saving the trajectory objects and performing cascade matching and IOU matching: for confirmed trajectory objects, the cascade matching method is used, and the cosine similarity cost function is used for matching, with higher priority; unmatched trajectory objects, unmatched detection objects, and unconfirmed trajectory objects are all matched together using IOU matching; IOU matching performs IOU matching between detection objects and trajectory objects, and outputs the matching result;

[0047] Based on the matching results, the detected objects are classified into successfully matched objects, unmatched detected objects, and unmatched trajectory objects. For successfully matched objects, the parameters of the trajectory objects are updated using a Kalman filter to improve the accuracy and stability of tracking. After three consecutive matches, the object is confirmed. For unmatched detected objects, new trajectory objects are initialized, with one trajectory object corresponding to each detected object for the next object matching. For unmatched trajectory objects, if their status is not confirmed (i.e., they have not undergone three consecutive matches), they are deleted. If their status is confirmed (i.e., they have undergone three consecutive matches), whether to delete them is determined based on the set disappearance frame limit. If the disappearance frame limit is exceeded, they are deleted; if the disappearance frame limit is not exceeded, they are retained.

[0048] In the cascaded matching stage, in addition to location information, the appearance feature vector of the detected object is matched with all appearance feature vectors of the tracked object since tracking began. The feature vector with the highest similarity is selected as the cost matrix, and then the Hungarian algorithm is used for matching.

[0049] like Figure 4 As shown, step (5) specifically includes the following steps:

[0050] (5a) Define the state space as a six-dimensional vector {x,y,vx,vy,ax,ay}, and define the coordinates x,y, velocity vector v = {vx,vy} and acceleration vector a = {ax,ay} respectively. Each set of data represents the feature state of an object in the detection map.

[0051] (5b) Define the action space as a continuous direction vector {dx,dy}, which indicates the direction in which the object will actively choose to move in the next state.

[0052] (5c) During the Kalman filter update process, the matched trajectory objects and detection objects are obtained, and DDPG prediction sampling is performed on 300 randomly selected directions; the DDPG model includes an Actor network, a Critic network, a replay buffer, and a DDPG master control. The Actor network and the Critic network adopt a fully connected neural network with two hidden layers. The two hidden layers refer to the first hidden layer and the second hidden layer. The first hidden layer has 400 nodes, and the second hidden layer has 300 nodes. The replay buffer is used as an experience cache to store each data point and provide an interface for batch reading of data.

[0053] The DDPG master controller is responsible for model initialization, reward function, model storage and updates. After model initialization, it creates the Actor and Critic networks and initializes the replay buffer, while defining hyperparameters for model updates. Model updates include Actor and Critic network updates. The model reads data from the replay buffer multiple times in batches, then uses the target Critic network to obtain the target Q-value. This target Q-value, along with the action, is then fed into the Critic network to obtain its loss value. The Critic network is then updated using gradient descent backpropagation. The updated Critic network is then used to calculate the Actor network's loss value, and gradient descent backpropagation is used to update the Actor network again. Finally, the Actor and Critic network parameters are softly updated to the corresponding target Actor and Critic networks using a moving average method.

[0054] (5d) For each DDPG prediction result, the reward is calculated by combining the Kalman filter prediction result. The reward function consists of two parts: the direction information reward function R1 and the distance information reward function R2.

[0055] For directional information, the cosine of the angle between the directions of the two vectors is used as the reward function for DDPG. The specific formula is as follows:

[0056]

[0057] Where d is the direction vector given by the agent, d′ is the direction vector measured by the Kalman filter, and R1 ranges from [-1,1], where -1 represents the worst predicted direction and 1 represents the best predicted direction.

[0058] For distance information, Euclidean distance is used, and an exponential function is employed to map the range of R² to [0,1]. The specific formula is as follows:

[0059]

[0060] Where x,y are the positions predicted by the direction vector given by the agent, x′,y′ are the positions obtained by the Kalman filter, and the range of R2 is (0,1], where 0 represents infinite distance and 1 represents closest distance;

[0061] Therefore, the total reward function R is defined as follows:

[0062]

[0063] When the Kalman filter tracks well, i.e., achieves three accurate matches, the total reward function R is defined as the sum of R1 and R2. However, when the object is occluded or the Kalman filter is not accurate enough, R = 0. In this case, there is no data available for learning, so no reward is provided.

[0064] like Figure 5 As shown, to accurately count traffic flow in a video, a detection line or detection area must be set up. In this invention, the system provides a customizable detection line. When the bottom midpoint of a vehicle's tracking frame crosses the detection line, the system counts the traffic flow based on the vehicle's direction of travel. If the vehicle is traveling away from the camera, the system records it as leaving; if the vehicle is traveling towards the camera, the system records it as entering.

[0065] like Figure 6 As shown, the system displays the first frame of the video on the interface, allowing users to easily draw detection lines. The green detection lines are straight lines used by the system to identify the vehicle's direction of travel. When the bottom center point of the tracked object crosses the detection line, the system performs statistical analysis based on the vehicle's direction of travel and displays the corresponding information on the image.

[0066] like Figure 7 As shown, the system marks the location, ID, and trajectory information of each vehicle. Whenever a vehicle passes the detection line, the system performs statistics and displays the results in the upper left and upper right corners.

[0067] The effectiveness of this invention can be illustrated by the following simulation experiments:

[0068] 1. Experimental conditions

[0069] The hardware environment included a Windows 11 operating system, an NVIDIA RTX 4060 GPU, and CUDA version 12.1. The software environment used Python 3.12 programming language, developed in Visual Studio Code, with PyTorch as the deep learning framework and PyQt5 as the graphical user interface library. For data sources, object detection data was obtained from online video streams using the YOLOv8l model, while reinforcement learning training data consisted of surveillance videos of approximate road sections to ensure the usability of the DDPG model. These experimental conditions provided a robust experimental environment for the stability and reliability of the algorithm in different scenarios.

[0070] 2. Experimental steps: The specific experimental steps are as follows:

[0071] (1) Select the YOLOv8l model, input the collected video data into the YOLOv8l model, and obtain the target detection results;

[0072] (2) Based on the reinforcement learning algorithm, namely the DDPG model, the optimization method is set as the AdamW optimization method, the initial learning rate is 0.001, 300 batches are trained, and the cosine decay method is adopted.

[0073] (3) Save the best model parameters during training for use in subsequent tests;

[0074] (4) Load the model parameters saved in step (3), input the video data to be predicted into the reinforcement learning model, i.e., the DDPG model, for testing, and obtain the tracking results;

[0075] (5) Output the tracking results of the DDPG model and analyze them in comparison with the actual situation.

[0076] 3. Experimental Results

[0077] By combining the YOLOv8l model and the DDPG reinforcement learning algorithm, multi-target vehicle tracking in traffic videos was successfully achieved. During the testing phase, good tracking performance was observed, maintaining ID stability even when vehicle features changed, and demonstrating good tracking performance while vehicles were in motion. Furthermore, the system can accurately count the number of vehicles crossing detection lines, realizing traffic flow statistics. Overall, the experimental results show that the algorithm can effectively meet the needs of multi-target tracking and traffic flow statistics, achieving the designed objectives.

[0078] In summary, this invention employs the YOLOv8 model for target detection, achieving rapid and accurate vehicle detection in traffic scenarios. By introducing feature recognition, cascaded matching, and target re-identification techniques, the SORT algorithm is improved, enhancing the accuracy of target tracking. Particularly in the cascaded matching stage, the use of a cosine similarity cost function and the target's appearance feature vector effectively reduces the target ID switching problem. Furthermore, the introduction of reinforcement learning techniques, through learning vehicle driving characteristics, enables accurate target prediction, further improving prediction accuracy.

[0079] The foregoing description introduces the basic principles and specific embodiments of the present invention. It should be noted that the embodiments of the present invention are not limited to the above-described examples. Any modifications, equivalent substitutions, or other changes made without departing from the scope of the principles of the present invention should be included within the protection scope of the present invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A traffic flow statistics method based on reinforcement learning, characterized in that: The method includes the following steps in sequence: (1) Collect video data of traffic scenes to obtain initial video samples; (2) Preprocess the video images in the initial video samples to obtain the preprocessed video images; (3) Input the preprocessed video image into the YOLOv8 network model to perform target detection on the preprocessed video image and detect the vehicles in the preprocessed video image; (4) The improved SORT algorithm is used to track the vehicles in the detected preprocessed video images. The current position of the target is predicted based on the historical motion trajectory of the target and matched with the target detected in the current frame to achieve continuous tracking of the target. (5) Based on step (4), combine the DDPG model to optimize the target tracking strategy, select appropriate actions according to the current environmental state, and obtain the target tracking model; (6) Input the traffic scene video data to be statistically analyzed into the target tracking model, and the target tracking model outputs the statistical results; In step (4), the improved SORT algorithm specifically refers to: during the tracking process, saving trajectory objects and performing cascade matching and IOU matching: for confirmed trajectory objects, the cascade matching method is used, and the cosine similarity cost function is used for matching, with higher priority; unmatched trajectory objects, unmatched detection objects, and unconfirmed trajectory objects are matched together using IOU matching. IOU matching performs IOU matching between the detected object and the trajectory object, and outputs the matching result; Based on the matching results, the detected objects are classified into successfully matched objects, unmatched detected objects, and unmatched trajectory objects. For successfully matched objects, the parameters of the trajectory objects are updated using a Kalman filter to improve the accuracy and stability of tracking. After three consecutive matches, the object is confirmed. For unmatched detected objects, new trajectory objects are initialized, with one trajectory object corresponding to each detected object for the next object matching. For unmatched trajectory objects, if their status is not confirmed (i.e., they have not undergone three consecutive matches), they are deleted. If their status is confirmed (i.e., they have undergone three consecutive matches), whether to delete them is determined based on the set disappearance frame limit. If the disappearance frame limit is exceeded, they are deleted; if the disappearance frame limit is not exceeded, they are retained. In the cascaded matching stage, in addition to location information, the appearance feature vector of the detected object is matched with all appearance feature vectors of the tracked object since tracking began. The feature vector with the highest similarity is selected as the cost matrix, and then the Hungarian algorithm is used for matching. Step (5) specifically includes the following steps: (5a) Define the state space as a six-dimensional vector , defining the coordinates , the velocity vector and the acceleration vector , each set of data representing the state of a feature in the detection map; (5b) Define the action space as a continuous direction vector. This direction vector indicates the direction in which the object will actively choose to move in the next state; (5c) During the Kalman filter update process, the matched trajectory objects and detection objects are obtained, and DDPG prediction sampling is performed on 300 randomly selected directions; the DDPG model includes an Actor network, a Critic network, a replay buffer, and a DDPG master control. The Actor network and the Critic network adopt a fully connected neural network with two hidden layers. The two hidden layers refer to the first hidden layer and the second hidden layer. The first hidden layer has 400 nodes, and the second hidden layer has 300 nodes. The replay buffer is used as an experience cache to store each data point and provide an interface for batch reading of data. The DDPG master controller is responsible for model initialization, reward function, model storage and updates. After model initialization, it creates the Actor and Critic networks and initializes the replay buffer, while defining hyperparameters for model updates. Model updates include Actor and Critic network updates. The model reads data from the replay buffer multiple times in batches, then uses the target Critic network to obtain the target Q-value. This target Q-value, along with the action, is then fed into the Critic network to obtain its loss value. The Critic network is then updated using gradient descent backpropagation. The updated Critic network is then used to calculate the Actor network's loss value, and gradient descent backpropagation is used to update the Actor network again. Finally, the Actor and Critic network parameters are softly updated to the corresponding target Actor and Critic networks using a moving average method. (5d) For each DDPG prediction result, the reward is calculated by combining the Kalman filter prediction result. The reward function consists of two parts: the directional information reward function. and distance information reward function ; For directional information, the cosine of the angle between the directions of the two vectors is used as the reward function for DDPG. The specific formula is as follows: ， in, The direction vector given to the agent. The direction vector is measured by the Kalman filter. The range is ,in, 1 represents the worst prediction direction, and 1 represents the best prediction direction; For distance information, Euclidean distance is used, and an exponential function is employed to... Range mapping to The specific formula is as follows: ， in, The predicted position given the direction vector of the agent. The position obtained by the Kalman filter. The range is Where 0 represents infinite distance and 1 represents closest distance; Therefore, the total reward function R is defined as follows: ， When the Kalman filter tracks well, i.e., achieves three accurate matches, the total reward function R is defined as follows: and The sum, but when the object is occluded or the Kalman filter is not accurate enough, At this point, there is no data available for learning, so no reward is offered.

2. The traffic flow statistics method based on reinforcement learning according to claim 1, characterized in that: Step (1) specifically refers to: collecting video data in traffic scenarios using a camera or existing traffic monitoring cameras.

3. The traffic flow statistics method based on reinforcement learning according to claim 1, characterized in that: In step (2), the preprocessing includes noise removal, camera shake reduction, image size and format adjustment; and setting a detection line or detection area in the video for traffic flow statistics.

Images