Traffic flow statistics and congestion analysis method and system based on video analysis

Abstract

The invention discloses a traffic flow statistics and congestion analysis method and system based on video analysis, and relates to the technical field of machine vision, and the method comprises the steps: obtaining a real-time video stream of a roadside camera; the video stream is monitored and maintained, and abnormity is reported when a picture is changed; performing data processing on the video stream to obtain processed data; performing target detection and tracking on the processed data to obtain detection and tracking results; performing image weather classification on the processed data to obtain a weather classification result; working condition division and parameter adjustment are carried out in combination with detection and tracking results and weather classification results; respectively judging whether the detection and tracking results pass through a starting line and an ending line, and respectively obtaining statistical results; respectively carrying out travel speed calculation, vehicle type classification and vehicle lane affiliation on the statistical result of passing through the end line, and recording information; and carrying out congestion degree analysis and flow statistics on each vehicle model according to the recorded information, and carrying out data reporting.

Description

Technical Field

[0001] The present invention belongs to the technical fields of traffic flow survey, congestion analysis and machine vision, and in particular relates to a traffic flow statistics and congestion analysis method and system based on video analysis. Background Art

[0002] At present, the monitoring and investigation of highway traffic conditions mainly rely on ground sensor coil technology. However, this traditional method has some significant defects in practical applications. First, the construction cost of ground sensor coils is high. Not only does it require a professional construction team to install it, but it also adds an additional economic burden on later maintenance. Secondly, ground sensor coils are easily damaged by external environmental influences, such as climate change, road construction, or oppression by heavy vehicles, which reduces the reliability of the system. In addition to ground sensor coils, some new types of traffic monitoring equipment have begun to emerge on the market, such as radar-based cameras and laser cameras. These devices can achieve more accurate traffic flow statistics and can effectively identify different types of vehicles. However, the introduction of these new devices is often accompanied by high installation costs, and additional cameras need to be installed on the basis of existing facilities. The implementation cost of large-scale replacement of existing equipment is also high. Therefore, in the current traffic monitoring technology, how to improve the accuracy and economy of monitoring is still a challenge that needs to be solved.

[0003] Current traffic monitoring technology mainly relies on ground sensor coils and new camera equipment, but both methods have significant defects and challenges. First, traditional ground sensor coil technology has high construction and maintenance costs. The installation of ground sensor coils requires a professional team to carry out complex excavation and laying, which not only increases the initial economic investment, but also may cause certain interference to traffic. In daily use, ground sensor coils are easily damaged by the external environment, such as weather changes and the passage of heavy vehicles, making their maintenance costs higher and bringing a continuous economic burden to managers. On the other hand, although the radar-based integrated cameras and laser cameras that have emerged in recent years have more advantages in monitoring accuracy, their procurement costs and subsequent installation costs cannot be ignored. The introduction of new equipment requires additional installation work and may involve the full replacement of existing monitoring facilities, which means huge implementation costs and is difficult to achieve widespread deployment in many cases. Therefore, current traffic monitoring technology urgently needs a more economical and efficient solution to overcome the limitations of existing methods. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a traffic flow statistics and congestion analysis method and system based on video analysis. Through a pure visual solution based on a convolutional neural network, effective traffic flow monitoring can be achieved using existing roadside cameras without adding additional cameras.

[0005] On the one hand, to achieve the above-mentioned purpose, the present invention provides a traffic flow statistics and congestion analysis method based on video analysis, comprising:

[0006] S1, initialize the start line, end line and lane line, and obtain the real-time video stream of the roadside camera;

[0007] S2. Monitor and maintain the video stream, and report anomalies if the image changes and / or the analysis program cannot process normally;

[0008] S3, performing data processing on the video stream to obtain processed data;

[0009] S4, performing target detection and tracking on the processed data to obtain detection and tracking results;

[0010] S5, performing image weather classification on the processed data to obtain a weather classification result;

[0011] S6. Combine the detection and tracking results and the weather classification results to divide the working conditions and adjust the parameters, and return to S3;

[0012] S7, judging whether the detection and tracking results have passed through the starting line and the ending line, respectively, and obtaining statistical results respectively;

[0013] S8, calculating the travel speed, classifying the vehicle type and the lane affiliation of the vehicle based on the statistical results of the vehicles passing the end line, and recording the information, and returning to S3;

[0014] S9. Analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

[0015] Optionally, determining whether the detection and tracking results have passed through a starting line and an ending line, and obtaining statistical results respectively include:

[0016] ;

[0017] ;

[0018] in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

[0019] Optionally, calculating the travel speed of each vehicle for the statistical results of the vehicles passing the end line includes:

[0020] ;

[0021] in, Indicates the travel speed of the current road; n indicates the total number of lanes; represents the average speed of the i-th lane; is the congestion coefficient.

[0022] Optionally, classifying the vehicle types according to the statistical results of vehicles passing the end line includes:

[0023] Cut the vehicle area according to the detection frame;

[0024] Generate a zero-filled mask image of the same size as the cut image;

[0025] Concatenate the cut image and the mask image to obtain data of several channels;

[0026] A classification neural network is used to classify vehicle models based on data from several channels.

[0027] Optionally, the vehicle lane assignment for the statistical results of the vehicles passing through the end line includes:

[0028] ;

[0029] in, It means that when y= The x-coordinate of the quadratic equation of the lane line when represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

[0030] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a traffic flow statistics and congestion analysis system based on video analysis, comprising:

[0031] Data acquisition module, data monitoring and maintenance module, data processing module, target detection and tracking module, image weather classification module, working condition division and parameter adjustment module, start line and end line judgment module, lane position judgment and vehicle type recognition module, result recording module;

[0032] The data acquisition module is used to initialize the start line, end line and lane line, and acquire the real-time video stream of the roadside camera;

[0033] The data monitoring and maintenance module is used to monitor and maintain the video stream and report anomalies if the picture changes and / or the analysis program cannot process normally;

[0034] The data processing module is used to perform data processing on the video stream to obtain processed data;

[0035] The target detection and tracking module is used to perform target detection and tracking on the processed data to obtain detection and tracking results;

[0036] The image weather classification module is used to perform image weather classification on the processed data to obtain a weather classification result;

[0037] The working condition division and parameter adjustment module is used to divide the working conditions and adjust the parameters in combination with the detection and tracking results and the weather classification results, and return them to the data processing module;

[0038] The starting line and ending line judgment module is used to judge whether the detection and tracking results have passed the starting line and the ending line, respectively, and obtain statistical results respectively;

[0039] The lane position judgment and vehicle type recognition module is used to calculate the travel speed, classify the vehicle type and the vehicle lane affiliation of the statistical results of the vehicle passing the end line, and record the information and return it to the data processing module;

[0040] The result recording module is used to analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

[0041] Optionally, judging whether the detection and tracking results have passed through a starting line and an ending line respectively includes:

[0042] ;

[0043] ;

[0044] in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

[0045] Optionally, calculating the travel speed of each vehicle for the statistical results of the vehicles passing the end line includes:

[0046] ;

[0047] in, Indicates the travel speed of the current road; n indicates the total number of lanes; represents the average speed of the i-th lane; is the congestion coefficient.

[0048] Optionally, classifying the vehicle types according to the statistical results of vehicles passing the end line includes:

[0049] Cut the vehicle area according to the detection frame;

[0050] Generate a zero-filled mask image of the same size as the cut image;

[0051] Concatenate the cut image and the mask image to obtain data of several channels;

[0052] A classification neural network is used to classify vehicle models based on data from several channels.

[0053] Optionally, the vehicle lane assignment for the statistical results of the vehicles passing through the end line includes:

[0054] ;

[0055] in, It means that when y= The x-coordinate of the quadratic equation of the lane line when represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

[0056] Technical effect of the invention: The invention discloses a traffic flow statistics and congestion analysis method and system based on video analysis. The system can monitor traffic flow 24 hours a day, ensure real-time analysis and feedback of data, and greatly improve the response speed of traffic management; based on convolutional neural network image analysis technology, the accuracy of target detection and vehicle type classification exceeds 95%, ensuring the reliability and accuracy of monitoring data; through automated data processing and statistical methods, manual operation and intervention are reduced, labor costs are saved, and overall work efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0058] Figure 1 This is a schematic diagram of the start line and the end line of an embodiment of the present invention;

[0059] Figure 2 A flow chart of a method for traffic flow statistics and congestion analysis based on video analysis according to an embodiment of the present invention;

[0060] Figure 3 The present invention is a schematic diagram of the structure of a traffic flow statistics and congestion analysis system based on video analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0063] In the current field of traffic monitoring, the technical problem that needs to be solved urgently is how to achieve accurate traffic situation surveys at a low cost based on existing roadside cameras. This involves multiple technical challenges, including: Data processing and analysis: How to effectively extract traffic flow information from images and videos captured by roadside cameras and analyze congestion conditions in real time. Vehicle type recognition and classification: How to identify and classify different types of vehicles through existing cameras to enhance the statistical accuracy of traffic flow. System integration: How to seamlessly integrate the newly developed algorithm with the existing roadside camera system to ensure the availability and stability of the system and avoid interference with traffic management. Cost control: Under the premise of ensuring monitoring accuracy, how to optimize the technical solution and reduce the implementation cost to make it attractive to various traffic management units, thereby promoting the popularization and application of technology. By solving the above technical problems, low-cost and accurate traffic situation surveys based on existing roadside cameras can be achieved, thereby providing more reliable data support for traffic management and decision-making.

[0064] like Figure 2 As shown, this embodiment provides a traffic flow statistics and congestion analysis method based on video analysis, including:

[0065] S1, initialize the start line, end line and lane line, and obtain the real-time video stream of the roadside camera;

[0066] S2. Monitor and maintain the video stream, and report anomalies if the image changes and / or the analysis program cannot process normally;

[0067] S3, performing data processing on the video stream to obtain processed data;

[0068] S4, performing target detection and tracking on the processed data to obtain detection and tracking results;

[0069] S5, performing image weather classification on the processed data to obtain a weather classification result;

[0070] S6. Combine the detection and tracking results and the weather classification results to divide the working conditions and adjust the parameters, and return to S3;

[0071] S7, judging whether the detection and tracking results have passed through the starting line and the ending line, respectively, and obtaining statistical results respectively;

[0072] S8, calculating the travel speed, classifying the vehicle type and the lane affiliation of the vehicle based on the statistical results of the vehicles passing the end line, and recording the information, and returning to S3;

[0073] S9. Analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

[0074] Furthermore, the data processing process includes:

[0075] When installing the device for the first time, the installer needs to connect the laptop to the same network segment as the industrial computer and enter the configuration function webpage to make the following settings:

[0076] Enter the camera location and the RTSP intranet video stream address of the camera to be analyzed.

[0077] After the video is successfully loaded, the lane recognition algorithm will recognize the lanes in the image. Each lane will be sorted from left to right and the coordinates of each lane will be saved as a fitted quadratic equation:

[0078] ;

[0079] Where i represents the i-th lane line; x and y are the image pixel coordinates respectively; , , are the quadratic fitting coefficients respectively.

[0080] At the same time, a video will be analyzed first. The analysis method is the same as the processing flow mentioned below. Here, we mainly obtain the track_id of the tracked vehicle, the vehicle type car_cls, the lane where the vehicle is located, and the grounding point Points of the tire on one side of the vehicle. The grounding point is also a reflection of the wheelbase and number of axles of the vehicle. The number of axles of a passenger car is 2, which is relatively simple. The wheelbase fluctuates little, about 2.5m, so car_cls is retained here as a passenger car, and the Points of the vehicle in each frame are recorded. When at least one passenger car passes completely in each lane, each track_id is analyzed. First, the Points closest to the lower boundary of the image are found and recorded as P t , assuming P t =[P1 t ,P2 t ],P1 t is the point closest to the lower boundary of the image, and then P2 t Search for the nearest point P1 to the center t-n , that is, the coordinates of point P2 n frames ago are near the coordinates of point P1 in the current frame, so we can get a line segment P1 t ~P2 t-n , the length of the line segment is about 2 times the wheelbase. Here the wheelbase is preset to 2.5m, so the length of the line segment is 5m. Because there is a complete vehicle trajectory, we can finally get a line segment starting from the lower boundary of the image with an actual length of 30m. Similarly, this line segment can be found in each lane. In each line segment, select the line connecting the points 10m away from the image as the starting line and the line connecting the points 30m away from the image as the ending line, and each line can be fitted into a straight line for subsequent calculations. See the starting line , End line Picture.

[0081] ;

[0082] ;

[0083] in, , are the image coordinates, , are the intercept and slope of the starting line, , are the intercept and slope of the end line respectively.

[0084] At this time, the actual distance between the two line segments is also obtained, and the actual distance between the two lines is d=20m.

[0085] The start and end lines are drawn on one of the frames and displayed on a canvas. The installer reviews the result to see if it is correct. If not, the result is redrawn on the canvas. and , and modify the distance d. Finally save the starting line , End line pixel coordinates , the actual distance d between the two lines, the equations of each lane line and the picture used for drawing are used as the initial parameters of the camera.

[0086] Target detection neural network algorithm: YOLO (You Only Look Once) is a target detection algorithm based on deep learning, which aims to achieve real-time target detection. Compared with traditional target detection algorithms, YOLO predicts multiple bounding boxes and their corresponding class probabilities at one time with a single neural network through regression problems, thereby significantly improving the speed and accuracy of target detection. YOLOv8-Pose is one of the latest versions of the YOLO series, specially designed for real-time target detection and pose estimation. Compared with traditional target detection methods, YOLOv8-Pose can not only identify target objects (such as vehicles, pedestrians, etc.), but also detect the key point positions of objects, thereby achieving more detailed analysis and understanding. The key point detected in this embodiment is the tire contact point of the vehicle. The key point information can more finely represent the wheelbase and number of axles of the vehicle, providing more accurate support for subsequent vehicle classification.

[0087] Multi-target tracking algorithm: ByteTrack is a multi-target tracking algorithm based on deep learning, which aims to efficiently and accurately track multiple targets in real time in dynamic scenes. The algorithm adopts a novel association strategy and uses feature aggregation and matching technology to effectively solve the common problems of occlusion, loss and reappearance of targets during motion. Compared with traditional tracking algorithms, ByteTrack can significantly improve the stability and accuracy of target tracking and is widely used in various computer vision tasks, such as video surveillance, intelligent transportation, etc.

[0088] Vehicle classification neural network algorithm: The vehicle detection frame and key points identified by the target detection neural network are synthesized into a four-channel image data. The synthesis method is as follows:

[0089] Cut the vehicle area according to the detection frame;

[0090] Generate a zero-filled mask of the same size as the cut image, modify the key point position to 255, and the rest of the area to 0;

[0091] Concatenate the cut image and the mask image to get a 4-channel data;

[0092] Use a classification neural network to classify the vehicle model of 4-channel data.

[0093] This method integrates image information and vehicle wheelbase and axle number information. Compared with traditional image classification, it adds preset features of classification objects and can achieve vehicle classification tasks more accurately. The classification results include: small passenger cars, medium-sized buses, large buses, small trucks, medium-sized trucks, large trucks, extra-large trucks, container trucks, and motorcycles. At the same time, the classification algorithm is used when the vehicle passes the end line to obtain the best viewing angle image to ensure the classification effect.

[0094] Lane line recognition neural network algorithm: can effectively identify lane markings such as solid lines and dotted lines.

[0095] Weather classification neural network algorithm: Use the classification neural network algorithm to judge the current weather conditions based on the image. The classification results include: sunny, cloudy, rainy, foggy, and snowy.

[0096] Furthermore, the data analysis statistical process includes:

[0097] When the processing software is started, the video stream will be read. To ensure real-time performance, the convolutional neural network algorithm will analyze the video at a rate of at least 12 frames per second and perform periodic statistical analysis:

[0098] Get the unique identifier track_id and the pixel coordinates of the detection box BBox=[x min ,y min ,x max ,y max ]、Key point pixel coordinates Points=[(x1,y1),...(x i ,y i )], set the x coordinate of the lower boundary midpoint of the BBox (x mid =(x min +x max ) / 2) respectively into the linear equations of the starting line and the ending line:

[0099] ;

[0100] ;

[0101] in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

[0102] When min < When y min First greater than or equal to When y min Between When y is between , the vehicle is considered to be in the area formed by the start line and the end line. min First time greater than The vehicle is considered to have passed the end line, and the end time t2 is recorded. The vehicle speed v = d / (t2 - t1) is calculated using the actual distance d between the known start line and the end line.

[0103] When the vehicle passes through the image frame of the end line, the vehicle image is cropped based on BBox and then spliced ​​with the mask image generated by Points into 4-channel data, and the image classification convolutional neural network is used to determine the model of the current vehicle.

[0104] When the vehicle passes through the image frame of the end line, according to the center point coordinates of the BBox (x', y') = (x min +x max ) / 2,(y min +y max ) / 2) Substitute into each lane line equation:

[0105] ;

[0106] in, It means that when y= The x-coordinate of the quadratic equation of the lane line when represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

[0107] Find the closest It is the lane line closest to the vehicle. Based on the size relationship, we can determine which lane the current vehicle belongs to.

[0108] When the vehicle passes the finish line, the number of vehicle types is increased by one.

[0109] The recorded information includes: frame index frame_id, vehicle tracking ID (track_id), time t1 when the vehicle passes the start line, time t2 when the vehicle passes the end line, vehicle speed v, vehicle model car_cls and lane information lane. Continue to process the next frame and repeat this process.

[0110] The data analysis module regularly (20 seconds by default) collects statistics and records information to perform congestion analysis and traffic flow statistics. The specific logic is as follows:

[0111] Get all record information within the statistical period;

[0112] Handling special situations, when individual vehicles are driving slowly or stopping abnormally, the average travel speed will be low. If the travel speed alone cannot determine whether it is a congested state, it is necessary to process the relationship between the number of slow-driving or abnormally stopped vehicles and the total number of vehicles that have not left the driving interval during the statistical period to make a judgment. The specific steps are: record the vehicles that have not left the statistical interval during the statistical period, that is, the recorded data has t1 but t2 is empty, and judge the number of continuous frames for each track_id that has not left. When the number of continuous frames is greater than 1 / 3 of the total processing frame rate of the statistical period, it is considered that there is slow passage or abnormal parking. When the number of slow passage or abnormal parking is less than 1 / 3 of the number of vehicles that have not left the interval, it is considered that individual vehicles are driving slowly or stopping abnormally. The congestion state is judged based on the average travel speed after excluding the speed of individual slow-driving or abnormally stopped vehicles. Otherwise, it is considered to be caused by traffic congestion;

[0113] After excluding vehicles with special conditions, the average speed is calculated for each lane as the travel speed v within the lane's statistical period;

[0114] The weighted average of the travel speeds of different lanes is the travel speed V of the current road;

[0115] ;

[0116] in, represents the average speed of the i-th lane, n represents the total number of lanes, is the congestion coefficient;

[0117] in .

[0118] Congestion status C is determined. When the average travel speed of the road is less than 10km / h, and no vehicles pass after excluding individual slow-moving or abnormally parked vehicles, and the average travel speed is 0km / h, it is considered that no vehicles pass and the congestion status is unblocked; if the travel speed is still less than 10km / h after excluding them, the travel speed is modified to the road design speed, and the congestion status is congested. When the travel speed is between 10km / h and 40km / h, the congestion status is general, and when the travel speed is greater than 40km / h, it is unblocked;

[0119] ;

[0120] The number of each vehicle model during the statistical period.

[0121] The relevant analysis results of the time period are recorded and saved locally, including camera location, time, congestion level, average travel speed, and traffic statistics of each lane.

[0122] Furthermore, the video stabilization process includes:

[0123] Since the roadside camera is installed on a gantry or lamp pole, the camera on the lamp pole may shake due to the influence of wind, so the current frame needs to be anti-shake processed before recognizing each frame of the image, so that the recognition and tracking results will be more accurate. Here, the optical flow method is used to estimate the motion vector between adjacent frames, and the camera shake is reversely compensated based on the estimated motion vector. The specific logic of video anti-shake is:

[0124] a. Apply the Lucas-Kanade optical flow method to adjacent frames to calculate the motion vector .

[0125] b. Based on the calculated motion vector , perform reverse compensation on each frame to generate a stable image frame. For the location Motion vector at . Compensated image It can be expressed as:

[0126] ;

[0127] c. Use bilinear interpolation to handle sub-pixel motion. To handle sub-pixel motion, bilinear interpolation can be used to calculate :

[0128] ;

[0129] in

[0130] ;

[0131] .

[0132] Furthermore, the process of operating condition division and coefficient adjustment includes:

[0133] Weather Condition Detection: Identify current weather conditions using a weather classification neural network algorithm.

[0134] Time period division: Automatically divide different time periods according to timestamp information.

[0135] Congestion assessment: Evaluate current traffic conditions by analyzing vehicle density and average speed.

[0136] Adaptive parameter adjustment: Dynamically adjust image processing parameters and target detection thresholds based on detection results.

[0137] Weather classification models: Use pre-trained weather classification models, such as those based on convolutional neural networks, to identify current weather conditions.

[0138] Weather classification results W: ;

[0139] Timestamp T: current timestamp.

[0140] Time division:

[0141] ;

[0142] ;

[0143] Parameter adaptive adjustment:

[0144] Image processing parameters:

[0145] Brightness L img and contrast C img :

[0146] ;

[0147] ;

[0148] The formula for image adjustment is: ,in is the source image pixel, is the output image pixel.

[0149] Object detection threshold τ:

[0150] ;

[0151] ByteTrack parameter adjustment:

[0152] Kalman filter parameters: Adjust the state transfer matrix A and observation matrix H of the Kalman filter to adapt to different working conditions.

[0153] ;

[0154] ;

[0155] Δt: represents the time interval, usually the time difference between frames.

[0156] In congested conditions, the speed of the target changes slowly, so a smaller Δt is used to reflect this feature.

[0157] In non-congested conditions, the speed of the target changes rapidly, so a larger 2Δt is used to reflect this feature.

[0158] Cost function of the Hungarian algorithm: Adjust the cost function to suit different working conditions.

[0159] ;

[0160] IoU: Intersection over Union between objects and detections.

[0161] λ: The weight coefficient of distance, which is set to a higher value in congested situations because the distances between objects are closer and higher weights are needed to distinguish objects.

[0162] μ: The weight coefficient of distance, which is set to a lower value in non-congested situations because the distance between targets is far and a high weight is not needed to distinguish targets.

[0163] Distance: Euclidean distance or other distance metric between the target and the detection.

[0164] Furthermore, data reporting includes:

[0165] After the analysis results are saved locally, the reporting module will read these results and upload them through the cloud interface or the data interface provided by other relevant departments.

[0166] Further, the perspective / resource monitoring includes:

[0167] At regular intervals (the default is 5 minutes), a frame of image is selected for lane line recognition, and the recognized lane line information is compared with the lane line information during parameter configuration. If a large deviation (greater than 5%) occurs, the situation will be reported to the cloud or the relevant department interface, and the relevant personnel will update the configuration parameters. At the same time, this module will also determine whether the image is normal and whether the central industrial computer processing is normal. If there is an abnormality, it will also be reported to notify the relevant personnel.

[0168] Through the implementation of the above technical solutions, real-time monitoring and analysis of vehicle flow can be achieved efficiently and accurately, providing effective support for traffic management.

[0169] like Figure 3 As shown, this embodiment also provides a traffic flow statistics and congestion analysis system based on video analysis, including:

[0170] Data acquisition module, data monitoring and maintenance module, data processing module, target detection and tracking module, image weather classification module, working condition division and parameter adjustment module, start line and end line judgment module, lane position judgment and vehicle type recognition module, result recording module;

[0171] The data acquisition module is used to initialize the start line, end line and lane line, and acquire the real-time video stream of the roadside camera;

[0172] The data monitoring and maintenance module is used to monitor and maintain the video stream and report anomalies if the picture changes and / or the analysis program cannot process normally;

[0173] The data processing module is used to perform data processing on the video stream to obtain processed data;

[0174] The target detection and tracking module is used to perform target detection and tracking on the processed data to obtain detection and tracking results;

[0175] The image weather classification module is used to perform image weather classification on the processed data to obtain a weather classification result;

[0176] The working condition division and parameter adjustment module is used to divide the working conditions and adjust the parameters in combination with the detection and tracking results and the weather classification results, and return them to the data processing module;

[0177] The starting line and ending line judgment module is used to judge whether the detection and tracking results have passed the starting line and the ending line, respectively, and obtain statistical results respectively;

[0178] The lane position judgment and vehicle type recognition module is used to calculate the travel speed, classify the vehicle type and the vehicle lane affiliation of the statistical results of the vehicle passing the end line, and record the information and return it to the data processing module;

[0179] The result recording module is used to analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

[0180] Furthermore, the data processing process includes:

[0181] When installing the device for the first time, the installer needs to connect the laptop to the same network segment as the industrial computer and enter the configuration function webpage to make the following settings:

[0182] Enter the camera location and the RTSP intranet video stream address of the camera to be analyzed.

[0183] After the video is successfully loaded, the lane recognition algorithm will recognize the lanes in the image. Each lane will be sorted from left to right and the coordinates of each lane will be saved as a fitted quadratic equation:

[0184] ;

[0185] Where i represents the i-th lane line; x and y are the image pixel coordinates respectively; , , are the quadratic fitting coefficients respectively.

[0186] At the same time, a video will be analyzed first. The analysis method is the same as the processing flow mentioned below. Here, we mainly obtain the track_id of the tracked vehicle, the vehicle type car_cls, the lane where the vehicle is located, and the grounding point Points of the tire on one side of the vehicle. The grounding point is also a reflection of the wheelbase and number of axles of the vehicle. The number of axles of a passenger car is 2, which is relatively simple. The wheelbase fluctuates little, about 2.5m, so car_cls is retained here as a passenger car. The Points of the vehicle in each frame are recorded. When at least one passenger car passes completely in each lane, each track_id is analyzed. First, the Points closest to the lower boundary of the image are found and recorded as P t , assuming P t =[P1 t ,P2 t ],P1 t is the point closest to the lower boundary of the image, and then P2 t Search for the nearest point P1 to the center t-n , that is, the coordinates of point P2 n frames ago are near the coordinates of point P1 in the current frame, so we can get a line segment P1 t ~P2 t-n , the length of the line segment is about 2 times the wheelbase. Here the wheelbase is preset to 2.5m, so the length of the line segment is 5m. Because there is a complete vehicle trajectory, we can finally get a line segment starting from the lower boundary of the image with an actual length of 30m. Similarly, this line segment can be found in each lane. In each line segment, select the line connecting the points 10m away from the image as the starting line and the line connecting the points 30m away from the image as the ending line, and each line can be fitted into a straight line for subsequent calculations. See the starting line , End line Picture.

[0187] ;

[0188] ;

[0189] in, , are the image coordinates, , are the intercept and slope of the starting line, , are the intercept and slope of the end line respectively.

[0190] At this time, the actual distance between the two line segments is also obtained, d=20m.

[0191] The start and end lines are drawn on one of the frames and displayed on a canvas. The installer reviews the result to see if it is correct. If not, the result is redrawn on the canvas. and , and modify the distance d. Finally save the starting line , End line pixel coordinates , the actual distance d between the two lines, the equations of each lane line and the picture used for drawing are used as the initial parameters of the camera.

[0192] Target detection neural network algorithm: YOLO (You Only Look Once) is a target detection algorithm based on deep learning, which aims to achieve real-time target detection. Compared with traditional target detection algorithms, YOLO uses a regression problem to predict multiple bounding boxes and their corresponding class probabilities at one time with a single neural network, thereby significantly improving the speed and accuracy of target detection. YOLOv8-Pose is one of the latest versions of the YOLO series, specially designed for real-time target detection and pose estimation. Compared with traditional target detection methods, YOLOv8-Pose can not only identify target objects (such as vehicles, pedestrians, etc.), but also detect the key point positions of objects, thereby achieving more detailed analysis and understanding. The key point detected in this application is the tire contact point of the vehicle. The key point information can more finely represent the wheelbase and number of axles of the vehicle, providing more accurate support for subsequent vehicle classification.

[0193] Multi-target tracking algorithm: ByteTrack is a multi-target tracking algorithm based on deep learning, which aims to efficiently and accurately track multiple targets in real time in dynamic scenes. The algorithm adopts a novel association strategy and uses feature aggregation and matching technology to effectively solve the common problems of occlusion, loss and reappearance of targets during motion. Compared with traditional tracking algorithms, ByteTrack can significantly improve the stability and accuracy of target tracking and is widely used in various computer vision tasks, such as video surveillance, intelligent transportation, etc.

[0194] Vehicle classification neural network algorithm: The vehicle detection frame and key points identified by the target detection neural network are synthesized into a four-channel image data. The synthesis method is as follows:

[0195] Cut the vehicle area according to the detection frame;

[0196] Generate a zero-filled mask of the same size as the cut image, modify the key point position to 255, and the rest of the area to 0;

[0197] Concatenate the cut image and the mask image to get a 4-channel data;

[0198] Use a classification neural network to classify the vehicle model of 4-channel data.

[0199] This method integrates image information and vehicle wheelbase and axle number information. Compared with traditional image classification, it adds preset features of classification objects and can achieve vehicle classification tasks more accurately. The classification results include: small passenger cars, medium-sized buses, large buses, small trucks, medium-sized trucks, large trucks, extra-large trucks, container trucks, and motorcycles. At the same time, the classification algorithm is used when the vehicle passes the end line to obtain the best viewing angle image to ensure the classification effect.

[0200] Furthermore, the data analysis statistical process includes:

[0201] When the processing software is started, the video stream will be read. To ensure real-time performance, the convolutional neural network algorithm will analyze the video at a rate of at least 12 frames per second and perform periodic statistical analysis:

[0202] Get the unique identifier track_id and the pixel coordinates of the detection box BBox=[x min ,y min ,x max ,y max ]、Key point pixel coordinates Points=[(x1,y1),...(x i ,y i )], set the x coordinate of the lower boundary midpoint of the BBox (x mid =(x min +xmax ) / 2) respectively into the linear equations of the starting line and the ending line:

[0203] ;

[0204] ;

[0205] in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

[0206] When min < When y min First greater than or equal to When y min Between When y is between , the vehicle is considered to be in the area formed by the start line and the end line. min First time greater than The vehicle is considered to have passed the end line, and the end time t2 is recorded. The vehicle speed v = d / (t2 - t1) is calculated using the actual distance d between the known start line and the end line.

[0207] When the vehicle passes through the image frame of the end line, the vehicle image is cropped based on BBox and then spliced ​​with the mask image generated by Points into 4-channel data, and the image classification convolutional neural network is used to determine the model of the current vehicle.

[0208] When the vehicle passes through the image frame of the end line, according to the center point coordinates of the BBox (x', y') = (x min +x max ) / 2,(y min +y max ) / 2) Substitute into each lane line equation:

[0209] ;

[0210] in, It means that when y= The x-coordinate of the quadratic equation of the lane line when represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

[0211] Find the closest It is the lane line closest to the vehicle. Based on the size relationship, we can determine which lane the current vehicle belongs to.

[0212] When the vehicle passes the finish line, the number of vehicle types is increased by one.

[0213] The recorded information includes: frame index frame_id, vehicle tracking ID (track_id), time t1 when the vehicle passes the start line, time t2 when the vehicle passes the end line, vehicle speed v, vehicle model car_cls and lane information lane. Continue to process the next frame and repeat this process.

[0214] The data analysis module regularly (20 seconds by default) collects statistics and records information to perform congestion analysis and traffic flow statistics. The specific logic is as follows:

[0215] Get all record information within the statistical period;

[0216] Handling special situations, when individual vehicles are driving slowly or stopping abnormally, the average travel speed will be low. If the travel speed alone cannot determine whether it is a congested state, it is necessary to process the relationship between the number of slow-driving or abnormally stopped vehicles and the total number of vehicles that have not left the driving interval during the statistical period to make a judgment. The specific steps are: record the vehicles that have not left the statistical interval during the statistical period, that is, the recorded data has t1 but t2 is empty, and judge the number of continuous frames for each track_id that has not left. When the number of continuous frames is greater than 1 / 3 of the total processing frame rate of the statistical period, it is considered that there is slow passage or abnormal parking. When the number of slow passage or abnormal parking is less than 1 / 3 of the number of vehicles that have not left the interval, it is considered that individual vehicles are driving slowly or stopping abnormally. The congestion state is judged based on the average travel speed after excluding the speed of individual slow-driving or abnormally stopped vehicles. Otherwise, it is considered to be caused by traffic congestion;

[0217] After excluding vehicles with special conditions, the average speed is calculated for each lane as the travel speed v within the lane's statistical period;

[0218] The weighted average of the travel speeds of different lanes is the travel speed V of the current road;

[0219] ;

[0220] in, represents the average speed of the i-th lane, n represents the total number of lanes, is the congestion coefficient;

[0221] in .

[0222] Congestion status C is determined. When the average travel speed of the road is less than 10km / h, and no vehicles pass after excluding individual slow-moving or abnormally parked vehicles, and the average travel speed is 0km / h, it is considered that no vehicles pass and the congestion status is unblocked; if the travel speed is still less than 10km / h after excluding them, the travel speed is modified to the road design speed, and the congestion status is congested. When the travel speed is between 10km / h and 40km / h, the congestion status is general, and when the travel speed is greater than 40km / h, it is unblocked;

[0223] ;

[0224] The number of each vehicle model during the statistical period.

[0225] The relevant analysis results of the time period are recorded and saved locally, including camera location, time, congestion level, average travel speed, and traffic statistics of each lane.

[0226] Furthermore, the video stabilization process includes:

[0227] Since the roadside camera is installed on a gantry or lamp pole, the camera on the lamp pole may shake due to the influence of wind, so the current frame needs to be anti-shake processed before recognizing each frame of the image, so that the recognition and tracking results will be more accurate. Here, the optical flow method is used to estimate the motion vector between adjacent frames, and the camera shake is reversely compensated based on the estimated motion vector. The specific logic of video anti-shake is:

[0228] a. Apply the Lucas-Kanade optical flow method to adjacent frames to calculate the motion vector .

[0229] b. Based on the calculated motion vector , perform reverse compensation on each frame to generate a stable image frame. For the location Motion vector at . Compensated image It can be expressed as:

[0230] ;

[0231] c. Use bilinear interpolation to handle sub-pixel motion. To handle sub-pixel motion, bilinear interpolation can be used to calculate :

[0232] ;

[0233] in

[0234] ;

[0235] .

[0236] The present invention discloses a traffic flow statistics and congestion analysis method and system based on video analysis, which can monitor traffic flow 24 hours a day, ensure real-time analysis and feedback of data, and greatly improve the response speed of traffic management; based on the image analysis technology of convolutional neural network, the accuracy of target detection and vehicle type classification exceeds 95%, ensuring the reliability and accuracy of monitoring data; through automated data processing and statistical methods, manual operation and intervention are reduced, labor costs are saved, and overall work efficiency is improved. By using existing roadside cameras, the high camera procurement and installation costs are avoided. At the same time, the deployment of the central industrial control computer also has cost advantages compared with other monitoring equipment, reducing the overall investment. Since the system is integrated in the central industrial control computer, maintenance and upgrading become more centralized and simple, reducing the maintenance cost caused by multi-device management. By obtaining detailed traffic flow data, the traffic management department can better allocate resources and optimize traffic signal control, thereby improving traffic efficiency and reducing economic losses caused by congestion. Real-time monitoring of traffic flow not only improves the efficiency of traffic management, but also provides data support for traffic safety, helps to deal with traffic accidents and abnormal events in a timely manner, and reduces the accident rate. By improving road traffic efficiency and reducing traffic congestion, it reduces vehicle fuel consumption and exhaust emissions, helping to reduce environmental pollution caused by traffic. The implementation of the present invention helps the construction of intelligent transportation systems, promotes the intelligent process of traffic monitoring and management, enhances the scientific and technological content of urban traffic management, and lays the foundation for the construction of smart cities in the future. In summary, the pure visual solution based on convolutional neural networks of the present invention not only performs well in terms of execution efficiency, economic benefits and social benefits, but also provides strong support for the intelligence and modernization of the field of traffic management.

[0237] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A traffic flow statistics and congestion analysis method based on video analysis, characterized in that: include: S1, initialize the start line, end line and lane line, and obtain the real-time video stream of the roadside camera; S2. Monitor and maintain the video stream, and report abnormalities if the picture changes and / or the analysis program cannot process normally; S3, performing data processing on the video stream to obtain processed data; S4, performing target detection and tracking on the processed data to obtain detection and tracking results; S5, performing image weather classification on the processed data to obtain a weather classification result; S6. Combine the detection and tracking results and the weather classification results to divide the working conditions and adjust the parameters, and return to S3; S7, judging whether the detection and tracking results have passed through the starting line and the ending line, respectively, and obtaining statistical results respectively; S8, calculating the travel speed, classifying the vehicle type and the lane affiliation of the vehicle based on the statistical results of the vehicles passing the end line, and recording the information, and returning to S3; S9. Analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

2. The method for traffic flow statistics and congestion analysis based on video analysis as claimed in claim 1, characterized in that: The detection and tracking results are respectively judged whether they have passed the starting line and the ending line, and the statistical results are obtained respectively, including: ； ； in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

3. The method for traffic flow statistics and congestion analysis based on video analysis as claimed in claim 1, characterized in that: The calculation of the travel speed for the statistical results of the vehicles passing the end line includes: ； in, Indicates the travel speed of the current road; n indicates the total number of lanes; represents the average speed of the i-th lane; is the congestion coefficient.

4. The method for traffic flow statistics and congestion analysis based on video analysis as claimed in claim 1, characterized in that: The vehicle type classification of the statistical results of passing the end line includes: Cut the vehicle area according to the detection frame; Generate a zero-filled mask image of the same size as the cut image; Concatenate the cut image and the mask image to obtain data of several channels; A classification neural network is used to classify vehicle models based on data from several channels.

5. The method for traffic flow statistics and congestion analysis based on video analysis as claimed in claim 1, characterized in that: The vehicle lane assignment for the statistical results of passing the end line includes: ； in, It means that when y= The x-coordinate of the quadratic equation of the lane line when Represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

6. A system for traffic flow statistics and congestion analysis method based on video analysis according to any one of claims 1 to 5, characterized in that: include: Data acquisition module, data monitoring and maintenance module, data processing module, target detection and tracking module, image weather classification module, working condition division and parameter adjustment module, start line and end line judgment module, lane position judgment and vehicle type recognition module, result recording module; The data acquisition module is used to initialize the start line, end line and lane line, and acquire the real-time video stream of the roadside camera; The data monitoring and maintenance module is used to monitor and maintain the video stream and report anomalies if the picture changes and / or the analysis program cannot process normally; The data processing module is used to perform data processing on the video stream to obtain processed data; The target detection and tracking module is used to perform target detection and tracking on the processed data to obtain detection and tracking results; The image weather classification module is used to perform image weather classification on the processed data to obtain a weather classification result; The working condition division and parameter adjustment module is used to divide the working conditions and adjust the parameters in combination with the detection and tracking results and the weather classification results, and return them to the data processing module; The starting line and ending line judgment module is used to judge whether the detection and tracking results have passed the starting line and the ending line, respectively, and obtain statistical results respectively; The lane position judgment and vehicle type recognition module is used to calculate the travel speed, classify the vehicle type and the vehicle lane affiliation of the statistical results of the vehicle passing the end line, and record the information and return it to the data processing module; The result recording module is used to analyze the congestion level and traffic statistics of each vehicle type based on the recorded information and report the data.

7. The system according to claim 6, characterized in that The determination of whether the detection and tracking results have passed through the starting line and the ending line respectively includes: ； ； in, Indicates that x= on the starting line The y coordinate of Indicates the slope of the linear fitting parameter of the starting line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the linear fitting parameter of the starting line; Indicates the end line x= The y coordinate of Indicates the slope of the linear fitting parameter of the end line; Indicates the x-coordinate of the midpoint of the lower boundary of the vehicle detection box; Indicates the intercept of the end line linear fit parameter.

8. The system according to claim 6, characterized in that The calculation of the travel speed for the statistical results of the vehicles passing the end line includes: ； in, Indicates the travel speed of the current road; n indicates the total number of lanes; represents the average speed of the i-th lane; is the congestion coefficient.

9. The system according to claim 6, characterized in that The vehicle type classification of the statistical results of passing the end line includes: Cut the vehicle area according to the detection frame; Generate a zero-filled mask image of the same size as the cut image; Concatenate the cut image and the mask image to obtain data of several channels; A classification neural network is used to classify vehicle models based on data from several channels.

10. The system according to claim 6, characterized in that The vehicle lane assignment for the statistical results of passing the end line includes: ； in, It means that when y= The x-coordinate of the quadratic equation of the lane line when Represents the quadratic coefficient of the i-th lane line; Indicates the y coordinate of the center of the vehicle detection frame; Represents the one-time coefficient of the i-th lane line; Represents the constant of the i-th lane line.

Images